Power BI Importing Data from Azure Databricks

[claudedjale]

Hi experts, 

I have a question. I am working with a customer right now who set up a power BI desktop app in their environment to query data from Azure Databricks. It looks like the configurations are set up properly because after initial set up, the customer can see sample data, but the moment they want to import the data using the "import" mode, the request is timing out. The customer noticed that client's IP is also showing up in the storage firewall as blocked(this is expected). The storage has a firwall configured and should be blocking direct calls from client and this is expected. When the client's IP is white listed in the storage firewall, it the "import" data works just fine.

The customer concerns is that client's IP are directly accessing the storage account. This is a security risk for them. Another point is that using a VDI, the customer is also able to "import" the data.

It is normal that the client IP is in direct communication with the storage endpoint when the "import" method is selected? Do you have any public and internal documentation that explains this? The customer is curious to know why his client's IP is communicating directly with the storage account and not Databricks. They were hoping that the call will stop at Databricks.

Any insight will be helpful.

Thanks!

[Shahid12523 · Accepted solution]

Yes, it’s expected behavior.
In Import mode, Power BI Desktop downloads the data directly from the Databricks storage layer, so the client IP talks to the storage account (hence shows up in firewall logs).

That’s why whitelisting the client IP makes Import work.

In DirectQuery, only Databricks queries storage, not the client.

To avoid exposing client IPs, use DirectQuery, VDI, Gateway, or Managed VNET.

[claudedjale]

Hey @Shahid12523 , This is helpful. Do you have any public documentation that can be shared with external users?

[claudedjale]

Thank you lbendlin, can you point me to any documentation that can be shared externally?

[lbendlin]

Hmm, can't find the exact page (will have to ask our specialist tomorrow). But it is similar to this

Fast copy in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

[claudedjale]

Thank you!

[lbendlin]

It is normal that the client IP is in direct communication with the storage endpoint when the "import" method is selected?

What is normal these days... Anyways, yes - this direct connection has been touted as a new hot feature in Databricks, aimed at reducing the network impact of these transfers.  Very much similar to the DMA (Direct Memory Access)  feature of the microprocessors of yore.