Forum Discussion

tikka's avatar
tikka
Frequent Visitor
7 years ago

Reading dataset from Azure Databricks

Hi,   We have a Databricks (Premium) environment set up in Azure. Databricks is also set up under a custom Azure Vnet. We are reading prepared datasets from PowerBI using the Databricks cluster's J...
Need Help
Anonymous's avatar
Anonymous
Not applicable
7 years ago

Did we find solution for this ? i am facing same issue, its forcing to use Gateway.

tikka's avatar
tikka
Frequent Visitor
7 years ago

Hi,

 

Unfortunately we have not found a solution yet. Both problems above relate to the fact that the Spark connector seems to iterate through all databases and tables in Databricks, and ultimately times out:

 

- dataset takes really long to refresh in PowerBI Desktop

- gateway cannot be created in PowerBI Service

 

For the later one, there is now an option to skip the connection test, which allows you to create the gateway, but does not resolve the performance issue.

 

As a workaround, we are using an Azure SQLServer to save the processed results from Databricks. Now connecting to Azure SQLServer from PowerBI works as expected, but this is an additional step though.

 

BR,

Tuomas

  • vijaykvs's avatar
    vijaykvs
    Microsoft Employee
    7 years ago

    We are also obersving that Direct Query connection mode to Databricks is very slow. Probably not a good solution at this moment unless performance improves drastically.