Forum Discussion
Refresh from Spark data source fails
I created a report in Power BI Desktop which loads data from a view in Azure Data Lake Storage Gen2. The view is connected to a Databricks Delta table with 500M rows. I'm using the Spark connector an...
Were you able to figure out the issue? Can you suggest the solution , i am also stuck in the same situation
This was almost a year ago, so I don't remember exactly when we did what or what made the issue finally disappear, but we don't experience this issue anymore. Initially, we scaled up the cluster every time we needed to do a full load on a large report, as opposed to just incremental refresh, but now we don't need to scale up the cluster anymore.
What we've done to improve the situation:
- We're using Power BI Premium
- We've upgraded to a newer version of the cluster's Databricks runtime (7.3 LTS)
- We're using slightly bigger nodes for the cluster (Standard_L4s instead of Standard_DS3_v2)