Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
Anonymous
Not applicable

​power bi spark connector poor performance

​power bi spark connector performance is extremely slow. it is taking 4 to 5 hours to process 6 gigs azure databricks delta tables into power bi premium nodes (p3). Both power bi premium capacity and azure databricks workspace are in same azure data center. Even after adjusting maxresultset and batch size performance is poor. I exported the delta tables into CSV files on ADLS Gen2 and was able to process it in 3X faster. What is wrong with power bi spark connector, we heard from databricks team that microsoft is working on the fix. Any updates / workarounds? 

5 REPLIES 5
Anonymous
Not applicable

Dear Team, 

I have tested power bi refresh using new spark connector available in recent sept 2020 power desktop release (version 2.85.681.0). I see the significant improvement in one the small scale model (<1 gig). In the past using old connector - model refreshed in 14 min while current refresh completed within 4 min and 11 sec. Using synapse as source - same model is refreshing in 3 min and 4 sec. Can someone explain what was the problem in the past and it was resolved in the new connector?     

Anyways - thank you for excellent work to resolve the performance issue. 

 

v-easonf-msft
Community Support
Community Support

Hi, @Anonymous 
Is it caused by  an incompatibility between Databricks version 6.4 (and above) and connector used by Power BI service?

Take a try to downgrading  Databricks to 5.5 LTS.

 

Best Regards,
Community Support Team _ Eason

Anonymous
Not applicable

hello Team, 

Still no improvements. Do you have benchmark numbers for tpch dataset refresh from ADB Delta table to Premium capacity node? Also do you believe that we will have to live with expcation that spark connector is slow until its re-enginnered in the next iteration / fix or shall we open support ticket to check if this is enviornmental issue? 

 

Best Reagrds

Shrikant Kulkarni 

 

Hi , @Anonymous 

Sorry I can't fix it on my side currently.

It is recommended to create a support ticket to investigate this issue.Once you create a support ticket and get any workaround, please kindly share it here for reference.
https://powerbi.microsoft.com/en-us/support/

 

Best Regards,
Community Support Team _ Eason
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Anonymous
Not applicable

Thank you!! support ticket was raised that time and microsoft power bi support was involved. In fact we did reached out to Adam saxson as well during that time.     

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Top Solution Authors