Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Anonymous
Not applicable

​power bi spark connector poor performance

​power bi spark connector performance is extremely slow. it is taking 4 to 5 hours to process 6 gigs azure databricks delta tables into power bi premium nodes (p3). Both power bi premium capacity and azure databricks workspace are in same azure data center. Even after adjusting maxresultset and batch size performance is poor. I exported the delta tables into CSV files on ADLS Gen2 and was able to process it in 3X faster. What is wrong with power bi spark connector, we heard from databricks team that microsoft is working on the fix. Any updates / workarounds? 

5 REPLIES 5
Anonymous
Not applicable

Dear Team, 

I have tested power bi refresh using new spark connector available in recent sept 2020 power desktop release (version 2.85.681.0). I see the significant improvement in one the small scale model (<1 gig). In the past using old connector - model refreshed in 14 min while current refresh completed within 4 min and 11 sec. Using synapse as source - same model is refreshing in 3 min and 4 sec. Can someone explain what was the problem in the past and it was resolved in the new connector?     

Anyways - thank you for excellent work to resolve the performance issue. 

 

v-easonf-msft
Community Support
Community Support

Hi, @Anonymous 
Is it caused by  an incompatibility between Databricks version 6.4 (and above) and connector used by Power BI service?

Take a try to downgrading  Databricks to 5.5 LTS.

 

Best Regards,
Community Support Team _ Eason

Anonymous
Not applicable

hello Team, 

Still no improvements. Do you have benchmark numbers for tpch dataset refresh from ADB Delta table to Premium capacity node? Also do you believe that we will have to live with expcation that spark connector is slow until its re-enginnered in the next iteration / fix or shall we open support ticket to check if this is enviornmental issue? 

 

Best Reagrds

Shrikant Kulkarni 

 

Hi , @Anonymous 

Sorry I can't fix it on my side currently.

It is recommended to create a support ticket to investigate this issue.Once you create a support ticket and get any workaround, please kindly share it here for reference.
https://powerbi.microsoft.com/en-us/support/

 

Best Regards,
Community Support Team _ Eason
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Anonymous
Not applicable

Thank you!! support ticket was raised that time and microsoft power bi support was involved. In fact we did reached out to Adam saxson as well during that time.     

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.