Starting December 3, join live sessions with database experts and the Microsoft product team to learn just how easy it is to get started
Learn moreGet certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now
power bi spark connector performance is extremely slow. it is taking 4 to 5 hours to process 6 gigs azure databricks delta tables into power bi premium nodes (p3). Both power bi premium capacity and azure databricks workspace are in same azure data center. Even after adjusting maxresultset and batch size performance is poor. I exported the delta tables into CSV files on ADLS Gen2 and was able to process it in 3X faster. What is wrong with power bi spark connector, we heard from databricks team that microsoft is working on the fix. Any updates / workarounds?
Dear Team,
I have tested power bi refresh using new spark connector available in recent sept 2020 power desktop release (version 2.85.681.0). I see the significant improvement in one the small scale model (<1 gig). In the past using old connector - model refreshed in 14 min while current refresh completed within 4 min and 11 sec. Using synapse as source - same model is refreshing in 3 min and 4 sec. Can someone explain what was the problem in the past and it was resolved in the new connector?
Anyways - thank you for excellent work to resolve the performance issue.
Hi, @Anonymous
Is it caused by an incompatibility between Databricks version 6.4 (and above) and connector used by Power BI service?
Take a try to downgrading Databricks to 5.5 LTS.
Best Regards,
Community Support Team _ Eason
hello Team,
Still no improvements. Do you have benchmark numbers for tpch dataset refresh from ADB Delta table to Premium capacity node? Also do you believe that we will have to live with expcation that spark connector is slow until its re-enginnered in the next iteration / fix or shall we open support ticket to check if this is enviornmental issue?
Best Reagrds
Shrikant Kulkarni
Hi , @Anonymous
Sorry I can't fix it on my side currently.
It is recommended to create a support ticket to investigate this issue.Once you create a support ticket and get any workaround, please kindly share it here for reference.
https://powerbi.microsoft.com/en-us/support/
Best Regards,
Community Support Team _ Eason
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Thank you!! support ticket was raised that time and microsoft power bi support was involved. In fact we did reached out to Adam saxson as well during that time.
Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.
User | Count |
---|---|
37 | |
30 | |
18 | |
14 | |
8 |
User | Count |
---|---|
50 | |
38 | |
31 | |
15 | |
13 |