​power bi spark connector poor performance

[Anonymous]

​power bi spark connector performance is extremely slow. it is taking 4 to 5 hours to process 6 gigs azure databricks delta tables into power bi premium nodes (p3). Both power bi premium capacity and azure databricks workspace are in same azure data center. Even after adjusting maxresultset and batch size performance is poor. I exported the delta tables into CSV files on ADLS Gen2 and was able to process it in 3X faster. What is wrong with power bi spark connector, we heard from databricks team that microsoft is working on the fix. Any updates / workarounds? 

[Anonymous]

Dear Team, 

I have tested power bi refresh using new spark connector available in recent sept 2020 power desktop release (version 2.85.681.0). I see the significant improvement in one the small scale model (<1 gig). In the past using old connector - model refreshed in 14 min while current refresh completed within 4 min and 11 sec. Using synapse as source - same model is refreshing in 3 min and 4 sec. Can someone explain what was the problem in the past and it was resolved in the new connector?     

Anyways - thank you for excellent work to resolve the performance issue. 

[v-easonf-msft]

Hi， @Anonymous 
Is it caused by  an incompatibility between Databricks version 6.4 (and above) and connector used by Power BI service?

Take a try to downgrading  Databricks to 5.5 LTS.

Best Regards,
Community Support Team _ Eason

[Anonymous]

hello Team, 

Still no improvements. Do you have benchmark numbers for tpch dataset refresh from ADB Delta table to Premium capacity node? Also do you believe that we will have to live with expcation that spark connector is slow until its re-enginnered in the next iteration / fix or shall we open support ticket to check if this is enviornmental issue? 

Best Reagrds

Shrikant Kulkarni 

[v-easonf-msft]

Hi , @Anonymous 

Sorry I can't fix it on my side currently.

It is recommended to create a support ticket to investigate this issue.Once you create a support ticket and get any workaround, please kindly share it here for reference.
https://powerbi.microsoft.com/en-us/support/

Best Regards,
Community Support Team _ Eason
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

Thank you!! support ticket was raised that time and microsoft power bi support was involved. In fact we did reached out to Adam saxson as well during that time.