Forum Discussion
Partitions taking too long to refresh
I have a big dataset with different partitions that I’m creating sequentially and/or in parallel, but everytime a new partition is trying to be refreshed in XMLA is increasing the time from 10 minutes to 4 hours. Not sure how to optimize this, I’m connecting to a catalog in databricks
2 Replies
- GilbertQ
Super User
Hi Anonymous
Can you confirm that when you are refreshing you are only refreshing the newly created partitions?
Hi Anonymous ,
Power BI supports advanced incremental refresh and real-time data features with the XMLA endpoint. This allows you to refresh only the data that has changed, rather than the entire dataset.
You can manage table partitions through the XMLA endpoint using tools like SQL Server Management Studio (SSMS), the open-source Tabular Editor, scripted with Tabular Model Scripting Language (TMSL), and programmatically with the Tabular Object Model (TOM)1. This can help optimize the refresh operation.
You may also optimize Data Layout The size of the Parquet files underneath the hood of a Delta table is crucial for query performance. To avoid the problem of too many tiny files, it’s best to have a file size between 16MB and 1GB.
Comprehensive Guide to Optimize Data Workloads | Databricks
Best regards.
Community Support Team_Caitlyn
