Forum Discussion
Defragment not working
Thanks again cpwebb , I followed the same steps to update the Parallel Loading Of Tables setting in Power BI Desktop.
Since the PeakMemory for the refresh as a whole is around 3.5 GB and I still face Memory capacity errors (even after tuning the power query for a table with very high MashupPeakMemory) , I looked into the calculated columns again and found something interesting:
We have 2 fact tables, and each fact table in linked to multiple dimention tables:
Fact table A with 350,000 rows.
Fact table B with 9 million rows.
There was a calculated column in Fact table A which was referencing both Fact table B and a dimension table linked to the table B.
Before removing that specific calculated column:
PeakMemory for the refresh as a whole: 3.5 GB
PeakMemory for Fact table A: 1.95 GB
PeakMemory for Fact table B: 2.85 GB
PeakMemory for the dimension table: 1.8 GB
After removing that specific calculated column:
PeakMemory for the refresh as a whole: 2.4 GB
PeakMemory for Fact table A: 1.3 GB
PeakMemory for Fact table B: 1.4 GB
PeakMemory for the dimension table: 175 MB
Considering the PeakMemory of 2.4GB, I still sometimes face memory errors despite being on an A2 SKU with 5GB of memory.
(I have defined an incremental refresh policy for the last 6 months with maxParallelism set to 3).
Since each table uses around 150-200MB of peak memory, and we have only 2 tables with a peak memory usage of around 1.4GB, could the parallel loading of tables be causing this error?
I believe that maxParallelism = 3 applies only to the partitions, not the tables. Is that correct? It would be great if we could schedule a refresh to refresh each table sequentially, similar to how we process the whole table in SSMS Analysis Services.
I even created a PowerShell script in Azure Runbooks to schedule a refresh with Logic Apps, but when I add "maxParallelism = 3" to the script, I get an error.
Thanks a lot!
No, maxParallelism applies to any individual refresh job inside a single Batch/Sequence command, so it can apply to tables or partitions. The setting I mentioned here https://blog.crossjoin.co.uk/2022/10/31/speed-up-power-bi-dataset-refresh-performance-in-premium-or-ppu-by-changing-the-parallel-loading-of-tables-setting/ controls the parallelism of refresh for tables, so setting this to 1 should help reduce the peak memory.
Can you also share the definition of the calculated column you were talking about?