Impact of Incremental Refresh on Power BI Service Model refresh (Fabric Capacity)

[Freece4802]

Hi, 

I was hoping to understand whether paritioning data and using incremental refresh would reduce the cached data stored as part of a model refresh? Microsoft highlight that as part of a model refresh, it stores the data temporary before its overwritten by new data. I'd assume that if the data is partitioned it would reduce the amount of data stored in cache. 

Any info would be useful. 

Thanks, 

[v-karpurapud · Accepted solution]

Hi @Freece4802

Welcome to the Microsoft Fabric Community Forum. Thank you @lbendlin for sharing useful insights.
 

Incremental refresh in Power BI does not reduce the cache used during the data processing(mashup) stage of a model refresh. This cache is managed by the Power Query engine and is specifically used for transforming and loading data before it is written to the model. Partitioning and incremental refresh do not influence this temporary cache.

However, incremental refresh significantly improves overall refresh performance and resource efficiency. It achieves this by refreshing only the partitions that have changed , enabling parallel processing of partitions, and reducing the volume of data that needs to be reprocessed. These optimizations lower memory and CPU usage during the model load stage, which is responsible for compressing and storing data in the in-memory engine.

Thus, while incremental refresh does not affect the mashup engine cache, it does enhance the efficiency and scalability of model refresh operations especially for large datasets by reducing refresh time and resource consumption.

For reference:
Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn

If this response resolves your query, kindly mark it as Accepted Solution to help other community members. A Kudos is also appreciated if you found the response helpful.

Thank you for being part of Fabric Community Forum.

Regards,
Karpurapu D,
Microsoft Fabric Community Support Team.

[v-karpurapud · Accepted solution]

Hi @Freece4802

Welcome to the Microsoft Fabric Community Forum. Thank you @lbendlin for sharing useful insights.
 

Incremental refresh in Power BI does not reduce the cache used during the data processing(mashup) stage of a model refresh. This cache is managed by the Power Query engine and is specifically used for transforming and loading data before it is written to the model. Partitioning and incremental refresh do not influence this temporary cache.

However, incremental refresh significantly improves overall refresh performance and resource efficiency. It achieves this by refreshing only the partitions that have changed , enabling parallel processing of partitions, and reducing the volume of data that needs to be reprocessed. These optimizations lower memory and CPU usage during the model load stage, which is responsible for compressing and storing data in the in-memory engine.

Thus, while incremental refresh does not affect the mashup engine cache, it does enhance the efficiency and scalability of model refresh operations especially for large datasets by reducing refresh time and resource consumption.

For reference:
Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn

If this response resolves your query, kindly mark it as Accepted Solution to help other community members. A Kudos is also appreciated if you found the response helpful.

Thank you for being part of Fabric Community Forum.

Regards,
Karpurapu D,
Microsoft Fabric Community Support Team.

[Freece4802]

Thanks @v-karpurapud - So the above would suggest that where i'm currently recieving Resource Governance issues due to large model size, this could be migitgated by the implimentation of incremental refresh as it would reduce the resource consumption and model size requiring updating? 

Thanks, 

[v-karpurapud · Accepted solution]

Hi @Freece4802 

Yes, incremental refresh can help address Resource Governance issues in Power BI, but it's important to note where its benefits apply. While it doesn't reduce cache usage during the initial data transformation , it significantly improves efficiency during model loading. Incremental refresh processes only the new or changed data, rather than the entire dataset, which lowers memory and CPU usage, speeds up refresh times, and reduces the likelihood of hitting governance limits. This is particularly helpful for larger models where full refreshes can cause delays.


Regards,
Karpurapu D.

[lbendlin]

Has nothing to do with each other. The cache is for user queries, not for mashups.