INcremental Refresh making every report refresh way longer

[RogerSteinberg]

Hi,

I just activated the incremental refresh for 6 reports. Since  I did that, every reports in the workspace are taking forever to refresh.

My incremental refresh works fine. The query folding is happening for each report. There are no other power query steps after the navigation step. Note that every connection to SQL Server is through a view.

Here's an example of the implementation of incremental refresh after the after the 45 mn refresh from April 30th. 

Almost every single report has failed since i've activated this feature.

[v-yangliu-msft]

Hi  @RogerSteinberg  ,

Note that:

1. When using incremental refresh for the first time, all historical data will be imported first, so it will be slow the first time, and will slowly become faster later.

2. The refresh time limit of the Power BI Pro dataset is two hours. For data sets with advanced capacity, the time limit is 5 hours, even if incremental refresh is used, unless through the XMLA endpoint, refresh operations are subject to the same two and five hour limits.

3. The process of refresh operation requires a lot of memory, which consumes a lot of available CPU resources. Ensure that there is enough CPU and memory, otherwise it will cause refresh failure.

 4. You can choose to refresh the configuration in free time, or allocate the refresh plan to different time periods, because when many plans are refreshed at the same time, the model used for refresh will be loaded and carried out. Once there are a lot of models in the memory , Will cause the memory to be full, so that the model generated by the subsequent refresh operation will not load in and enter the queued state, so that the report will be loaded very slowly or even fail.

This is the relevant content of incremental refresh, I hope it will help you:

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-troubleshoot#configuring-...

Best Regards,

Liu Yang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Sumanth_23 · Accepted solution]

hi @RogerSteinberg - Once you setup the Incremental refresh - Power BI will execute a full load on your dataset, following which the next refreshes will be based on the incremental setup. 

Also since this is the case - you should try and schedule your refreshes at different times. I have noticed that Power BI will slow down all refreshes if resources are being used to refresh multiple datasets. 

Below is the details from the Microsoft documentation on the same:  https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview

Refresh

After publishing to the service, you perform an initial refresh operation on the dataset. This should be an individual (manual) refresh so you can monitor progress. The initial refresh operation can take quite a while to complete. Partitions must be created, historical data loaded, objects such as relationships and hierarchies are built or rebuilt, and calculated objects are recalculated.

Subsequent refresh operations, either individual or scheduled are much faster because only the incremental refresh partition(s) is refreshed. Other processing operations must still occur, like merging partitions and recalculation, but it usually takes only a small fraction of time compared to the initial refresh.

Hope this helps! 

Do vote/mark this post as a solution and provide kudos if it helped you with your question. This would also help others on the community who may have similar questions. Thanks!

[RogerSteinberg]

So do you mean I have to perform a manual refresh in the PBI service first then the scheduled refresh should be quick ?

[Sumanth_23]

Hi @RogerSteinberg, Yes, that is the approach recommended by Microsoft - the first refresh could run for an extended time as - partitions must be created, historical data loaded, objects such as relationships and hierarchies are built or rebuilt, and calculated objects are recalculated.

Do vote/mark this post as a solution and provide kudos 👍 if it helped you with your question. This would also help others on the community who may have similar questions. Thanks!