Incremental Refresh in Power BI dataflow Gen 1 refreshing from past 13 hours

If the data source for your incremental refresh does not fold then you will not see any performance improvements, you will only get the "free" partition management.

NOTE:  The partition size limit is 10 GB.  Your amount of data is not nearly large enough for incremental refresh to make much sense (unless your data source has timeout issues)

Hi SamD02 ,
It's expected that the initial load for incremental refresh in Power BI Dataflow Gen1 can take a significant amount of time, especially with a large dataset like yours containing over 21 million records. Since you're storing 30 years of data and applying a refresh policy for only the past 2 days, the first refresh has to load and partition the entire historical dataset, which is resource-intensive and time-consuming. A 13-hour refresh duration for the first load, while long, may still be within the realm of normal depending on the complexity of your transformations, data source performance, and dataflow design.

However, to optimize performance, ensure that your queries are folded properly (i.e., query folding is applied), avoid unnecessary transformations before filters are applied, and consider breaking down the data into smaller, more manageable tables if possible. Once the initial load is done, subsequent refreshes should be significantly faster as only recent partitions will be refreshed.

Thankyou, lbendlin, rohit1991 , for your response.

Hi SamD02,

We appreciate your inquiry through the Microsoft Fabric Community Forum.
In addition to the information provided by lbendlin and rohit1991 , please follow the steps below to optimize performance:

1. Adjust Historical Data Range.If your data begins from 2016, storing 30 years of history may create unnecessary overhead. It is advisable to reduce the stored rows value to 10 years to mitigate partition load and streamline metadata management.

2. Since your data consistently loads as yyyy-MM-01, consider increasing the refresh rows to 5 days (from the current 2 days) to provide a better buffer and to avoid missed partitions due to processing delays.

3. If feasible, consider upgrading to Dataflow Gen2, which leverages an enhanced compute engine for faster refreshes and improved scalability.

Additionally, for better insights into what might be slowing down the refresh, please utilize Power Query diagnostics or the Dataflow refresh history in the workspace.

Kindly refer to the following link for more information:
Dataflows best practices - Power BI | Microsoft Learn

If you find our response helpful, please mark it as the accepted solution and provide kudos. This will assist other community members facing similar queries.

Thank you.