Forum Discussion
Datflow gen2 incremental refresh parameters
Hi smpa01 , Thank you for reaching out to the Microsoft Community Forum.
Yes, your understanding of incremental refresh is correct and aligns with how it works in Microsoft Fabric and Power BI. When dealing with mutable data, using a filter column like OrderDate defines a sliding window, for example, the last 14 or 50 days. Only data within this window is considered during each refresh. Any changes outside it will be ignored unless you increase the window.
Within that window, the data is logically split into buckets based on your chosen granularity, typically daily in your case. For each bucket, the system checks whether the maximum value of ModifiedDate has changed compared to the last refresh. If it has, the bucket is refreshed; if not, it’s skipped. This is how Fabric efficiently refreshes only the changed data.
You’re also right that monthly buckets behave differently, if anything in the month changes, the entire month’s data is refreshed. It doesn’t track day-level changes within the month, so daily buckets are better for precise, recent updates, especially when you're working within the 50-bucket system limit. You are correct, if ModifiedDate isn’t updated when a record changes, the system will miss that change entirely. Incremental refresh relies fully on the accuracy of this column to detect updates.
Incremental refresh and real-time data for semantic models
Incremental refresh in Dataflow Gen2
Overview of query evaluation and query folding in Power Query
How Direct Lake mode works with Power BI reporting
If this helped solve the issue, please consider marking it “Accept as Solution” so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Yes, that's right. Immutable data is easy to manage (append new rows at the bottom yeterday's row). Mutable data manging is more challenging which is why Delta Merge or UPSERT concept comes into play in a db /lakehouse env. But there are sources which can only be consumed by df and how one makes sense of IR in respect to the same mutable data source scenario was the objective of creating this post. In business vase. on not works with immutable but also mutable data.
at the end of the day this falls back onto manual partition management. That is something you can do with Semantic Models. I am not aware of a way to do that with the buckets of DF Gen2.