Forum Discussion
Helper IRecommended Incremental refresh settings for BC365 fact tables (like GL Entry)?
Super UserIf these data points are not immutable then incremental refresh is not the right tool for what you are trying to achieve.
Custom partition management is not (yet) available for dataflows.
Consider loading your data directly into a semantic model. There you have fine grain partition control.
Or consider loading your data into a data warehouse or data lake with pipelines.
useruserhi91Thanks for your reply. However, of course "systemModifiedAt" is NOT immutable - it changes. The whole point of an incremental refresh is to identify rows that have a dateTime value changed recently, indicating that the row has indeed changed, and must be updated. If this is not possible, what's the point of this incremental refresh feature? Can you give a concrete example of which columns i can use as input for grouping and column changes, and how these columns ought to behave? Thanks a lot!
- lbendlin
The whole point of an incremental refresh is to identify rows that have a dateTime value changed recently, indicating that the row has indeed changed, and must be updated. If this is not possible, what's the point of this incremental refresh feature?What you are looking for is differential refresh, something Power BI is incapable of. Incremental refresh can only go down to partition level for semantic models and bucket/file level for dataflows.
Any changes to past data will require a full or manual refresh of all affected partitions. You would have to create and maintain that process yourself. Unfortunately that is not currently possible with dataflows. LuitwielerMSFT when can we refresh individual buckets?
- useruserhi91
In data semantics, "incremental" refresh has throughout my years always meant to change only updated or new records. This still seems to be the case when googling the term.
Also, from the MS documentation on incremental refresh in Dataflow Gen2, I see:
When you use dataflows for data ingestion and transformation, there are scenarios where you specifically need to refresh only new or updated data—especially as your data continues to grow. The incremental refresh feature addresses this need.."
And:
"Each bucket contains the data that changed since the last refresh. The dataflow knows what changed by checking the maximum value in the column that you specified. If the maximum value changed for that bucket, the dataflow retrieves the whole bucket and replaces the data in the destination"Am I misunderstanding something here - shouldn't it just simply look for updated data, and replace the rows that were updated, and only those rows in my table with new data?