Forum Discussion
Helper IRecommended Incremental refresh settings for BC365 fact tables (like GL Entry)?
Helper IThanks for your reply. However, of course "systemModifiedAt" is NOT immutable - it changes. The whole point of an incremental refresh is to identify rows that have a dateTime value changed recently, indicating that the row has indeed changed, and must be updated. If this is not possible, what's the point of this incremental refresh feature? Can you give a concrete example of which columns i can use as input for grouping and column changes, and how these columns ought to behave? Thanks a lot!
Super UserThe whole point of an incremental refresh is to identify rows that have a dateTime value changed recently, indicating that the row has indeed changed, and must be updated. If this is not possible, what's the point of this incremental refresh feature?
What you are looking for is differential refresh, something Power BI is incapable of. Incremental refresh can only go down to partition level for semantic models and bucket/file level for dataflows.
Any changes to past data will require a full or manual refresh of all affected partitions. You would have to create and maintain that process yourself. Unfortunately that is not currently possible with dataflows. LuitwielerMSFT when can we refresh individual buckets?
- useruserhi91
In data semantics, "incremental" refresh has throughout my years always meant to change only updated or new records. This still seems to be the case when googling the term.
Also, from the MS documentation on incremental refresh in Dataflow Gen2, I see:
When you use dataflows for data ingestion and transformation, there are scenarios where you specifically need to refresh only new or updated data—especially as your data continues to grow. The incremental refresh feature addresses this need.."
And:
"Each bucket contains the data that changed since the last refresh. The dataflow knows what changed by checking the maximum value in the column that you specified. If the maximum value changed for that bucket, the dataflow retrieves the whole bucket and replaces the data in the destination"Am I misunderstanding something here - shouldn't it just simply look for updated data, and replace the rows that were updated, and only those rows in my table with new data?
- lbendlin
Yes, this is a misunderstanding. The lowest you can go is to replace the bucket or partition. You cannot replace individual rows.
- useruserhi91
Well that is interesting. Thanks for clarifying, this was not clear to me from the documentation. I found the wording of incremental refresh misleading. What is the use case for this feature then?