Forum Discussion
Dataflow incremental refresh duplicates data
If that's the case, how does Power BI know which records need to be updated?It's called incremental refresh, not differential refresh.
Incremental refresh CANNOT update records. The best you can do is flush and fill partitions.
You could use the "Detect data changes" option but what this will do is create a shadow copy ("canary") of your semantic model, effectively doubling the storage needs.
In essence - it is YOU who needs to know which rows have changed, and who has to decide if and when to refresh the impacted partitions.
Duplicates are usually caused by using an unsuitable datetime field like Last Modified Date. The partition boundaries must be formed from an immutable datetime or date integer, like Order Created Date.
Thanks. I followed these steps https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview#filter-data to test it on a semantic model and my the date used for filter is the same datetime that changes eachtime the record gets an update on the database which is called updated_time (I got duplicated values too).
What you mentioned is that this date should be, for example, the date when the record was created in the database? If that's the case, how does Power BI know which records need to be updated?
For example, if I use the inserted_date, which is equal to 01/Jan/2020, and the record was updated yesterday (18/Dec/2025), with my policy being the one shown in the image, if I select inserted_date, it won't detect that the record has been refreshed because the data was modified, right? If yes, how can I handle data refreshed in a time before my start date to refresh?
Thanks!If that's the case, how does Power BI know which records need to be updated?It's called incremental refresh, not differential refresh.
Incremental refresh CANNOT update records. The best you can do is flush and fill partitions.
You could use the "Detect data changes" option but what this will do is create a shadow copy ("canary") of your semantic model, effectively doubling the storage needs.
In essence - it is YOU who needs to know which rows have changed, and who has to decide if and when to refresh the impacted partitions.
