Dataflow incremental refresh duplicates data

[julsr]

Hello everyone!

I have several Dataflows Gen 1 in my workspace. These dataflows contain data from multiple years, and I refresh them daily. However, I am using Incremental Refresh settings to update the data with the following setting: only when a specific date attribute is updated in the dataset, I store the last 20 years of data and refresh only the last 3 days.

The issue is that when the incremental refresh runs, it updates the data, but it causes duplicate values (this does not happen without incremental refresh enabled). Does anyone have an idea of what could be causing this? I need to use this feature as it improves my refresh time.

Thanks!

[v-prasare]

Hi @julsr,

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

@lbendlin , thanks for your prompt response

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support

[lbendlin]

Duplicates are usually caused by using an unsuitable datetime field like Last Modified Date. The partition boundaries must be formed from an immutable datetime or date integer, like Order Created Date.

[julsr]

Thanks. I followed these steps https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview#filter-data to test it on a semantic model and my the date used for filter is the same datetime that changes eachtime the record gets an update on the database which is called updated_time (I got duplicated values too).

What you mentioned is that this date should be, for example, the date when the record was created in the database? If that's the case, how does Power BI know which records need to be updated?

For example, if I use the inserted_date, which is equal to 01/Jan/2020, and the record was updated yesterday (18/Dec/2025), with my policy being the one shown in the image, if I select inserted_date, it won't detect that the record has been refreshed because the data was modified, right? If yes, how can I handle data refreshed in a time before my start date to refresh?

Thanks!

[lbendlin]

 If that's the case, how does Power BI know which records need to be updated?

It's called incremental refresh, not differential refresh. 

Incremental refresh CANNOT update records. The best you can do is flush and fill partitions.

You could use the "Detect data changes"  option but what this will do is create a shadow copy ("canary") of your semantic model, effectively doubling the storage needs.

In essence - it is YOU who needs to know which rows have changed, and who has to decide if and when to refresh the impacted partitions.