Forum Discussion
Dataflow Incremental Refresh Duplicates
I have set up incremental refresh on a flow as follows:
I am pulling from a SQL database table and it's last update date as above.
I have pulled the table into my dataset and I am getting duplicate keys. I'm not sure what I'm missing? Why am I getting duplicates?
1 Reply
Hi. That might be because the "Detect data changes". As the name says this is incremental. That means that if you modify a row from a year ago but you are only specifing the refresh should be from last day, then you will have last day + changes from history. The changes are a new row that looks almost the same as the one before but with the change.
The detect data changes is NOT recommended unless you know how to handle it as data perspective. You usually use a change or modify date and things like that.
Incremental refresh cannot update rows from old history, it can add new rows with the change. If your warehouse or source is build for that changes you can add columns like "last update" flag to filter it in the power query.
I hope that helps,
