Forum Discussion

Dayna's avatar
Dayna
Helper V
4 years ago

Incremental refresh causing duplicate rows

Hello,   I've implemented incremental refreshes on one of our fact tables, called 'Transactions'.   I can see on a given day there's been a change to some rows, and when this dataflow is consumed...
Need Help
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi I have a similar issue 

we have last updated date field across which we are enabling incremental refresh 

I followed as per MS documentation 

currently we enabled refresh for last 15 days.

so what's happening is it is updating and refreshing the data for these 15 days window. 
now problem here is if I have a record which last updated 4 months back and is updated today again, it is not removing the 4 months back row and is getting added resulting in duplicates 

how can we adress this issue ??

  • lbendlin's avatar
    lbendlin
    Super User
    2 years ago

     

    That field is not suitable for incremental refresh. Use an immutable field like Create Date. You can theoretically use the Last Modified Date for change tracking but ideally you should know your data and should know which partitions to refresh when.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      The moment I use Created Date for incremental refresh, when i publish to service, the frist refresh goes on for hours and gets timed out and fails. I tried that couple of times. 

      • lbendlin's avatar
        lbendlin
        Super User
        2 years ago

        Read about bootstrapping.  You can prepare the partitions without filling them.  Then fill them one by one.