Forum Discussion

petersmith's avatar
petersmith
New Member
7 years ago

Dataflow incremental refresh with primary key

Hi All,   Loving dataflows, but would love the ability to do an incremental refresh in more of an 'upsert' style. That is, maintain the uniqueness of a primary key within the dataset. Any ideas on ...
Dataflow
Need Help
GilbertQ's avatar
GilbertQ
Super User
7 years ago

Hi there

That would work as either by making the incremental refresh look at the DateTime column of the updated data.

Or you could look at the following below when setting it up

 

GilbertQ's avatar
GilbertQ
Super User
7 years ago

Hi there

 

What you can do is to also enable the option for Detect Changes, where you can update the column based on a date, which will be included in the incremental process

 

  • Anonymous's avatar
    Anonymous
    Not applicable
    7 years ago

    Hi GilbertQ  Thanks. But this does not take care of my scenario. If only depends on Max date in "Changes detected". If my date column is not falling in the rangeof teh incremental refresh cycle , it will still miss the older rows. 

    Let me know if I missed something

    • GilbertQ's avatar
      GilbertQ
      Super User
      7 years ago
      Hi there

      My understanding is that it will also look for changes to the changed column and if it is newer than the last date, it will process that data too?
      • Anonymous's avatar
        Anonymous
        Not applicable
        7 years ago

        Yes it does.. but you will need another date column thats different from the Incremental refresh date.
        Do you have any example or scenario that can help me understand how to use this option?