Forum Discussion

Thorns's avatar
Thorns
Frequent Visitor
9 months ago

Dataflow Incremental Refresh Causing Duplicate Records

I created a Gen2 Dataflow the sources its data from a Salesforce object and loaded to my warehouse.  I then setup incremental refresh to review everything created in the past 3 years with a bucket si...
data engineering
Thorns's avatar
Thorns
Frequent Visitor
9 months ago

Thanks for the information!  I will stop using incremental refresh.

 

Where in the Gen2 Dataflow does it allow for upsert?  I do not see that as an option.

v-ssriganesh's avatar
v-ssriganesh
Community Support
9 months ago

Hello Thorns,

Thanks for the update. glad you're pausing incremental refresh to avoid more duplicates.

I apologize for the confusion in my earlier suggestion upon double-checking the Dataflow Gen2 documentation, "upsert" isn't directly available as an update method for any destination, including Warehouse. For Warehouse specifically, the only supported option is replace, with no append or upsert in the UI.

To achieve an incremental upsert-like behavior (adding new records and updating changed ones without duplicates):

  • Switch your destination to a Fabric Lakehouse table (which supports both append and replace). This allows more flexibility