Forum Discussion

arkiboys2's avatar
arkiboys2
Helper IV
1 year ago
Solved

microsoft fabric dataflow - delta

Hello, in the current ADF dataflow, I use a delta sink with alter row to upsert if or delete if or insert if is there a similar functionality in fabric dataflow? thank you
data engineering
  • FabianSchut's avatar
    FabianSchut
    1 year ago

    Hi, currently there is no such functionality in DataFlow Gen2. You could use an append to write data incrementally to the destination and write your own Python or SQL-script to handle changes in the data.

jigneshpatel's avatar
jigneshpatel
Regular Visitor
1 year ago

1) I have marked this topic as new as it was updated last a year back. I want to know if there is any soluton for UPSERT
2) also now dataflow has increamental refresh option. Is that an alternative of upsert? 
3) When we use append with additional implementation to have only new or updated rows from source. The updated rows gets appended as new records in the destination. I want it to be updated not inserted? In this thread solution says to have extra logic in python spark script to handle this issue. Can you give more detail on that? 
4) Can we add any script in the dataflow which will search record in destination and update if found or insert if not found?


Jignesh