topic Re: microsoft fabric dataflow - delta in Data Engineering

microsoft fabric dataflow - delta

arkiboys2 — Fri, 01 Nov 2024 05:14:59 GMT

Hello,

in the current ADF dataflow, I use a delta sink with alter row to upsert if or delete if or insert if

is there a similar functionality in fabric dataflow?

thank you

Re: microsoft fabric dataflow - delta

FabianSchut — Fri, 01 Nov 2024 06:26:00 GMT

Hi, currently there is no such functionality in DataFlow Gen2. You could use an append to write data incrementally to the destination and write your own Python or SQL-script to handle changes in the data.

Re: microsoft fabric dataflow - delta

arkiboys2 — Wed, 06 Nov 2024 12:00:45 GMT

thank you

Re: microsoft fabric dataflow - delta

jigneshpatel — Thu, 02 Jan 2025 06:23:46 GMT

1) I have marked this topic as new as it was updated last a year back. I want to know if there is any soluton for UPSERT
2) also now dataflow has increamental refresh option. Is that an alternative of upsert?
3) When we use append with additional implementation to have only new or updated rows from source. The updated rows gets appended as new records in the destination. I want it to be updated not inserted? In this thread solution says to have extra logic in python spark script to handle this issue. Can you give more detail on that?
4) Can we add any script in the dataflow which will search record in destination and update if found or insert if not found?

Jignesh