topic Re: Calling a dataflow gen2 in a data pipeline in append mode in Dataflow

Calling a dataflow gen2 in a data pipeline in append mode

pmscorca — Wed, 03 Jul 2024 05:51:20 GMT

Hi,

I've implemented a dataflow gen2 to perform some transformations respect to source data and then copying them in the destination.

Moreover, I've created a data pipeline to use this dataflow but I've noticed and tried that when running the pipeline the destination is overwritten.

Now, does a dataflow allow to write data in the destination in append mode and not in overwrite mode, please?

I don't understand because a such option doesn't exist. Thanks

Re: Calling a dataflow gen2 in a data pipeline in append mode

NandanHegde — Wed, 03 Jul 2024 08:57:59 GMT

If you turn off the automatic settings,

you can use append function rather than replace in dataflow gnen 2

Re: Calling a dataflow gen2 in a data pipeline in append mode

bhanu_bi — Thu, 04 Jul 2024 20:09:10 GMT

NandanHegde In case of Append, if we are using a source like Sharepoint Folder or otherwise and want to bring the incremental changes only, the data is getting duplicated in Lakehouse. Is there a way to get around this.

Re: Calling a dataflow gen2 in a data pipeline in append mode

frithjof_v — Thu, 04 Jul 2024 20:35:20 GMT

If you want to append only new records, you can achieve that by following the pattern in these docs https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2

If you want to do Upsert, then I think you need to use Notebook. It is shown code examples for Upsert in this tutorial: https://microsoftlearning.github.io/mslearn-fabric/Instructions/Labs/03b-medallion-lakehouse.html