Forum Discussion
Post ProdigyCalling a dataflow gen2 in a data pipeline in append mode
Hi,
I've implemented a dataflow gen2 to perform some transformations respect to source data and then copying them in the destination.
Moreover, I've created a data pipeline to use this dataflow but I've noticed and tried that when running the pipeline the destination is overwritten.
Now, does a dataflow allow to write data in the destination in append mode and not in overwrite mode, please?
I don't understand because a such option doesn't exist. Thanks
If you turn off the automatic settings,
you can use append function rather than replace in dataflow gnen 2
3 Replies
- NandanHegde
Super User
If you turn off the automatic settings,
you can use append function rather than replace in dataflow gnen 2
NandanHegde In case of Append, if we are using a source like Sharepoint Folder or otherwise and want to bring the incremental changes only, the data is getting duplicated in Lakehouse. Is there a way to get around this.
frithjof_vCommunity Champion
If you want to append only new records, you can achieve that by following the pattern in these docs https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2
If you want to do Upsert, then I think you need to use Notebook. It is shown code examples for Upsert in this tutorial: https://microsoftlearning.github.io/mslearn-fabric/Instructions/Labs/03b-medallion-lakehouse.html
