Forum Discussion
Post ProdigyCalling a dataflow gen2 in a data pipeline in append mode
If you turn off the automatic settings,
you can use append function rather than replace in dataflow gnen 2
Super UserIf you turn off the automatic settings,
you can use append function rather than replace in dataflow gnen 2
NandanHegde In case of Append, if we are using a source like Sharepoint Folder or otherwise and want to bring the incremental changes only, the data is getting duplicated in Lakehouse. Is there a way to get around this.
frithjof_vCommunity Champion
If you want to append only new records, you can achieve that by following the pattern in these docs https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2
If you want to do Upsert, then I think you need to use Notebook. It is shown code examples for Upsert in this tutorial: https://microsoftlearning.github.io/mslearn-fabric/Instructions/Labs/03b-medallion-lakehouse.html
