Forum Discussion
Updating
Dataflows only offers two data destination operations:
- Append
- Replace
In that sense, it doesn't support any sort of delete operations or "Upserts" to update existing records.
If your destination was a SQL Database, you could perhaps leverage a combination of a Data Pipeline, a Dataflow Gen2 and a SQL Stored procedure to perhaps load the data inside of a "staging table" in your SQL Database and then run a stored procedure that will take that table and figure out, based on your own logic, what records to delete, update or add to your main table.
I'm not sure how feasible that is with a Lakehouse as a destination, but you could try asking more questions specific to Lakehouse in the Lakehouse community forum (link below):
Hi elavery95 ,
Thanks for using Fabric Community.
Can you please check this - Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
I suggest you incremental refresh rather than maintaining 2 tables.
FYI: We will have more features in upcoming releases.
Docs to refer - What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Please let me know incase of further queries.
Hi elavery95 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help .
Hi elavery95 ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Thanks