Forum Discussion
Overwrite not working in Copy Data?
- 4 months ago
Hi alloowishus , Your expectation is understandable, but in Fabric Copy activity the write modes don’t manage existing data, they only control how the incoming batch is written. So once duplicates were created with Append, switching to Overwrite won’t clean up what’s already there because there’s no step that reconciles or removes prior rows.
If you want the table to always reset back to exactly what’s in the source, you need to make that explicit in the pipeline by clearing the target before the load, for example, a SQL/Notebook step to delete or truncate the table, then run the copy. That gives you a true full refresh every time.
If instead you want to prevent duplicates going forward without wiping the table, you should switch to an Upsert pattern with a defined key. That way incoming rows update existing ones instead of being added again, which is the only way to make the load idempotent.
Well then what is the difference between overwrite and upsert? Having to truncate all these tables before hand is something I would expect to be baked in, as it is a very common function when doing ETLs. When I googled the difference between Overwite and Upsert, this is what I get (which I guess is wrong?)
Hi,
The definitions stated above for Overwrite and Upsert is absolutely correct. However, lets try to analyze more for your situation. As per MSFT, the available connectors are as below:
Can you let me know what is your sink for Overwrite? Is it a lakehouse or warehouse table.
Also, for the solution, there are workarounds for the bug in Overwrite function in copy activity.
1. We can remove tableActionOption= 'OverwriteSchema' to 'Overwrite' and it works.
microsoft fabric - Azure Data Factory Copy Activity appends table in sink - Stack Overflow
2. Can refer the previous community post for the same idea which was put across for copy activity issue. Solved: Re: Copy Job OVERWRITE - Microsoft Fabric Community
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Thanks
Ati Puri