Forum Discussion
Copy data activity not overwriting
We have also noted some odd behaviour with the Copy Data acivity.
It appears to have started from the 15th August for us. Where data is not being overwritten even though the overwrite option is set.
We managed to solve this issue with help from Mictosoft Support. The overwrite option when selected in a copy data activity has been updated to carry out OverwriteSchema.
"OverwriteSchema" action does not delete any data. Instead, it creates a new version of the Delta table with the updated schema, allowing users to leverage the time travel feature to view all previous versions of the table.
To perform an Overwrite, where the current data is completely replaced with the newly copied data, select 'Add dynamic content' for the Table Action and enter 'Overwrite'.
Thanks for the update. Do you know if there is any documentation on the change? I've been looking here https://learn.microsoft.com/en-us/fabric/data-factory/copy-data-activity but couldn't find anything.
Hi,
Same issue this side, which is clearly that overwrite now appends instead of its intended functionality. Not a user-side issue.
Must say, working with Fabric has been challenging at times...
After spending 2 hours trying to understand why my ID has duplicates, I found out that records are duplicated in my LAKEHOUSE copy pipeline which I used the copy assitant to generate. Then I tried to use both append and overwrite to see if that fixes the issues, and it didn't.
Thank you for providing this workaround for what is clearly a bug! I wonder how anyone would be able to know this!
I have tried it, and I am not sure why, but after adding overwrite to dynamic content, it threw an error.
I ended up creating a notebook to purge tables. Probably that is the way that it was intended to happen (still the question on why Overwrite option exits in data pipeline copy process.
