Forum Discussion
Copy Job OVERWRITE
I'm playing around with the Copy Job, pulling data from a SQL enabled source into a Fabric Warehouse. Ideally I would like to have the option to OVERWRITE the data. But that option only exists when you choose a LAKEHOUSE destination.
Referencing this article: What is Copy job in Data Factory - Microsoft Fabric | Microsoft Learn
and this statement: By default, Copy job appends new data, so you keep a full history. If you prefer, you can choose to merge (update existing rows using a key column) or overwrite (replace existing data).
There is no mention that this only applies to LAKEHOUSE destinations. With Warehouses, you are stuck with APPEND, and no options to select anything else, nor does it indicate that it is OVERWRITE.
Here I was thinking my 17M row table was getting OVERWRITTEN every run when in fact I ended up with 100M rows!
This object has a long ways to go, IMHO.
Thoughts? Objections? Comments? Bad jokes?
Hi ToddChitt,
Microsoft got back to me via teams and this is by design.
Poor design, but design nonetheless.
I have voted for your idea: Copy Job needs more functionality - Microsoft Fabric Community
to have this added.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
4 Replies
Hi ToddChitt,
This is an interesting find, I'm pretty sure I have a copy job that is overwriting data in a warehouse, not just appending data.... I hope..... I will look at this tomorrow morning.
If this is the case that it only appends, this is a pretty major lack in featureset here.
One workaround that immediately comes to mind is to use a pipeline, first have a script task to truncate the table, then run the copy job from the pipeline - though at this point you might as well just use a copy activity in the pipeline and skip the copy job all together.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi ToddChitt,
I'm joining your rant officially now. I see the same behaviour you describe.
Despite being configured as a full copy, it appends rows instead of truncating them.
CopyJobs need to be punted back to preview so they can be fixed.
I'm raising this as a bug in a ticket to Microsoft.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
tayloramy Go for it! I will let you take the lead on this one. Please include me in the correspondence, if you can: [email protected].
Much appreciated.
Hi ToddChitt,
Microsoft got back to me via teams and this is by design.
Poor design, but design nonetheless.
I have voted for your idea: Copy Job needs more functionality - Microsoft Fabric Community
to have this added.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
