Forum Discussion
Copy Job does an Append for FULL Load, cannot be changed
I played around with Copy Job when it first came out. Now I am trying to set one up for real. I'm starting with one table of about a million records that needs a Full load every night. The job has run three times and I have 3 million records. It is configured for FULL Load. In the Advanced settings, under Write settings, it is set to Append. Ah, THAT is the issue. But I cannot select anything else. Where is the OVERWRITE option? What am I missing? Do I have to manually TRUNCATE the destination tables before running the Copy Job?
I sort of assumed that a FULL LOAD implied overwrte.
Hi ToddChitt,
I think when you first create the copy job you need to click on the advanced settings and change it from append to overwrite. As far as I know there's no way to change this after the copy job has been created.
I've always used a script task in a pipeline to truncate the table first, then run the copy job. I've found that to be more reliable.
3 Replies
Hi ToddChitt,
I think when you first create the copy job you need to click on the advanced settings and change it from append to overwrite. As far as I know there's no way to change this after the copy job has been created.
I've always used a script task in a pipeline to truncate the table first, then run the copy job. I've found that to be more reliable.Turns out, there is NO option to OVERWRITE when using a Copy Job that A) uses FULL Load and B) targets a Fabric Warehouse. (Missing functionality, IMHO). But if you select Incremental Load, there is an otpion to Reset that turns the NEXT run into a Full Load.
Microsoft could not have made this any more difficult to grasp from a user perspective. You can do an OVERWRITE if you send to a Lakehouse, but NOT a Warehouse.
Readers: Want better and more functionality from Copy Job? Vote on this item: Copy Job needs more functionality - Microsoft Fabric Community
Hi ToddChitt,
Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to tayloramy for sharing valuable insights.
Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.
Thank you for being part of the Microsoft Fabric Community.
