Forum Discussion
Overwrite not working in Copy Data?
Hi alloowishus , Your expectation is understandable, but in Fabric Copy activity the write modes don’t manage existing data, they only control how the incoming batch is written. So once duplicates were created with Append, switching to Overwrite won’t clean up what’s already there because there’s no step that reconciles or removes prior rows.
If you want the table to always reset back to exactly what’s in the source, you need to make that explicit in the pipeline by clearing the target before the load, for example, a SQL/Notebook step to delete or truncate the table, then run the copy. That gives you a true full refresh every time.
If instead you want to prevent duplicates going forward without wiping the table, you should switch to an Upsert pattern with a defined key. That way incoming rows update existing ones instead of being added again, which is the only way to make the load idempotent.
Hi alloowishus , Thank you for reaching out to the Microsoft Community Forum.
In Fabric Data Factory, Overwrite in a Copy activity does not mean drop and recreate the table. It means replace the data being written in that operation, which for Lakehouse/Delta tables is typically file/partition level overwrite. So, when you switched from Append to Overwrite, it only overwrote the incoming write scope and did not remove the previously appended duplicate rows. That’s why the duplicates remained until you manually deleted the table.
If you need a clean reload every time, you should explicitly clear the table first, don’t rely on Overwrite to reset the entire table.