Forum Discussion
Overwrite not working in Copy Data?
Hi alloowishus , Your expectation is understandable, but in Fabric Copy activity the write modes don’t manage existing data, they only control how the incoming batch is written. So once duplicates were created with Append, switching to Overwrite won’t clean up what’s already there because there’s no step that reconciles or removes prior rows.
If you want the table to always reset back to exactly what’s in the source, you need to make that explicit in the pipeline by clearing the target before the load, for example, a SQL/Notebook step to delete or truncate the table, then run the copy. That gives you a true full refresh every time.
If instead you want to prevent duplicates going forward without wiping the table, you should switch to an Upsert pattern with a defined key. That way incoming rows update existing ones instead of being added again, which is the only way to make the load idempotent.
Hi Everyone,
When using Overwrite mode in Copy Activity for a Lakehouse Delta table, Fabric doesn't simply truncate and reload, it issues a DELETE on the existing partition or full table before writing. If the schema or partition hasn't changed, the Overwrite sometimes behaves like Append on the first run after switching modes.
Try these steps:
- In the Copy Activity sink settings, set Write behavior = Overwrite
- Add a Delete activity before the Copy Activity targeting the same table (TRUNCATE TABLE via a Warehouse, or a Notebook step with spark.sql("DROP TABLE IF EXISTS...") + recreate)
- Alternatively, use a Notebook with df.write.format("delta").mode("overwrite").save(...) for full control
If you're targeting a specific partition, make sure partitionBy is consistent mixing partition schemas between runs can cause Overwrite to only affect the matched partition.