Best Approaches to Avoid Duplicates When Using Copy Activity to Load Data into an Existing Table

Hi HamidBee ,

Here are some of my personal thoughts on your question:

1. In Microsoft Fabric, you can configure the Copy Activity to perform upserts by using the Mapping Data Flow feature. Upsert combines the insert and update operations. It checks if a record already exists in the target table based on a unique key. If it exists, it updates the record; if not, it inserts a new one.

You can look at this document below: 

A guide to Fabric Dataflows for Azure Data Factory Mapping Data Flow users - Microsoft Fabric | Microsoft Learn

2. Implement an incremental load strategy where only new or changed records are loaded. This can be achieved by maintaining a watermark or timestamp column to track the last loaded record.

I think you can take a look about this document:

Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.