Incremental Refresh on Dataflow Gen 2 creating duplicate records

Hi craigmday, 
Thank you for reaching out to the Microsoft Fabric Community Forum. 

The duplicates occur because Dataflow Gen2 incremental refresh does not perform row-level deduplication or upsert based on primary/composite keys. When the incremental refresh runs, it reprocesses the defined time window (last 3 days) and writes those records again to the Lakehouse. Since Lakehouse tables do not enforce primary keys, the same rows can be inserted again. In your setup the filter column is Transaction Date, but change detection uses Journal Date. Using different columns can cause the same incremental partition to be processed again, which results in duplicate rows.

To resolve this, configure the incremental refresh to use the same column for both filtering and change detection (e.g., Transaction Date) and ensure the incremental filter step supports query folding and is applied early in Power Query. If duplicates can still occur, implement deduplication using the composite key columns in the Dataflow or use a Lakehouse MERGE (upsert) process with a staging table so existing rows are updated instead of inserted again. This ensures the incremental refresh only loads new data and prevents duplicate records.

Hope this clarifies. Let us know if you have any doubts regarding this. Please feel free to reach out to us. We will be happy to help.

Thank you for using the Microsoft Fabric Community Forum.