Dataflow Incremental Refresh Causing Duplicate Records

Hello Thorns,
Thank you for reaching out to the Microsoft fabric community forum.

The duplicate records in your warehouse table are likely due to the incremental refresh feature in Dataflow Gen2 not supporting query folding for Salesforce objects. This causes the full dataset to be pulled per bucket refresh, leading to improper replacement and duplicates, especially for modified records.

To fix this:

1. Disable incremental refresh to stop further duplicates.
2. Manually implement incremental load:

• Add a reference query to get the max LastModifiedDate from your warehouse table.
• Filter Salesforce data where LastModifiedDate > that value in your main query.
• Use "upsert" (with Id as primary key) or "append" as the destination update method.

1. If handling deletes, include the IsDeleted field and add logic to remove corresponding rows and Test with a manual refresh, then schedule.

For existing duplicates, run a one-time deduplication before applying the new setup. This should resolve the issue effectively.

Best regards,
Ganesh Singamshetty.