Forum Discussion
Microsoft EmployeePower BI Incremental refresh not removing duplicate records
Hi AnnOminous ,
Thank you lbendlin for your inputs, You're absolutely right Power BI Incremental Refresh is designed for append only scenarios based on an immutable datetime column, such as CreatedDate. It's not intended to handle differential updates or Slowly Changing Dimensions Type 2 (SCD2), where rows may change over time and retain history using LastModifiedDate.
In this case, since duplicate user_ids are being introduced due to updated records appearing in new partitions, the only reliable solution is to reprocess all partitions that may be affected by those changes. This ensures that deduplication logic can be applied across the entire dataset, not just within each partition.
If full reprocessing isn't feasible or efficient, a common workaround is to:
Load all user records including duplicatesinto your Users table, Then use a DAX calculated table or virtual relationship to filter for only the latest record per user_id based on a ModifiedDate or EffectiveDate field. This way, you maintain the integrity of incremental refresh while ensuring only the latest version of each user is used in your model relationships and visuals.
Hope this helps.
Best Regards,
Chaithra E.
Super UserIt's called "Incremental" refresh for a reason. It is neither Differential Refresh nor is it supporting SCD2.
The partitions have to be based on datetime or dateinteger values, and these values need to filter an immutable date field. So "ID created" is good, "ID Last Modified" is bad.
To deduplicate - in your scenario - you would have to reprocess all affected partitions.