Forum Discussion
Microsoft EmployeePower BI Incremental refresh not removing duplicate records
Hi AnnOminous ,
Thank you lbendlin for your inputs, You're absolutely right Power BI Incremental Refresh is designed for append only scenarios based on an immutable datetime column, such as CreatedDate. It's not intended to handle differential updates or Slowly Changing Dimensions Type 2 (SCD2), where rows may change over time and retain history using LastModifiedDate.
In this case, since duplicate user_ids are being introduced due to updated records appearing in new partitions, the only reliable solution is to reprocess all partitions that may be affected by those changes. This ensures that deduplication logic can be applied across the entire dataset, not just within each partition.
If full reprocessing isn't feasible or efficient, a common workaround is to:
Load all user records including duplicatesinto your Users table, Then use a DAX calculated table or virtual relationship to filter for only the latest record per user_id based on a ModifiedDate or EffectiveDate field. This way, you maintain the integrity of incremental refresh while ensuring only the latest version of each user is used in your model relationships and visuals.
Hope this helps.
Best Regards,
Chaithra E.
Community SupportHi AnnOminous ,
Yes, reprocessing every partition is effectively the same as performing a full refresh of the table, ensuring that your deduplication logic is applied across the entire dataset, not just within individual partitions.
If you're using Power BI Premium or Premium Per User (PPU) and have XMLA endpoints enabled, you can manage this through SQL Server Management Studio (SSMS) without needing to republish the dataset. In SSMS, you can connect to your Power BI workspace, navigate to the relevant table, and use the "Process Full" option to reprocess all partitions. This will reload the data for the entire table and allow your Power Query steps including deduplication to be applied globally.
So while the logic for deduplication itself must still exist in Power Query or be handled via DAX, SSMS can be used to trigger the full reprocessing when needed.
Thanks,
Chaithra E.
Microsoft Employeev-echaithra, thank you again - this is all very helpful information.