Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Be one of the first to start using Fabric Databases. View on-demand sessions with database experts and the Microsoft product team to learn just how easy it is to get started. Watch now

Reply
Anonymous
Not applicable

Incremental Refresh with Dataflow is duplicating

I have a dataflow that I'm trying to setup incremental refresh. I have set it up but no the dataflow has some rows that have duplicate of my primary key from the base fact table. I can see the 2 rows for the same primary key number in the dataflow using power query. So my question is how is this happening?

my refresh is set up like

PowerBIRookie_0-1656090956768.png

 

The UPDATED column is a datetime column in the fact table that changes every time the record is updated. 

 

My Goal is to load 2 years of data (fact more then that... like 10 years... but I just want a rolling 2 years) and I want to reload or refresh records in the dataflow that have been modified within the last 5 days or so. But if the record exists already in the dataflow the modified record should overwrite the existing record that has the same pk. How do I do that? Cause currently is loads the original record then if that record was modified it loads that one too as a second record in the dataflow.

2 REPLIES 2
otravers
Community Champion
Community Champion

Make sure that one of your date filters doesn't have the = sign, i.e. use >RangeStart and <= RangeEnd or >=RangeStart and < RangeEnd. Here's a good entry to guide you:

http://biinsight.com/implementing-incremental-refresh-in-power-bi-part-1/

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals
Anonymous
Not applicable

Yes I made sure about that.. I read that article here is the filter 

Table.SelectRows(#"Removed columns", each DateTime.From([UPDATED]) >= RangeStart and DateTime.From([UPDATED]) < RangeEnd)
 
but this only applies to the desktop not the service... so I'm checking the dataflow using power query to see the duplicated row with the primary key (first column)
PowerBIRookie_0-1656093057466.png

 



Helpful resources

Announcements
ArunFabCon

Microsoft Fabric Community Conference 2025

Arun Ulag shares exciting details about the Microsoft Fabric Conference 2025, which will be held in Las Vegas, NV.

December 2024

A Year in Review - December 2024

Find out what content was popular in the Fabric community during 2024.