Forum Discussion

Mohan128256's avatar
Mohan128256
Helper IV
1 year ago

Incremental Refresh Data duplication, data mismatch

Hello All,

 

I have a table which has around 1.2B of records.

 

I have setup incremtnal load. as 3 yrs archive and 1 day as incremental refresh.

 

but the problem is, though the incremental refresh is getting done, i see some of the days data is getting duplicated or less than the actual source data.

 

I did download the power bi service dataset data for recent days and exported. Checked with actual source data. for the same dates. I found there is huge difference between the values as you see in above image.

 

Can anyone please suggest or guide on WHAT COULD BE A POSSIBLILITY TO HAVE DATA MISMATCH OR DATA DUPLICATE BETWEEN INCREMENTAL REFRESHED DATASET to ACTUAL SOURCE?

 

Thanks,

Mohan V.

Data Protection
General Comment
Interesting Links
Need Help
Show and Tell
Tips and Tricks
Tutorial Requests

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi Mohan128256 

     

    Thanks for the reply from parry2k , please allow me to make some additions.

     

    If both the RangeStart and RangeEnd parameters in a query have equal signs, some rows may meet the conditions of both partitions, resulting in duplicate data in the model.

     

    https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-configure#filter-data 

     

    If our replies is helpful to you, please consider marking our replies as solution. This will help more users who are facing the same or similar difficulties. Thank you!

     

    Best Regards,
    Yulia Xu