Forum Discussion

santhiya's avatar
santhiya
Helper I
3 years ago

Incremental Refresh creating Duplicate Records

Hi community!!!
I am using the following method to view reports in Power BI 

Dataflow -> Dataset -> Report 

I used import mode and I have configured Incremental Refresh for My dataflow and dataset. In the configuration I have selected the following 

store last 10 years 

Refresh last 2 months . I am filtering based on updated_at column , if the updated_at column is in the last 2 months there is no problem but when it is in the last 10 years two records will be loaded into my model . I can remove this duplicates with Sort based on updated_at and remove duplicate but what it does is break my query folding which indirectly breaks the logic of Incremental refresh as well. Instead of Loading only the last 2 months entire period will be loaded and filter will be applied . But how to avoid this , whether I need to avoid the duplicates by configuring the Incremental refresh in some other way or removing duplicates without breaking query folding or some other suggestions please?


 

Thanks

 

1 Reply

  • AntrikshSharma's avatar
    AntrikshSharma
    Community Champion

    santhiya Generally duplicates are introduced in Incremental Refresh when in the both Start and End criteria you specify the "=" so you need to remove it from either one of them.