Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Abishek
Helper I
Helper I

How to handle duplicate in incremental refresh

HI,

New data coming through incremental refresh was creating duplicates. because of this my relationship fails and incremental refresh did not do well.

I only need to keep the data coming form the current refresh and delete the old data coming form incremental refresh.

I have tried multiple approach to solve this problem. FInally ended up in thinking of sorting the table in descending of the date column and removing duplicates ID’s will make the refresh smooth and faster. But I could see this causing a query folding issue.

is this a correct way to approach this problem, or is there any other way to do this.

you guidance will be much useful for me.

Thanks.
Abishek R.

 

1 REPLY 1
amitchandak
Super User
Super User

@Abishek , if you need only new data, then do not do incremental setting, it will be replaced with new data.

 

Incremental is only needed if you need to delete and append.

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.