Incremental refresh duplicates

[Anonymous]

Hi,

I am trying to set up the incremental refresh. However, I am having issues with duplicates after publishing the report. I have tried to follow this blog post https://nolock.medium.com/parquet-adls-gen2-etl-and-incremental-refresh-in-one-power-bi-dataset-f426... and use date modified of parquet file. However, it looks like, those modified parquet files, still contain some overlapping values. Is there any recommended approach for that? Thank you.  

[arvindsingh802]

Adding to what @GilbertQ  suggested try using insert date column if you have in your table

when using this, make sure data is not getting truncated and loaded again for any change.

If this post helps, then please consider Accept it as the solution and give it a thumbs up

[GilbertQ · Accepted solution]

Hi @Anonymous 

You have to find a datetime column where there are no duplicate values over the different days.

I would suggest finding a column which has these attributes. Using the modified date will not be a good way to get this done.