Forum Discussion
Incremental refresh duplicates
Hi,
I am trying to set up the incremental refresh. However, I am having issues with duplicates after publishing the report. I have tried to follow this blog post https://nolock.medium.com/parquet-adls-gen2-etl-and-incremental-refresh-in-one-power-bi-dataset-f4269ae6b2a0 and use date modified of parquet file. However, it looks like, those modified parquet files, still contain some overlapping values. Is there any recommended approach for that? Thank you.
Hi Anonymous
You have to find a datetime column where there are no duplicate values over the different days.
I would suggest finding a column which has these attributes. Using the modified date will not be a good way to get this done.
2 Replies
Hi Anonymous
You have to find a datetime column where there are no duplicate values over the different days.
I would suggest finding a column which has these attributes. Using the modified date will not be a good way to get this done.
Adding to what GilbertQ suggested try using insert date column if you have in your table
when using this, make sure data is not getting truncated and loaded again for any change.
If this post helps, then please consider Accept it as the solution and give it a thumbs up
