How to automatically clean up Duplicate Data on a data Refresh

[dd88]

I have setup and cleaned up the data, using these steps

https://www.youtube.com/watch?v=QV98Jndj5Fo

Fill up/ down with conditions in Power Query - YouTube

I also have removed  duplicate data

Select Transform Data, select column, select Remove Duplicates

I have a new dataset. On the Home ribbon, I select Refresh to import the data.  The results are duplicate data displays.

My questions are:

How can I automatically clean up Duplicate Data on a Refresh

To avoid manually removing dupicates each time a new data version exists.

Or, is this something I need to do and manually prepare the data, on every new data set,

TIA

[Idrissshatila · Accepted solution]

Hello,

yes for every new Dataset you should do the cleaning work onpower query and then as you refresh the data the steps you did will perform. But you should do the steps for every new dataset.

If I answered your question, please mark my post as solution so it would appeare to others, Appreciate your Kudos 👍

Follow me on Linkedin