Does data cleaning automatically happen monthly?

[kowas2]

Hi! I'm doing up a report on employee termination. Every month we get a new updated excel file of terminated employees, but this file is just the previous month's list with new names appended

E.g.

JANUARY

Winnie

Robin

Bruce

FEBRUARY

Winnie

Robin

Bruce 

Kathryn

So what I did was upload all existing files into a folder (which i set as the source for PBI)

And i used the editor (query?) to combine them all and then remove duplicate rows.

My question is: when i drop fresh files into the source folder, will the duplicate rows be automatically removed? Or do i have to remove duplicate rows manually again. Do i have to press something?

The column names will be exactly the same.

I'm sorry for the stupid question! I'm a complete beginner to pbi and wasn't sure what keywords to google. thanks for any help!!

[danextian · Accepted solution]

hi @kowas2 

Since you steps involve combining worksheets then removing the duplicates, Power BI will remove the duplicates when you refresh the semantic model. Just be aware though that that Power BI will keep the first row it sees for each distinct value which may not be necessarily ordered by month.

[Olufemi7 · Accepted solution]

Hello @kowas2,

In Microsoft Power BI, your data cleaning steps (like Remove Duplicates) are saved in Power Query and only run when you refresh.

• Add new file → nothing happens yet
• Click Refresh (or use scheduled refresh)
• Power BI recombines files and removes duplicates again

You don’t need to remove duplicates manually each time.

If you want it monthly without action:

• Set up Scheduled Refresh in the Service

Docs:
Working with duplicate values 
Data refresh in Power BI 

[cengizhanarslan]

Yes, the duplicate removal happens automatically on every refresh. The steps you built in Power Query — combining the folder files and removing duplicates — are saved as a reusable recipe that runs every time the data refreshes. You never have to redo them manually.

[danextian · Accepted solution]

hi @kowas2 

Since you steps involve combining worksheets then removing the duplicates, Power BI will remove the duplicates when you refresh the semantic model. Just be aware though that that Power BI will keep the first row it sees for each distinct value which may not be necessarily ordered by month.

[grazitti_sapna]

Hi @kowas2,

Yes, Duplicate rows will get deleted automatically, in transfrom whatever stepe you perform are saved as applied steps which you can check on right side in Trasform data window. 

🌟 I hope this solution helps you unlock your Power BI potential! If you found it helpful, click 'Mark as Solution' to guide others toward the answers they need.
💡 Love the effort? Drop the kudos! Your appreciation fuels community spirit and innovation.
🎖 As a proud SuperUser and Microsoft Partner, we’re here to empower your data journey and the Power BI Community at large.
🔗 Curious to explore more? [Discover here].
Let’s keep building smarter solutions together!

[johnt75]

You need to set up a scheduled refresh for your dataset. That will then look for all the files in the folder and remove the duplicate rows.