Forum Discussion
Data Cleaning - Power Query - Removing Duplicates
Hi everyone, so I have data source that comes from an FTP folder with .csv extension file. It updates everyday with new data. BUT, there were data that has been duplicated coming from previous day that is shown on the current day.
Say today is Monday, some data from previous day (Sunday) is also include on the update. How can I remove the duplicated data and ensure that it will not be duplicated in the future?
Hi ronaldbalza2023 ,
The Folder. Files function (used when connecting to a folder as data source) allows one to see when a file in it was last modified. You can sort the Date created column prior to removing duplicates and wrap that sort step in Table.Buffer.
This datetime column must be included in one of the criteria when removing duplicates. You can remove it afterwards if you don't need it. This tecnique requires that you do not use Combine & Transform Data feature as this doesn't take into account when a file was created.
1 Reply
Hi ronaldbalza2023 ,
The Folder. Files function (used when connecting to a folder as data source) allows one to see when a file in it was last modified. You can sort the Date created column prior to removing duplicates and wrap that sort step in Table.Buffer.
This datetime column must be included in one of the criteria when removing duplicates. You can remove it afterwards if you don't need it. This tecnique requires that you do not use Combine & Transform Data feature as this doesn't take into account when a file was created.