This is best Fabric, Power BI, SQL and AI community event. How do we know? The last event sold out! Save €200 with code FABCMTY200.
Register nowA new Data Days event is coming soon! This time we’re going bigger than ever. Fabric, Power BI, SQL, AI and more. Don't miss out.
Hi everyone, so I have data source that comes from an FTP folder with .csv extension file. It updates everyday with new data. BUT, there were data that has been duplicated coming from previous day that is shown on the current day.
Say today is Monday, some data from previous day (Sunday) is also include on the update. How can I remove the duplicated data and ensure that it will not be duplicated in the future?
Solved! Go to Solution.
Hi @ronaldbalza2023 ,
The Folder. Files function (used when connecting to a folder as data source) allows one to see when a file in it was last modified. You can sort the Date created column prior to removing duplicates and wrap that sort step in Table.Buffer.
This datetime column must be included in one of the criteria when removing duplicates. You can remove it afterwards if you don't need it. This tecnique requires that you do not use Combine & Transform Data feature as this doesn't take into account when a file was created.
Hi @ronaldbalza2023 ,
The Folder. Files function (used when connecting to a folder as data source) allows one to see when a file in it was last modified. You can sort the Date created column prior to removing duplicates and wrap that sort step in Table.Buffer.
This datetime column must be included in one of the criteria when removing duplicates. You can remove it afterwards if you don't need it. This tecnique requires that you do not use Combine & Transform Data feature as this doesn't take into account when a file was created.
Check out the May 2026 Power BI update to learn about new features.
Sign up to receive a private message when registration opens and key events begin.
If you have recently started exploring Fabric, we'd love to hear how it's going. Your feedback can help with product improvements.
| User | Count |
|---|---|
| 27 | |
| 25 | |
| 25 | |
| 21 | |
| 14 |
| User | Count |
|---|---|
| 50 | |
| 44 | |
| 20 | |
| 18 | |
| 18 |