Hello,
The way my data source is set up is using shared folders. Within the shared folders there are two seperate folders, the first is the historical data which does not get updated and the second folder is new data/changes to hisaorical data. When the new data gets refreshed weekly it loads in duplicate records for rows that have been changed.
1. I am looking for a solution that will remove these duplicate records on an ongoing basis when my data set gets refreshed. The unique idenifier is unique for each amount, there should only be one row with the same unique idenifier and the old one should be exlcuded to avoid duplicates.
2. I am wondering about incremental refresh "Detect data changes". Can I use this feature to allow me to only refresh new and changed data.
Here is an example of a dataset more simple to use if needed. Right now the last changed dates are the same even for the duplicate records since I just did a data extract for the CSV files from the system but going forward the last changed dates will differ when the data gets updated in the new data folder (Since the historical data will never change):
| Unique Idenifier | Amount | Date | Last Changed Date |
| Test1234 | 50 | 02-Jul-20 | 20-Jun-22 |
| Test1234 | 50 | 02-Jul-20 | 20-Jun-22 |
| Test89 | 100 | 30-Aug-21 | 20-Jun-22 |
| Test89 | 150 | 30-Aug-21 | 20-Jun-22 |
