Forum Discussion
Incremental refresh and duplicate records
Hello everybody,
I've recently started exploring the possibilities of the incremental refresh features to see if I can apply it to my reports.
The situation is as follows: My data source is a Web.Contents REST API. I've set it up so that I get the records that have a 'changed' date set between my the RangStart and RangeEnd parameters. This is the only datetime column I have available.
The problem I'm having is the fact that certain records can be modified multple times between different date ranges and because that my data source contains duplicate records. The only thing that doesn't change the every records primary key.
Now I know Incremental refresh isn't designed to support cases where the filtered date column is updated in the source system and that an update is interpreted as an insertion and a deletion but isn't there some workaround that can be applied?
I've tried adding a Table.Distinct(#"Sort", {"ID"}) tot the query but it seems that this is evaulated over the subsets of data that is recieved with every incremental refresh and not over the dataset as a whole.
Below are my incrementel setup settings. And all the query transformations that happen after the data fetched.
2 Replies
I don't think I can. I only have the 'changed' column to work with, which either tells me when the record was created or last updated.
Somehow I doubt I can avoid having these duplicate records in my dataset. What I need is the most recent row for each 'ID' colum, perhaps I can do something in my M query to realise this?
