Forum Discussion
Incremental refresh - changes in old data
- 6 years ago
Hi Anonymous ,
Based on my understand, if we set store rows for where "extraction date" is in the last 5 years and refresh rows where "extraction date" is in the last 30 days, then the refresh start at 2020-03-14 06:00:00 AM, it will refresh the rows where "extraction date" is between 2020-02-13 06:00:00 AM and 2020-03-14 06:00:00 AM, and all the rows where "extraction date" is between 2015-03-14 06:00:00 AM and between 2020-02-13 06:00:00 AM will keep as the same, and then the rows where "extraction date" is earlier than the 2020-02-13 06:00:00 AM will be dropped.
So In your scenario, if the data in 2018 changed to earlier than 2020-02-13 06:00:00 AM,those rows will not change and keep as old if the refresh range is only set as 30 days. If it change to 2020-03-12 08:00:00, Based on our test, it will keep the old row and add a new rows:
Please refer to following document about more information of Incremental Refresh:
Best regards,
Hi Anonymous ,
Based on my understand, if we set store rows for where "extraction date" is in the last 5 years and refresh rows where "extraction date" is in the last 30 days, then the refresh start at 2020-03-14 06:00:00 AM, it will refresh the rows where "extraction date" is between 2020-02-13 06:00:00 AM and 2020-03-14 06:00:00 AM, and all the rows where "extraction date" is between 2015-03-14 06:00:00 AM and between 2020-02-13 06:00:00 AM will keep as the same, and then the rows where "extraction date" is earlier than the 2020-02-13 06:00:00 AM will be dropped.
So In your scenario, if the data in 2018 changed to earlier than 2020-02-13 06:00:00 AM,those rows will not change and keep as old if the refresh range is only set as 30 days. If it change to 2020-03-12 08:00:00, Based on our test, it will keep the old row and add a new rows:
Please refer to following document about more information of Incremental Refresh:
Best regards,
- Anonymous5 years agoNot applicable
is there a way to prevent duplication of records, and just update the "old" record with the"new" one? without having to refresh on the entire period of 5 years?
- IMett5 years agoHelper III
Does anyone have a proposal how to solve this? We have exactly the same issue.