Forum Discussion
Managning refresh of data
Hi toafk ,
Yes, you've understood correctly.
IR will only add onto what is already in the initial dataset so, if you want all five years of your data, you'll need to essentially bulk pre-load it into your dataset, then start IR running to keep adding on the new months as they happen.
Your dataset query would essentially be: "Give me all data from five years ago until [currentDate]". Then you IR parameters would override this to only refresh data within a shorter time period that you specify.
Pete
Thanks again Pete,
what I done in a different case.
1) Created af Parameter StartDate
2) Build a list dates from StartDate till Today
3) Build the query, which calls the Rest Api one time for each date.
4. Setup Incremental Refresh, so in only refreshes data for the last 21 days (data can be change within 3 weeks), when data is refreshed the second time.
Still step 3) takes very long. Is this the best way to do it?
Hi toafk ,
I think this is the best way to do it as I assume this will give you some error/failure handling i.e. if one day request fails, it can be re-run, but if you try and make a larger single API call, it will be much harder/take longer to re-run that inividual call to fix.
Overall, I think you need to weigh up the time taken to implement this against the time saved in the future once it's set up, and you have a cloud-based dataset that holds all of your data with little-to-no maintenance.
Pete