Forum Discussion
Managning refresh of data
I've a query, requesting data from a Rest API. The query got a time out from the server. Since the API supports getting data within a date range, I've spilt the data in three queries and appended them.
This works great. In order to speed up data refresh, I would like only to freshed data from the current month. It seem Power Bi only support refreshing all or none of the data? How can a partial refresh be implemented?
Thx Torben
5 Replies
Hi toafk ,
It sounds like you want something like incremental refresh (IR).
You would need to get a static position for your seed data, but you could then use the API to collect only the most recent data and pass to the Service.
More info on IR:
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
One thing to note - you must ensure that your report/dataset that will be running with IR is in its final production state and can stand the test of time. Once you start the IR process, you will not be able to download or retain the incrementally-built data if you later want to change something.
Pete
Hi Pete,
can you elaborate a little on the "static positon for you seed data". I'm not natural english Speaking, so I need to understand, what you mean. Would it be like bulk loading data, and the use IR from this point?
In that case, how do I create the static position?
I actually have data 5 years back. IR makes a call to the rest api for data for every single date. This is going to take hours to complete the intial load on since it's going to call the API approx 2000 times.
thx
TorbenHi toafk ,
Yes, you've understood correctly.
IR will only add onto what is already in the initial dataset so, if you want all five years of your data, you'll need to essentially bulk pre-load it into your dataset, then start IR running to keep adding on the new months as they happen.
Your dataset query would essentially be: "Give me all data from five years ago until [currentDate]". Then you IR parameters would override this to only refresh data within a shorter time period that you specify.
Pete
