Retrieving data by working around REST API limitations through dynamic web query

[ek2112]

Hi all,

I have a data source with timelog entries, with about 30,000 records. Until now, I was able to get the data out of the data source through direct authentication that fetched all the records through a single query that returns values as JSON (Get data -> From web option). 

Now, the data source is changing their API and limiting the records fetched through a single REST call to 1000 and they are rolling this out starting 1st week of June.

I can get all the records and load them into Power BI through, say, 6/4 using the existing authentication. My question (or rather questions) as follows:

a) Going forward, once the authentication changes with the aforementioned limiting effect of fetching only 1000 records at a time, how do I dynamically modify the source url to get worklogs on a weekly basis, say from 6/4 to 6/10 and append them to the existing data?

b) How to do this on a weekly basis?

Is there a way to dynamically update the Web.Contents() URI and append the resultset to the table? Or through parameters or both? Thank you for your time.

[v-chuncz-msft · Accepted solution]

@ek2112,

You may take a look at the post below.

https://community.powerbi.com/t5/Desktop/Stopping-when-looping-through-a-paginated-API-feed/m-p/3103...

[ek2112]

@v-chuncz-msft This got me thinking...and helped. Thanks.