Run from last timestamp & append.

[tanuki]

Hello!

I'm using a Web API as a data source. The API call looks like this:

Source = Json.Document(Web.Contents("https://URLwithvariouscompanyparametersgoeshere?" & "limit=1000" &"since='30-06-2021'" & "timeOrder=asc", [Headers=[Authorization="Bearer MyAPIkeygoeshere"]])),

So it says "fetch me the first 1000 records since 30 June 2021".

The API has a limit of 40000 per call (in the sample code, I've set it to 1000); It does not return any field with the total number of rows/records available, so it doesn't seem possible to set a function along the lines of "number of iterations = records / 40000"

I know nothing of M (but I can handle data transforms quite well). What I would like to do is:

1. Get first batch of results (say, 40000 results from start of data set) and save to a table - call it "all_results"
2. From "all_results" obtain last datetime
3. Rerun API call with "&since='last datetime'" parameter (ie: pull the next bunch of results)
4. Append results to "all_results"

It doesn't need to be a loop - just pull the latest batch of results so that it doesn't need to pull the entirety of the set each time.

Many thanks in advance for any pointers!

[v-kelly-msft]

Hi @tanuki ,

Follow the reference below:

https://kohera.be/power-bi/the-hidden-gems-of-the-function-web-contents/

Best Regards,
Kelly

Did I answer your question? Mark my post as a solution!

[tanuki]

Close, but not quite. The article gives good ideas, but the part that would be of interest to me seems to not have been written yet. 😅