Run from last timestamp & append.

Hello!

I'm using a Web API as a data source. The API call looks like this:

Source = Json.Document(Web.Contents("https://URLwithvariouscompanyparametersgoeshere?" & "limit=1000" &"since='30-06-2021'" & "timeOrder=asc", [Headers=[Authorization="Bearer MyAPIkeygoeshere"]])),

So it says "fetch me the first 1000 records since 30 June 2021".

The API has a limit of 40000 per call (in the sample code, I've set it to 1000); It does not return any field with the total number of rows/records available, so it doesn't seem possible to set a function along the lines of "number of iterations = records / 40000"

I know nothing of M (but I can handle data transforms quite well). What I would like to do is:

1. Get first batch of results (say, 40000 results from start of data set) and save to a table - call it "all_results"
2. From "all_results" obtain last datetime
3. Rerun API call with "&since='last datetime'" parameter (ie: pull the next bunch of results)
4. Append results to "all_results"

It doesn't need to be a loop - just pull the latest batch of results so that it doesn't need to pull the entirety of the set each time.

Many thanks in advance for any pointers!