API search results with random next page

[cccarv82]

Hi all,

I'm trying to import data from an API where I have a limitation of 25 results per page for the search and when I have a result with more than 25 results I have a column with the "next page" string to do the search.

My question is... How can I automate this process? I mean... if I have 100 pages of results, how can I get all the results pages into one table using this random address?

PS.: The random part is the string after "&next=". Each page returns a different string.

Example: 

/api/v3/search/stories?query=created%3A2021-12-01..*+is%3Astory&page_size=25&next=225357e4865eac91a3eb3008dd60c124885304a1~49

In the above example I need to perform another search using the base URL plus this string to get the next page and so on...

[lbendlin]

Read about API pagination. My recommended approach is to use List.Generate to harvest all URLs and then combine their contents in one step.