Forum Discussion
API Pagination in JSON Body - How to Access with Power Query in Custom Connector?
This piece of code looks iffy:
link = (response)[paging][next]
Thank you!
This issue was is that I was only passing the url through the GetPage function in GetAllPages. I also needed to pass through another argument in GetPage containing the next page metadata/url
There are two option to approach this
- iterating through all pages, harvesting/accumulating the data, and repeating until no nextpage is found
- iterating through all pages, ignoring the data, only harvesting the URLs. Then add a cstom column that fetches the content for each URL and combines them in one step
The second approach seems to be wasteful as you seemingly fetch each URL twice. In reality this is most likely covered by the browser engine cache so there is no performance penalty. The benefit is that you don't have to lug the data around during the iteration, so performance is ultimately even better.
Thank you lbendlin,
Do you think that using the second way might help me get around API limits? The API I am working with case a limit of 250 calls a minutes and I think the way I did it (the first option in your list) is making me hit that limit.
Thanks!
And do you happen to have any examples or documentation for the second approach? It sounds very intriguing, and I am noticing performance issues doing it the first approach.