Dynamic API Pagination with next page Cursor in a Record

[Anonymous]

Hello,

I am having issues with writing a custom Query to get some data from an API.

In the initial response, I get these 2 results:

  data - (that has the 1st page of the data needed) and "pagination" which has this:

In order to get the next page of data, we need to make another API request with the value from endCursor added to the API uri in this format (...api.com/blabla?after=endCursorValue)

I've looked trough everything on the forum here and on the internet but I still can't find a suitable solution for this issue.

I'm sure that I need some kind of a loop that will add all the data in one list and request the next page of data using the endCursor value till there are no next pages (endCursor = "";  or hasNextPage = FALSE)

Thanks for your support!

[AmandaMulryan]

@v-easonf-msft @Anonymous

Here is what my data looks like in regard to the API loop:

 

Here is the query I've made but it does not work. It keeps saying "The field 'pagination.hasNextPage' of the record wasn't found. 

I am in need of code assistance if anyone is willing to help me.  

[AmandaMulryan]

Did you ever figure this out? I am trying to do the same thing with Samsara and am so close but can't seem to get it exactly right. Any help would be greatly appreciated! Thanks

@Anonymous

[Anonymous]

Here is a copy of the Pagination code from the API documentation:

import requests

url = "https://api.samsara.com/fleet/vehicles"
params = {}
headers = {
    "Authorization": "Bearer TOKEN"
}

hasNextPage = True
while hasNextPage:
    response = requests.request("GET", url, headers=headers, params=params).json()
    for vehicle in response["data"]:
        # process each vehicle
    hasNextPage = response["pagination"]["hasNextPage"]
    params["after"] = response["pagination"]["endCursor"]

I'm having a hard time translating this into M script. Can someone help me out with the M code that would work?

[v-easonf-msft]

Hi, @Anonymous 

You need to implement a for-loop using List.Generate.

About how to implement Cursor-Based Pagination using List.Generate to extract large data sets from Web APIs, please refer to below thread.

Iterate Over Dynamic Web API Pages With Power Query – How To Resolve Cursor-Based Pagination 

Best Regards,
Community Support Team _ Eason