Is it possible to automatically use the nextPageUrl from an API call as the new web source?

[Anonymous]

I have got an API Call, which gives me 500 Results and a field under Page Details (nextPageUrl) with the rest of the data.

Is it possible to add this new link for as many times as there are additional Pages?

[amithg]

Hi.  I have tried to to follow the steps in SQL Code Space: Power BI/Power Query: API Response Pagination Part 2 - Paginate by Page Cursor to get data into my sheet (400k records, 30 per loop) but I keep getting no records.  Any help would be appreciated:

let    

    ufnGetList = (cursor) =>

        let 

            result = Json.Document(Web.Contents("https://apiendpoint" & "?cursor=" &cursor, [Headers=[Authorization="xxx"]])),

            nextCursor = try result[data][cursor] otherwise null,

            results = try result[data][results] otherwise null,

            record = [results = results, nextCursor = nextCursor]

        in 

            record,

    resultSet = List.Generate(

                    () => ufnGetList(""),

                    each _[nextCursor] <> null,

                    each ufnGetList(_[nextCursor]),

                    each [nextCursor = _[nextcursor], results = _[results]]

                    )

    in

        resultSet

the JSON of each result looks like:

{

    "status": 200,

    "data": {

        "cursor": "FGluY2x1ZGVfY29udGV4dF91dWlkDXF1ZXJ5QW5kRmV0Y2gBFHAxVE51WGdCeFVhMWFiZ2ltb3dUAAAAABtWSaEWOHRuNHZFVXNUazIyMC1QckhwUlFOUQ==",

        "limit": 50,

        "results": [

             ...

            ],

Any idea what I am doing wrong?  The authorization and api data retrieve works fine for the first page but I can only retrieve 30 records, not all 400k.  Thanks!

[amitchandak · Accepted solution]

@Anonymous , refer if this can help you a bit

https://datachant.com/2016/06/27/cursor-based-pagination-power-query/

https://medium.com/@marktiedemann/how-to-do-pagination-in-power-query-430460c17c78