Forum Discussion

tigertedd's avatar
tigertedd
Regular Visitor
1 year ago

Paginated results from an API in Power Query

A friendly developer has helped me create an API for a system he's built for me, so I can export the data into Power BI.

 

I've managed to import it, but  I'm running into a problem that the bigger data sets result in a line in the JsonResponse applied step that ends with some code: ?top=200&skip=200.

 

I've learned that this is to do with pagination, and that it's important to efficiently retrieve the data. But my tables are all limited to 200 rows.

 

How do I get the rest oft he data?

 

This is the code in the advanced editor that I'm currently using (this was generated with the help of ChatGPT, I'm quite skilled with the copy and paste buttons, but don't really know what I'm looking at):

 

 

let
url = "https://finder.bloodsandbeyond.co.uk/myurl",
headers = [
#"x-api-token-name" = "x-api-token", // Add this line if it is required
#"x-api-token" = "????????????????????????????" // Replace with your actual API key
],
source = Web.Contents(url, [Headers=headers]),
jsonResponse = Json.Document(source),
value = jsonResponse[value],
#"Converted to Table" = Table.FromList(value, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "lab_id", "assigned_user_id", "lab_invoice_id", "user_invoice_id", "appointment_at", "matched_at", "sample_posted_at", "is_under_18", "is_urgent", "patient_fee", "patient_cost", "collections_fee", "collections_cost", "reference", "status", "status_reason", "name", "address_line_1", "address_line_2", "city", "post_code", "lat", "lng", "phone", "email", "patients", "preferred_datetimes", "notes", "admin_notes", "deleted_at", "created_at", "updated_at", "confirmed_at", "is_issue", "invoice_notes", "issue_type"}, {"id", "lab_id", "assigned_user_id", "lab_invoice_id", "user_invoice_id", "appointment_at", "matched_at", "sample_posted_at", "is_under_18", "is_urgent", "patient_fee", "patient_cost", "collections_fee", "collections_cost", "reference", "status", "status_reason", "name", "address_line_1", "address_line_2", "city", "post_code", "lat", "lng", "phone", "email", "patients", "preferred_datetimes", "notes", "admin_notes", "deleted_at", "created_at", "updated_at", "confirmed_at", "is_issue", "invoice_notes", "issue_type"})
in
#"Expanded Column1"

 

 

 

Chat GPT has come up with a few other ideas, but at best it just ends up resulting in 200 rows again:

 

 

let
    // Initial URL
    initialUrl = "https://finder.bloodsandbeyond.co.uk/myURL",

    // Function to fetch each page of data
    GetAppointments = (url as text) as table =>
    let
        // Fetch the data from the API
        source = Json.Document(Web.Contents(url, [Headers = [#"x-api-token" = "?????????????????????????"]])),
        // Extract the value part of the response
        value = source[value],
        // Check if there's a next link for pagination
        nextLink = try source[#"@odata.nextLink"] otherwise null,
        // Convert the result to a table
        result = Table.FromList(value, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
        // Expand the columns in the result
        expanded = Table.ExpandRecordColumn(result, "Column1", {"id", "lab_id", "assigned_user_id", "lab_invoice_id", "user_invoice_id", "appointment_at", "matched_at", "sample_posted_at", "is_under_18", "is_urgent", "patient_fee", "patient_cost", "collections_fee", "collections_cost", "reference", "status", "status_reason", "name", "address_line_1", "address_line_2", "city", "post_code", "lat", "lng", "phone", "email", "patients", "preferred_datetimes", "notes", "admin_notes", "deleted_at", "created_at", "updated_at", "confirmed_at", "is_issue", "invoice_notes", "issue_type"}),
        // If there's more data, recursively fetch the next page
        finalResult = if nextLink <> null then
            Table.Combine({expanded, @GetAppointments(nextLink)})
        else
            expanded
    in
        finalResult,

    // Call the function to get all the data automatically
    allData = GetAppointments(initialUrl)
in
    allData

 

 

 

Any help will be greatly appreciated.

10 Replies

  • Does your friendly developer have any kind of documentation for the API they created?  How is the API indicating that the last of the pages is reached?

    • tigertedd's avatar
      tigertedd
      Regular Visitor

      His response is: "Just the standard OData way, the next link would end up null/empty."

    • tigertedd's avatar
      tigertedd
      Regular Visitor

      No specific documentation. But his response to the question is: "Just the standard OData way, the next link would end up null/empty."

      • lbendlin's avatar
        lbendlin
        Icon for Super User rankSuper User

        Aha! So the call response includes the data and the pointer? There's a convenience function for that. Search for "Power BI API paging".