Forum Discussion
Paginated results from an API in Power Query
Does your friendly developer have any kind of documentation for the API they created? How is the API indicating that the last of the pages is reached?
- tigertedd1 year agoRegular Visitor
His response is: "Just the standard OData way, the next link would end up null/empty."
- tigertedd1 year agoRegular Visitor
No specific documentation. But his response to the question is: "Just the standard OData way, the next link would end up null/empty."
- lbendlin1 year agoSuper User
Aha! So the call response includes the data and the pointer? There's a convenience function for that. Search for "Power BI API paging".
- tigertedd1 year agoRegular Visitor
Are you referrring to: Table.GenerateByPage?
I've tried using this, but power query says the expression is unrecognised.
I looked at this page: Helper functions for M extensions for Power Query connectors - Power Query | Microsoft Learn - which says at the top that it's not part of the laibrary yet and needs to be copied.I managed to get as far as createing this code, which resulted in a row 201 appearing, but with an error, which was progress of a sort:
let // Define the Table.GenerateByPage function for pagination Table.GenerateByPage = (getNextPage as function) as table => let listOfPages = List.Generate( () => getNextPage(null), // Get the first page of data (lastPage) => lastPage <> null, // Stop when the function returns null (lastPage) => getNextPage(lastPage) // Pass the previous page to the next function call ), // Concatenate the pages together tableOfPages = Table.FromList(listOfPages, Splitter.SplitByNothing(), {"Column1"}), firstRow = tableOfPages{0}? in // If we didn't get back any pages of data, return an empty table // Otherwise set the table type based on the columns of the first page if (firstRow = null) then Table.FromRows({}) else if (Table.IsEmpty(firstRow[Column1])) then firstRow[Column1] else Value.ReplaceType( Table.ExpandTableColumn(tableOfPages, "Column1", Table.ColumnNames(firstRow[Column1])), Value.Type(firstRow[Column1]) ), // Define the getNextPage function for your API getNextPage = (lastPage as nullable any) => let // Construct the API URL with pagination logic url = "https://finder.bloodsandbeyond.co.uk/myURL" & (if lastPage = null then "" else "?$skip=" & Number.ToText(lastPage)), headers = [ #"x-api-token" = "?????????????????" ], source = Web.Contents(url, [Headers=headers]), jsonResponse = Json.Document(source), value = jsonResponse[value], // Check if the value is null or empty nextPage = if value = null or List.IsEmpty(value) then null else value in // Return the result as a table or null if nextPage = null then null else Table.FromList(nextPage, Splitter.SplitByNothing(), null, null, ExtraValues.Error), // Use Table.GenerateByPage to get all data allData = Table.GenerateByPage(getNextPage) in allDataWhen I expand the error, it was:
Expression.Error: We cannot convert a value of type Table to type Number.
Details:
Value=[Table]
Type=[Type]#I'm guessing some part of this code tries to covert a table to a number, but I don't know enough about it to decipher it. Chat GPT disappeared up it's own tail trying to work out a fix. I'm optimistically hoping that someone smart enough can see the code and it's a simple fix. I feel like I'm really close.