Forum Discussion
Rest API - call the next page URL without knowing total rows or total pages
- 4 years ago
Hi Anonymous
The next page url is stored in Result[_pagination][next] so try this
let Source = List.Generate( () => [ URL = "https://api2.frontapp.com/" , Result = Json.Document(Web.Contents(URL)) ], each [URL] <> null, each [ URL = [Result][_pagination][next] , Result = Json.Document(Web.Contents([URL])) ] ) in SourceThis code will end assuming that [_pagination][next] is null when there are no more records to retrieve.
Regards
Phil
Hi, Not sure if anyone out there can help. I used above example to get pagination working and data being pulled from Salesforce.
Below is my code:
let
url = "https://asx--qa.sandbox.my.salesforce.com/services/oauth2/token",
headers = [#"Content-Type"="application/x-www-form-urlencoded"],
body="grant_type=password&client_id=" & sf_client_id & "&client_secret=" & sf_client_secret & "&username=" & sf_username & "&password=" & sf_password,
response = Web.Contents(url,[Content = Text.ToBinary(body),Headers = headers]),
result1 = Json.Document(response),
token = result1[access_token],
qry = "SELECT Id,Name,Active__c,BillingStreet,BillingCity,BillingState,BillingPostalCode,BillingCountry,Phone,Fax,Website,CreatedDate,LastModifiedDate FROM Account where Active__c = true and (RecordType.Name = 'Customer' or RecordType.Name = 'Prospect') and Listings_Prospect__c = true",
Source = List.Generate(
() => [ URL= sf_base_url,
Result=Json.Document(Web.Contents(URL,[RelativePath="/services/data/v60.0/query/", Query= [q=(qry)], Headers = [Authorization = "Bearer " & token]])) ],
each try [URL] <> null otherwise null <> null,
each [
URL = if ([Result][nextRecordsUrl] <> null) then sf_base_url else null, Result=Json.Document(Web.Contents(URL,[RelativePath=[Result][nextRecordsUrl], Query= [q=""], Headers = [Authorization = "Bearer " & token]]))]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"URL", "Result"}, {"Column1.URL", "Column1.Result"}),
#"Expanded Column1.Result" = Table.ExpandRecordColumn(#"Expanded Column1", "Column1.Result", {"totalSize", "done", "nextRecordsUrl", "records"}, {"Column1.Result.totalSize", "Column1.Result.done", "Column1.Result.nextRecordsUrl", "Column1.Result.records"}),
#"Expanded Column1.Result.records" = Table.ExpandListColumn(#"Expanded Column1.Result", "Column1.Result.records"),
#"Expanded Column1.Result.records1" = Table.ExpandRecordColumn(#"Expanded Column1.Result.records", "Column1.Result.records", {"attributes", "Id", "Name", "Active__c", "BillingStreet", "BillingCity", "BillingState", "BillingPostalCode", "BillingCountry", "Phone", "Fax", "Website", "CreatedDate", "LastModifiedDate"}, {"attributes", "Id", "Name", "Status1", "BillingStreet", "BillingCity", "BillingState", "BillingPostalCode", "BillingCountry", "Phone", "Fax", "Website", "CreatedDate", "LastModifiedDate"}),
#"Uppercased Text" = Table.TransformColumns(Table.TransformColumnTypes(#"Expanded Column1.Result.records1", {{"Status1", type text}}, "en-AU"),{{"Name", Text.Upper, type text}, {"Status1", Text.Upper, type text}}),
#"Added Custom" = Table.AddColumn(#"Uppercased Text", "Status", each if ([Status1] = "TRUE") then "Active" else "InActive"),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Status1"}),
#"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Status", type text}}),
#"Sorted Rows" = Table.Sort(#"Changed Type",{{"Name", Order.Ascending}}),
#"Removed Columns1" = Table.RemoveColumns(#"Sorted Rows",{"Column1.URL", "Column1.Result.totalSize", "Column1.Result.done", "Column1.Result.nextRecordsUrl", "attributes"})
in
#"Removed Columns1"
How ever when I go to Data source settings, I get error "Some data sources may not be listed because of hand-authored queries". I want the published report to get the fresh data from API in near real time. If I do not have this pagination, the error goes away and in power bi I have option to only refresh once daily. What is the option to have published report getting real time data via API?