Forum Discussion
how to create a query that paginates?
Hi
Apologies if this is construed as a cross post. if I have a working example of a query that paginates that Imke helped me with - see below.
let
Pagination = List.Skip(List.Generate( () => [Table = #table({}, {{}}) ,Page = 1, Counter=0],
each Table.RowCount([Table])>0 or [Counter]=0,
each [ WebCall = Json.Document(Web.Contents("https://api.capsulecrm.com/api/v2/opportunities?perPage=100&embed=tags&page="&Text.From([Page])&"",[Headers=[Authorization="Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"]])),
Page = [Page]+1,
Counter = [Counter]+1,
Table = Table.FromRecords(WebCall[opportunities])
]
,each [Table]
) ,1),
#"Converted to Table" = Table.FromList(Pagination, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
#"Converted to Table"
So what I'm trying to acomplish now, is create a query that paginates based on the query above but Posts a filter in the body of the request. To put it bluntly, it's doing my head in, so I'm hoping that someone can assist? It uses a different URL however, https://api.capsulecrm.com/api/v2/opportunities/filters/results and I need to post the following in the body of the request;
{ "filter" : { "conditions": [ { "field": "isClosed", "operator": "is", "value": false } ] }}
This is what I have coded thus far.
let
obj = "{""filter"":{""conditions"":[{""field"":""isClosed"",""operator"":""is"",""value"":false }]}}",
authKey = "Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
url = "https://api.capsulecrm.com/api/v2/opportunities/filters/results?perPage=100&page=",
Pagination = List.Skip(List.Generate( () => [Table = #table({}, {{}}) ,Page = 1, Counter=0],
each Table.RowCount([Table])>0 or [Counter]=0, // Condition under which the next execution will happen
each [ WebCall = Json.Document(Web.Contents(url&Text.From([Page]) & obj,[Headers=[#"Authorization"=authKey, #"Content-Type"="application/json"],Content = Text.ToBinary(obj)])),
Page = [Page]+1,
Counter = [Counter]+1,// internal counter
Table = Table.FromRecords(WebCall[opportunities])
]
,each [Table]
) ,1)
in
Pagination
Obviously the code above does not work, otherwise I wouldnt be posting here :smileywink:. I do however think I'm on the right track. See error produced below;
DataSource.Error: Web.Contents failed to get contents from 'https://api.capsulecrm.com/api/v2/opportunities/filters/results?perPage=100&page=1%7B%22filter%22:%7B%22conditions%22:%5B%7B%22field%22:%22isClosed%22,%22operator%22:%22is%22,%22value%22:false%20%7D%5D%7D%7D' (400): Bad Request
Details:
DataSourceKind=Web
DataSourcePath=https://api.capsulecrm.com/api/v2/opportunities/filters/results
Url=https://api.capsulecrm.com/api/v2/opportunities/filters/results?perPage=100&page=1%7B%22filter%22:%7B%22conditions%22:%5B%7B%22field%22:%22isClosed%22,%22operator%22:%22is%22,%22value%22:false%20%7D%5D%7D%7D
Hope someone can help
Kind Regards - Grant
- ImkeF9 years ago
Community Champion
Please check out this post and see if you can get it working: https://eriksvensen.wordpress.com/2014/09/15/specifying-json-query-in-power-query-example-statistics-sweden/
- spocx8 years ago
Helper I
Hi ImkeF
I am as well a total novice joining this great thread.
I am trying to use the code you helped Grant to build earlier in this thread to accomplish a looped rest api call.
I have basically just changed the url in the webcall and the Pagination values that is defined for the page i am trying to get data from. The Pagination values for the system can be found here: https://developer.itrp.com/v1/general/pagination/
let Pagination = List.Skip(List.Generate( () => [Table = #table({}, {{}}) ,Page = 1, Counter=0], // Start Value each Table.RowCount([Table])>0 or [Counter]=0, // Condition under which the next execution will happen each [ WebCall = Json.Document(Web.Contents("https://api.itrp.qa/requests?per_page=100&page="&Text.From([Page])&"")), // retrieve results per call Page = [Page]+1, Counter = [Counter]+1,// internal counter Table = Table.FromRecords(WebCall[requests]) // steps of your further query ] ,each [Table] ) ,1), #"Converted to Table" = Table.FromList(Pagination, Splitter.SplitByNothing(), null, null, ExtraValues.Error) in #"Converted to Table"But i recieve the following error when trying to convert to table:
Any kind of help is much appreciated.
- ImkeF8 years ago
Community Champion
Please try with additional curly brackets in "Table" like so:
let Pagination = List.Skip(List.Generate( () => [Table = #table({}, {{}}) ,Page = 1, Counter=0], // Start Value each Table.RowCount([Table])>0 or [Counter]=0, // Condition under which the next execution will happen each [ WebCall = Json.Document(Web.Contents("https://api.itrp.qa/requests?per_page=100&page="&Text.From([Page])&"")), // retrieve results per call Page = [Page]+1, Counter = [Counter]+1,// internal counter Table = Table.FromRecords( { WebCall[requests] } ) // steps of your further query ] ,each [Table] ) ,1), #"Converted to Table" = Table.FromList(Pagination, Splitter.SplitByNothing(), null, null, ExtraValues.Error) in #"Converted to Table"