Forum Discussion
how to create a query that paginates?
Hi ImkeF,
How would I go about doing this with a JIRA call, where I have a startAt and total number of records? I've included them below as an example, I can't get my head around how I would call them? Any help pointing me in the right direction would be great thanks.
let
Pagination = startAt": 0 "total": 161
each [Last_Key] <> null and [Last_Key] <> "", // Condition under which the next execution will happen
each [WebCall = Json.Document(Web.Contents("https://companyName.atlassian.net/rest/agile/1.0/board?startAt"&??,[Headers=[ContentType="application/json", Authorization="Auth="]])), // retrieve results per call
Last_Key = if [Counter]<=1 then 1 else WebCall[lastKey] ,// determine the LastKey for the next execution
Counter = [Counter]+1,// internal counter
#"Converted to Table" = Record.ToTable(WebCall), // steps of your further query
Value = #"Converted to Table"{1}[Value] // last step of your further queries
],
each [Value]),1),
Pagination1 = Pagination{0}
in
Pagination1Not as neat as the previous query, but you could try something along these lines:
Source1 = {0..161},
#"Converted to Table" = Table.FromList(Source1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Add Pagination Number" = Table.AddColumn(#"Converted to Table", "startAt", each [Column1]+1),
#"Added Custom" = Table.AddColumn(#"Add Pagination Number", "Custom", each Json.Document(Web.Contents("yourlink?startAt=0", [Headers=[your details],Query=[startAt=Text.From([startAt])]]))),
in
#"Added Custom"I'm not sure whether you actually need 'startAt' in the api link, but you can play around with removing it to get it to work?
- ImkeF8 years ago
Community Champion
Agree with DBa: No need to use the complicated List.Generate here. "Just" create a table with one row per api-call needed that holds all the variables/parameters that are needed to create the distinct URLs.
If one call can fetch 50 records, then your table might just need four rows and you could start with a list like this:
{1..Number.RoundUp(161/50)}
- Anonymous8 years agoNot applicable
Hi ImkeF & Community,
Need help with a similar problem.
Same as rest, started from DataChatn's blog and got stuck in some implementation.
https://datachant.com/2016/06/27/cursor-based-pagination-power-query/
I am trying to iterate over an API source, to fetch some data(pagination).
I have to do a "POST" request, and in the "content/query parameters", provide information for the next page (not part of url)
this information goes in as parameter named "AFTER" ( as seen in below snapshot- this is the value that we get from [data][next] response - keep reading)
In the responce of any query, I get three fields.
[data][post] --> this is my data part,
[data][remaining] -->count of entries left,
[data][next]-->a 64base code to be placed in the next query paramter to tell, starting point for next fetch
below documentation for reference:
The first query need to go either without "after" parameter, or we can set it to a stationary value. later queries can go and use "next"'s value for "after".
I have tried to do following - can someone please see, if there are some evident errors I have made here
let iterations = 10000 // just for initialization auth_key ="Basic myAuthKey", url = "https://api.socialbakers.com/0/facebook/page/posts", header= [#"Authorization" = auth_key, #"Content-Type" = "application/json; charset=utf-8"], content= "{ ""date_start"":""2017-08-16"", ""date_end"":""2017-11-10"", ""profile"":""88147621212"", ""fields"":[""id"",""created"",""message ], ""limit"":5, ""after"":[next] // Not sure if this is the right way to use the next value here }", // DO I need to pass just URL or Content as well in this method & define header wihtin the method? FnGetOnePage = (url) as record => let Source =Json.Document(Web.Contents(url,[Headers = header,Content = Text.ToBinary(content)])), data = try Source[data][posts] otherwise null, next = try Source[data][next] otherwise "somefixedvalue?", remaining = try Source[data][remaining] otherwise null, res = [Data = data, Next = next,rem = remaining] in res, GeneratedList = List.Generate( ()=>[i=0,res=FnGetOnePage(url)], each [i] <iterations and [res][rem]>0,
// Not sure about the next two lines, if they are making sense in my scenario-
// Depending how we designed our method - really confused about these each [i=[i] +1 , res = FnGetOnePage([res][Next])], each[res][Data]) in
GeneratedListAny help is this reagrd highly appriciated.
Cheers.
- Anonymous8 years agoNot applicable
Resolved - Million thumbs up ImkeF for your help.
Your solution is working smoothly.
Can't thank you enough.
Regards
emudria