Forum Discussion
Cursor Paginated data from Web API
Hi Anonymous ,
one day, we'll get there 😉
let
myList = List.Generate( ()=>
[Result = Query4("https://api.brighttalk.com/v1/channel/10737/webcasts"), Counter = 0],
each [Counter] < 5,
each [Result = Query4(Table.LastN([Result]{[Name = "link"]}[Table], 1){0}[#"Attribute:href"]), Counter = [Counter]+1],
each [Result])
in
myList
Hi ImkeF,
Any chance you can also help me, very similar issue to this thread but I have to drill down a few steps to get my 'next' url
Function GetURL:
(url as text) as table =>
let
Source = Json.Document(Web.Contents(url)),
results = Source[results],
#"Converted to Table" = Table.FromList(results, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "properties", "createdAt", "updatedAt", "archived"}, {"id", "properties", "createdAt", "updatedAt", "archived"}),
#"Expanded properties" = Table.ExpandRecordColumn(#"Expanded Column1", "properties", {"amount", "createdate", "dealstage", "hs_lastmodifieddate", "hs_object_id", "printing_kpi_by"}, {"amount", "createdate", "dealstage", "hs_lastmodifieddate", "hs_object_id", "printing_kpi_by"})//& Number.ToText(after))),
in
#"Expanded properties"
the result if just requesting the first URL & not drilling down (apologies for using a table but having errors every time I try to load screenshots):
| Name | Value |
| results | List |
| paging | Records |
The next URL is within the 'paging' records, the following steps are required to get to it:
let
source = Json.Document(Web.Contents("https://api.hubapi.com/crm/v3/objects/deals?limit=100&paginateAssociations=false&archived=false&hapikey=APIKEY&properties=dealstage&after=0")),
#"Converted to Table" = Record.ToTable(source),
Value = #"Converted to Table"{1}[Value],
next = Value[next],
#"Converted to Table1" = Record.ToTable(next)
in
#"Converted to Table1"
Which returns the following table:
| Name | Value |
| after | 1677291168 |
| link | https://api.hubapi.com/crm/v3/objects/deals?archived=false&paginateAssociations=false&limit=100&after=1677291168&properties=dealstage |
This is the code I am trying to tweak based on this thread, I am just unsure how to reference the next link correctly
let
myList = List.Generate( ()=>
[Result = GetURL("https://api.hubapi.com/crm/v3/objects/deals?limit=100&paginateAssociations=false&archived=false&hapikey=APIKEY&properties=dealstage&after=0"), Counter = 0],
each [Counter] < 5,
each [Result = GetURL(Table.LastN([Result]{[Name = "link"]}[next], 1){0}[#"Attribute:value"]), Counter = [Counter] +1],
each [Result])
in
myList
It returns a list of 5, row 1 table pulls through correctly as 'Table' the next 4 show as errors.
"Expression.Error: the key didn't match any rows in the table
Details:
Key =
Name=link
Table=[Table]
Any help would be greatly appreciated,
Thanks!