Forum Discussion
DBATTIN
5 years agoFrequent Visitor
Getting Paginated Data from REST API
I am attempting to get data from a RST API where the number of records is limited to 1,000 records. The data ia paginated. I have the following script but I still only get 1,000 records let Sou...
v-yingjl
4 years agoCommunity Support
Hi DBATTIN ,
You can refer this query from the simliar issue about offset and limit in REST API:
Paginate Rest API via Offset and Limit method
let
MYKEY="xxxxxx",//token string
EntitiesPerPage = 500,
Token= "&access_token=" & MYKEY,
Limit="&limit=" & Text.From(EntitiesPerPage),
Url = "https://api.searchsoftware.nl/v2/jobs?include=categories|contacts" & Limit,
GetJson = (Url) =>
let
RawData = Web.Contents(Url & Token),
Json = Json.Document(RawData)
in Json,
GetEntityCount = () =>
let Url = Url & "&offset=0" & Token,
Json = GetJson(Url),
Count = Json[#"total_count"]
in
Count,
GetPage = (Index) =>
let
//(option A)offset equal to previous row count
offset = "&offset=" & Text.From(Index * EntitiesPerPage),
//(option B)offset equal to page numer
//offset = "&offset=" & Text.From(Index),
Url = Url & offset & Token,
Json = GetJson(Url),
Value = Json[#"jobs"]
in
Value,
EntityCount = GetEntityCount(),
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage),
PageIndices = { 0 .. PageCount - 1 },
Pages = List.Transform(PageIndices, each GetPage(_)),
Entities = List.Union(Pages),
Table = Table.FromList(Entities, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
Table
Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.