Forum Discussion
yuvapraveen
5 years agoRegular Visitor
Paginate API POST Response
I am trying to accumulate API response from an URL. Initial response contains three JSON objects 1. events-containing the list that I wanted to accumulate and create a dataset 2. next_page- containin...
yuvapraveen
5 years agoRegular Visitor
I have also tried the below snippet, however the result only gives the content from the first page. There are in total 217 responses however I only get response for the first 100.
let
// Create a camoflouged secret from POSTMAN and plug in here
AuthKey = "Basic <token-here",
EntitiesPerPage = 100,
Url = "https://adb-xxxxxx.3.azuredatabricks.net/api/2.0/clusters/events",
Offset1 = "{
""cluster_id"":""xxxxx"",
""event_types"": [""STARTING"",""TERMINATING"",""CREATING""],
""limit"":" & Text.From(EntitiesPerPage) & "}",
GetJson = (Offset_passed) =>
let
offset_set = Offset_passed,
RawData = Web.Contents(Url,[
Headers = [#"Authorization"=AuthKey ,
#"Content-Type"="application/json"],
Content = Text.ToBinary(offset_set), Timeout=#duration(0,2,0,0)
]
),
Json = Json.Document(RawData)
in Json,
GetEntities = () =>
let ent_Offset = Offset1,
Json = GetJson(ent_Offset),
EntityCount = Json[#"total_count"]
in
EntityCount,
FnGetOnePage =
(cur_Offset) as record =>
let
Json = GetJson(cur_Offset),
data = try Json[#"events"] otherwise null,
next = try Json[#"next_page"] otherwise null,
res = [Data=data, Next=next]
in
res,
EntityCount = GetEntities(),
PageCount = Number.RoundUp(EntityCount / EntitiesPerPage),
GeneratedList = List.Generate(
()=>[i=0, res = FnGetOnePage(Offset1)],
each [i]<PageCount and [res][Data]<>null,
each [i=[i]+1, res = FnGetOnePage([res][Next])],
each [res][Data]),
Entities = List.Union(GeneratedList)
in
Entities