Forum Discussion
Paginated api call issue - Function do not call the last page
Hi folks!
I have an API endpoint, which returns data on a paginated way. I have created my function, and I used List.Generate() because for each page I call, I recieve my data, and a 'next' field, which tells me which, url has to be called next. If the next field is empty / blank, then I got to the last page, so the iterative api calling should stop.
The problem is, that my function stops before it would call the last page. The reason is, that as I get to the last page the next field is empty, thus, the List.Generate() stops because of meeting the ExitCondition criteria.
Does anyone has a clue, how to fix this issue?
My code can be seen below,
let
fn_getExpData = (initialUrl as text) =>
let
BaseUrl = "https://xxx.xxxxx.xxxxx.co.hu/2/expenses",
AllData = List.Generate(
() =>
[
CurrentUrl = initialUrl,
Data = {},
Continue = true
],
each [Continue],
each
let
CurrentUrl = try [CurrentUrl] otherwise null,
ExitCondition = CurrentUrl = null,
QueryParts = if not ExitCondition and Text.Contains(CurrentUrl, "?")
then Text.Split(Text.AfterDelimiter(CurrentUrl, "?"), "&")
else {},
QueryNames = List.Transform(QueryParts, each Text.BeforeDelimiter(_, "=")),
QueryValues = List.Transform(QueryParts, each Text.AfterDelimiter(_, "=")),
QueryParams = if not ExitCondition then Record.FromList(QueryValues, QueryNames) else [],
Timestamp = DateTime.LocalNow(),
Response = if not ExitCondition
then try Json.Document(Web.Contents(BaseUrl, [Query=QueryParams, Headers=[Authorization="Bearer " & Token]])) otherwise null
else null,
DataChunk = try Response[results][expenses] otherwise {},
DataWithMetadata = if not ExitCondition then
List.Transform(DataChunk, each
Record.AddField(
Record.AddField(_, "API_URL", CurrentUrl),
"Response_Timestamp", Timestamp
)
)
else {},
NextUrl = try Response[next] otherwise null,
NewContinue = NextUrl <> null and not ExitCondition
in
[
CurrentUrl = NextUrl,
Data = DataWithMetadata,
Continue = NewContinue
],
each [Data]
),
FlattenedData = List.Combine(AllData)
in
FlattenedData
in
fn_getExpData
Thanks in advance
David
freidav1984
Hi, The issue with yourList.Generate()function stopping before retrieving the last page is due to the ExitCondition being triggered too early. Specifically, when thenextfield isnull(or empty),List.Generate()immediately stops before processing the last page's data.
use this codelet
fn_getExpData = (initialUrl as text) =>
let
BaseUrl = "https://xxx.xxxxx.xxxxx.co.hu/2/expenses",AllData = List.Generate(
() =>
[
CurrentUrl = initialUrl,
Data = {},
Continue = true
],
each [Continue], // Keep iterating while Continue = true
each
let
CurrentUrl = [CurrentUrl],// Fetch API response
Response = try Json.Document(Web.Contents(BaseUrl, [Query=if CurrentUrl <> null then [next=CurrentUrl] else [], Headers=[Authorization="Bearer " & Token]])) otherwise null,// Extract Data
DataChunk = try Response[results][expenses] otherwise {},// Add metadata
Timestamp = DateTime.LocalNow(),
DataWithMetadata = List.Transform(DataChunk, each
Record.AddField(
Record.AddField(_, "API_URL", CurrentUrl),
"Response_Timestamp", Timestamp
)
),// Get Next Page URL
NextUrl = try Response[next] otherwise null,// Continue if there's data OR if there's another page
NewContinue = NextUrl <> null
in
[
CurrentUrl = NextUrl,
Data = DataWithMetadata,
Continue = NewContinue
],
each [Data] // Extract only the Data part
),FlattenedData = List.Combine(AllData) // Flatten list
in
FlattenedData
in
fn_getExpDataDid I answer your question? Mark my post as a solution! Appreciate your Kudos !!
4 Replies
- johnbasha33
Super User
freidav1984
Hi, The issue with yourList.Generate()function stopping before retrieving the last page is due to the ExitCondition being triggered too early. Specifically, when thenextfield isnull(or empty),List.Generate()immediately stops before processing the last page's data.
use this codelet
fn_getExpData = (initialUrl as text) =>
let
BaseUrl = "https://xxx.xxxxx.xxxxx.co.hu/2/expenses",AllData = List.Generate(
() =>
[
CurrentUrl = initialUrl,
Data = {},
Continue = true
],
each [Continue], // Keep iterating while Continue = true
each
let
CurrentUrl = [CurrentUrl],// Fetch API response
Response = try Json.Document(Web.Contents(BaseUrl, [Query=if CurrentUrl <> null then [next=CurrentUrl] else [], Headers=[Authorization="Bearer " & Token]])) otherwise null,// Extract Data
DataChunk = try Response[results][expenses] otherwise {},// Add metadata
Timestamp = DateTime.LocalNow(),
DataWithMetadata = List.Transform(DataChunk, each
Record.AddField(
Record.AddField(_, "API_URL", CurrentUrl),
"Response_Timestamp", Timestamp
)
),// Get Next Page URL
NextUrl = try Response[next] otherwise null,// Continue if there's data OR if there's another page
NewContinue = NextUrl <> null
in
[
CurrentUrl = NextUrl,
Data = DataWithMetadata,
Continue = NewContinue
],
each [Data] // Extract only the Data part
),FlattenedData = List.Combine(AllData) // Flatten list
in
FlattenedData
in
fn_getExpDataDid I answer your question? Mark my post as a solution! Appreciate your Kudos !!
- v-menakakota
Community Support
hi freidav1984 ,
I would also take a moment to thank johnbasha33 , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.
- v-menakakota
Community Support
Hi freidav1984 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.- v-menakakota
Community Support
Hi freidav1984 ,
I hope the information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.