Forum Discussion

freidav1984's avatar
freidav1984
Frequent Visitor
1 year ago
Solved

Paginated api call issue - Function do not call the last page

Hi folks!

 

I have an API endpoint, which returns data on a paginated way. I have created my function, and I used List.Generate() because for each page I call, I recieve my data, and a 'next' field, which tells me which, url has to be called next. If the next field is empty / blank, then I got to the last page, so the iterative api calling should stop. 

 

The problem is, that my function stops before it would call the last page. The reason is, that as I get to the last page the next field is empty, thus, the List.Generate() stops because of meeting the ExitCondition criteria. 

 

Does anyone has a clue, how to fix this issue?

 

My code can be seen below,

 

 

let
    fn_getExpData = (initialUrl as text) =>
    let
        BaseUrl = "https://xxx.xxxxx.xxxxx.co.hu/2/expenses",


        AllData = List.Generate(
            () => 
                [
                    CurrentUrl = initialUrl, 
                    Data = {}, 
                    Continue = true
                ], 
            each [Continue], 
            each 
                let
 
                    CurrentUrl = try [CurrentUrl] otherwise null,

 
                    ExitCondition = CurrentUrl = null,

 
                    QueryParts = if not ExitCondition and Text.Contains(CurrentUrl, "?") 
                        then Text.Split(Text.AfterDelimiter(CurrentUrl, "?"), "&") 
                        else {},

                    QueryNames = List.Transform(QueryParts, each Text.BeforeDelimiter(_, "=")),
                    QueryValues = List.Transform(QueryParts, each Text.AfterDelimiter(_, "=")),
                    QueryParams = if not ExitCondition then Record.FromList(QueryValues, QueryNames) else [],

 
                    Timestamp = DateTime.LocalNow(),

 
                    Response = if not ExitCondition 
                        then try Json.Document(Web.Contents(BaseUrl, [Query=QueryParams, Headers=[Authorization="Bearer " & Token]])) otherwise null
                        else null,
                    
 
                    DataChunk = try Response[results][expenses] otherwise {},

 
                    DataWithMetadata = if not ExitCondition then 
                        List.Transform(DataChunk, each 
                            Record.AddField(
                                Record.AddField(_, "API_URL", CurrentUrl), 
                                "Response_Timestamp", Timestamp
                            )
                        ) 
                        else {},
 
                    NextUrl = try Response[next] otherwise null,

 
                    NewContinue = NextUrl <> null and not ExitCondition
                in 
                    [
                        CurrentUrl = NextUrl, 
                        Data = DataWithMetadata, 
                        Continue = NewContinue
                    ],
            each [Data]
        ),

 
        FlattenedData = List.Combine(AllData)
    in 
        FlattenedData
in 
    fn_getExpData

 

Thanks in advance 

 

David

  • freidav1984 
    Hi, The issue with your List.Generate() function stopping before retrieving the last page is due to the ExitCondition being triggered too early. Specifically, when the next field is null (or empty), List.Generate() immediately stops before processing the last page's data.

    use this code

    let
    fn_getExpData = (initialUrl as text) =>
    let
    BaseUrl = "https://xxx.xxxxx.xxxxx.co.hu/2/expenses",

    AllData = List.Generate(
    () =>
    [
    CurrentUrl = initialUrl,
    Data = {},
    Continue = true
    ],
    each [Continue], // Keep iterating while Continue = true
    each
    let
    CurrentUrl = [CurrentUrl],

    // Fetch API response
    Response = try Json.Document(Web.Contents(BaseUrl, [Query=if CurrentUrl <> null then [next=CurrentUrl] else [], Headers=[Authorization="Bearer " & Token]])) otherwise null,

    // Extract Data
    DataChunk = try Response[results][expenses] otherwise {},

    // Add metadata
    Timestamp = DateTime.LocalNow(),
    DataWithMetadata = List.Transform(DataChunk, each
    Record.AddField(
    Record.AddField(_, "API_URL", CurrentUrl),
    "Response_Timestamp", Timestamp
    )
    ),

    // Get Next Page URL
    NextUrl = try Response[next] otherwise null,

    // Continue if there's data OR if there's another page
    NewContinue = NextUrl <> null
    in
    [
    CurrentUrl = NextUrl,
    Data = DataWithMetadata,
    Continue = NewContinue
    ],
    each [Data] // Extract only the Data part
    ),

    FlattenedData = List.Combine(AllData) // Flatten list
    in
    FlattenedData
    in
    fn_getExpData


    Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!

4 Replies

  • freidav1984 
    Hi, The issue with your List.Generate() function stopping before retrieving the last page is due to the ExitCondition being triggered too early. Specifically, when the next field is null (or empty), List.Generate() immediately stops before processing the last page's data.

    use this code

    let
    fn_getExpData = (initialUrl as text) =>
    let
    BaseUrl = "https://xxx.xxxxx.xxxxx.co.hu/2/expenses",

    AllData = List.Generate(
    () =>
    [
    CurrentUrl = initialUrl,
    Data = {},
    Continue = true
    ],
    each [Continue], // Keep iterating while Continue = true
    each
    let
    CurrentUrl = [CurrentUrl],

    // Fetch API response
    Response = try Json.Document(Web.Contents(BaseUrl, [Query=if CurrentUrl <> null then [next=CurrentUrl] else [], Headers=[Authorization="Bearer " & Token]])) otherwise null,

    // Extract Data
    DataChunk = try Response[results][expenses] otherwise {},

    // Add metadata
    Timestamp = DateTime.LocalNow(),
    DataWithMetadata = List.Transform(DataChunk, each
    Record.AddField(
    Record.AddField(_, "API_URL", CurrentUrl),
    "Response_Timestamp", Timestamp
    )
    ),

    // Get Next Page URL
    NextUrl = try Response[next] otherwise null,

    // Continue if there's data OR if there's another page
    NewContinue = NextUrl <> null
    in
    [
    CurrentUrl = NextUrl,
    Data = DataWithMetadata,
    Continue = NewContinue
    ],
    each [Data] // Extract only the Data part
    ),

    FlattenedData = List.Combine(AllData) // Flatten list
    in
    FlattenedData
    in
    fn_getExpData


    Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!

  • v-menakakota's avatar
    v-menakakota
    Icon for Community Support rankCommunity Support

    hi freidav1984 ,

    I would also take a moment to thank johnbasha33 , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference. 

     

    May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

    Thank you.

     

    • v-menakakota's avatar
      v-menakakota
      Icon for Community Support rankCommunity Support

      Hi freidav1984 ,


      I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

      Thank you.

      • v-menakakota's avatar
        v-menakakota
        Icon for Community Support rankCommunity Support

        Hi  freidav1984 ,

        I hope the information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

        Thank you.