Forum Discussion

HCJ2026's avatar
HCJ2026
Frequent Visitor
6 months ago
Solved

Get data - Json.document

Hi,
I'm trying to get data from an API to our system AutoPilot.

When I use this promt:

 Json.Document(

        Web.Contents(

            "https://api.autopilot.dk/v2/Timeregistrations",

            [

                Headers = [

                    #"accept" = "application/json",

                    #"Content-Type" = "application/json",

                    #"APPartnerKey" =

                    #"APCompanyKey" =

                ],

                Content = Text.ToBinary("{""page"":1,""pageSize"":29999}")

            ]

        )

    )
I only get the one page of date - but I would like to get all data.
Can someone assist me in getting that to happend?

  • HCJ2026's avatar
    HCJ2026
    5 months ago

    Hi Olufemi7  & v-sshirivolu 
    Thank you very much for your quick response to my request. 
    Yesterday we finaly succeded in getting the query correct and the I now get the expected data in my datamodel - thank you very much for your input and support.

9 Replies

  • v-sshirivolu's avatar
    v-sshirivolu
    Community Support

    Hi HCJ2026,

     

    Your current API request is set to fetch page 1 with a fixed pageSize of 29,999 records. Most modern APIs, including AutoPilot, use pagination, so each request only returns a single page of data even if more pages are available. This is why you only receive the first page. To get all records, you’ll need to add pagination logic and request each page in order, then combine the results. In Power Query (M language), this usually involves creating a function to retrieve one page and then using List.Generate or similar logic to loop through page numbers until no more data is returned.
    After collecting all pages, you can combine them into one table. Json.Document itself does not require any changes, rather, you need to add pagination logic so Power Query makes multiple API calls. This way, your query will return all timeregistrations, not just the first page.

    Refer these Microsoft Docs - 

    https://learn.microsoft.com/en-us/powerquery-m/web-contents

    https://learn.microsoft.com/en-us/power-query/samples/trippin/5-paging/readme

    https://learn.microsoft.com/en-us/power-query/handling-paging

     

    Thank you.

  • Hello HCJ2026

    This is expected behaviour the AutoPilot API is paginated.

    Your query explicitly requests only page 1, so Power Query will return just that page. Power Query does not automatically fetch all pages; you must loop through them in M.


    You need to implement pagination, for example with List.Generate:

    let
        BaseUrl = "https://api.autopilot.dk/v2/Timeregistrations",
    
        GetPage = (Page as number) =>
            Json.Document(
                Web.Contents(
                    BaseUrl,
                    [
                        Headers = [
                            #"accept" = "application/json",
                            #"Content-Type" = "application/json",
                            #"APPartnerKey" = "YOUR_KEY",
                            #"APCompanyKey" = "YOUR_KEY"
                        ],
                        Content = Text.ToBinary(
                            "{""page"":" & Number.ToText(Page) & ",""pageSize"":29999}"
                        )
                    ]
                )
            ),
    
        Pages =
            List.Generate(
                () => [Page = 1, Data = GetPage(1)],
                each List.Count([Data]) > 0,
                each [Page = [Page] + 1, Data = GetPage([Page])],
                each [Data]
            ),
    
        Combined = List.Combine(Pages),
        Result = Table.FromRecords(Combined)
    in
        Result

     

    Microsoft docs confirm that REST APIs must be paged manually in Power Query and all pages must be explicitly combined: Paging 

    • HCJ2026's avatar
      HCJ2026
      Frequent Visitor

      Thank you very much for your response.
      I still get this error when I try to set it up:

       

      • Olufemi7's avatar
        Olufemi7
        Super User

        Hello HCJ2026

        The 500 error suggests the API is rejecting the request structure rather than the pagination logic itself.

        In Power Query, when Content is included in Web.Contents, the request is automatically sent as POST. If this endpoint expects a GET request with query parameters instead, that can result in a 500 error.

        Try switching from Content to Query and reduce the page size to something smaller (e.g., 500):

        let
            BaseUrl = "https://api.autopilot.dk/v2/Timeregistrations",
        
            GetPage = (Page as number) =>
                Json.Document(
                    Web.Contents(
                        BaseUrl,
                        [
                            Headers = [
                                #"accept" = "application/json",
                                #"APPartnerKey" = "YOUR_KEY",
                                #"APCompanyKey" = "YOUR_KEY"
                            ],
                            Query = [
                                page = Number.ToText(Page),
                                pageSize = "500"
                            ]
                        ]
                    )
                ),
        
            Pages =
                List.Generate(
                    () => [Page = 1, Data = GetPage(1)],
                    each List.Count([Data]) > 0,
                    each [Page = [Page] + 1, Data = GetPage([Page])],
                    each [Data]
                ),
        
            Combined = List.Combine(Pages),
            Result = Table.FromRecords(Combined)
        
        in
            Result

         

        If the API response is wrapped (for example { data = [...] }), the loop condition would need to reference that field instead.

        Could you confirm whether the endpoint is documented as GET or POST?