Forum Discussion

ChadC's avatar
ChadC
Frequent Visitor
6 years ago
Solved

Help with List.Generate() in making API calls

Hi All,    I'm building a custom connector to utilize an API. I'm attempting to handle pagination, but am having a hard time. Here's what I'm trying to do: An call is made to the API. A response...
Need Help
  • ChadC's avatar
    ChadC
    6 years ago

    Hi v-xicai ,

     

    Thank you for your response. The issue was indeed related to null values being returned in subsequent calls. However, this was because the API I'm using required an "accept" header to be included, which I had overlooked. Below is my updated code:

     

    let
        HasContToken = (responseHeaders) as logical => if (Record.HasFields(responseHeaders[Headers], {"Continuation"})) then 
                    true
                 else
                    false,
        GetPage = (Scope, ContinuationToken) =>
            let
                RawData = Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Query = [
                        limit = "1000"
                    ],
                    Headers = [
                        #"Continuation" = ContinuationToken,
                        accept = "application/json"
                   ]])
            in
                RawData,
        Pages = List.Generate(() => Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Headers = [
                        accept = "application/json"
                    ],
                    Query = [
                        limit = "1000"
                    ]]),
                each HasContToken(Value.Metadata(_)),
                each GetPage(Scope, Value.Metadata(_)[Headers][Continuation])),
        JSON = List.Transform(Pages, (_) => Json.Document(_)),
        List = List.Combine(JSON),
        Table = Table.FromList(List, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
    in
        Table;
v-xicai's avatar
v-xicai
Community Support
6 years ago

Hi ChadC ,

 

When combining files, be sure to choose "Skip files with errors." see more: https://www.sqlchick.com/entries/2018/5/6/querying-data-in-azure-data-lake-store-with-power-bi.

 

There may be some nulls in your dataset which generated the error when the parsing was attempted, replace the null values with an empty structure "<div></div>" which was then parsed . See The parameter is expected to be of type Text.Type or Binary.Type.

 

If you still have this issue for Power BI, you'd better create a support ticket in Power BI Support , Scroll down and click "CREATE SUPPORT TICKET", to get further help.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

ChadC's avatar
ChadC
Frequent Visitor
6 years ago

Hi v-xicai ,

 

Thank you for your response. The issue was indeed related to null values being returned in subsequent calls. However, this was because the API I'm using required an "accept" header to be included, which I had overlooked. Below is my updated code:

 

let
        HasContToken = (responseHeaders) as logical => if (Record.HasFields(responseHeaders[Headers], {"Continuation"})) then 
                    true
                 else
                    false,
        GetPage = (Scope, ContinuationToken) =>
            let
                RawData = Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Query = [
                        limit = "1000"
                    ],
                    Headers = [
                        #"Continuation" = ContinuationToken,
                        accept = "application/json"
                   ]])
            in
                RawData,
        Pages = List.Generate(() => Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Headers = [
                        accept = "application/json"
                    ],
                    Query = [
                        limit = "1000"
                    ]]),
                each HasContToken(Value.Metadata(_)),
                each GetPage(Scope, Value.Metadata(_)[Headers][Continuation])),
        JSON = List.Transform(Pages, (_) => Json.Document(_)),
        List = List.Combine(JSON),
        Table = Table.FromList(List, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
    in
        Table;