Forum Discussion

h4tt3n's avatar
h4tt3n
Helper V
5 years ago

How to overcome API call row limit?

Hello,

 

I am attempting to extract measurement data from an API with a limit of 1000 rows per call. Each row consists of a data value and a timestamp. I have set up a Power Query script that connects to the API and can retrieve any 1000 concecutive rows of data. 

 

What I need is an iterative script that retrieves the data I need in chunks of 1000 rows and merges them ino a single table. Below is my current script. The Source line is the one of interest here.

 

 

 

let
    Source = Json.Document(Web.Contents(""&url&"/TagsData/"&tagid&"/After?subscription-key="&key&"&after="&after&"&count="&count&"")),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"tag", "data"}, {"tag", "data"}),
    #"Removed Columns" = Table.RemoveColumns(#"Expanded Column1",{"tag"}),
    #"Expanded data" = Table.ExpandListColumn(#"Removed Columns", "data"),
    #"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"d", "v"}, {"d", "v"}),
    #"Changed Type" = Table.TransformColumnTypes(#"Expanded data1",{{"d", type datetime}, {"v", type number}})
in
    #"Changed Type"

 

 

 

 The body parameter "after" sets the datetime after which you want data from. The parameter "count" sets the number of rows, and is limited to 1000.

 

One challenge is that the API is not paginated, so I will need to keep track of timestamps and number of returned rows for each call.

 

I have been working with PowerBI professionally for the last year, but my Power Query scripting skills are not good enough for this challenge. Thanks in advance for any help.

 

Cheers, Mike

Need Help

4 Replies

  • Jimmy801's avatar
    Jimmy801
    Community Champion
    5 years ago

    Hello h4tt3n 

     

    there are a lot of examples of how to do in this forum.

    They use all List.Generate where you have a parameter when to stop to generate to api calls like giving an error or empty page.

    just start out to search for api List.Generate.

    If you have any specific question feel free to come back


    If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
    Kudoes are nice too

    Have fun

    Jimmy

    • Anonymous's avatar
      Anonymous
      Not applicable
      3 years ago

      mahoneypat Thanks for that wonderful video. I have implemented skip and offset method while pulling data from API. Earlier it used to limit 100 records but using this I am able to load complete data. Next challenge is refresh doesn't work in Power BI Service. Returns error saying This dataset contains dynamic data source. 

      Any suggestions so that refresh works..Please note I already tried approach mentioned in below link

      https://blog.crossjoin.co.uk/2016/08/23/web-contents-m-functions-and-dataset-refresh-errors-in-power-bi/

      but then after adding skip and top as query parameters.. it pulls same set of records everytime.. so this is not working for me. Do you suggest any other option which will resolve dynamic data source issue in Service?

  • JosephDodd's avatar
    JosephDodd
    Frequent Visitor
    3 years ago

    mahoneypat Hey Pat, can't you use the theories and code in this community post to update the Salesforce Report connector to automatically call the next set of rows until it returns blank?