Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
h4tt3n
Resolver II
Resolver II

How to overcome API call row limit?

Hello,

 

I am attempting to extract measurement data from an API with a limit of 1000 rows per call. Each row consists of a data value and a timestamp. I have set up a Power Query script that connects to the API and can retrieve any 1000 concecutive rows of data. 

 

What I need is an iterative script that retrieves the data I need in chunks of 1000 rows and merges them ino a single table. Below is my current script. The Source line is the one of interest here.

 

 

 

let
    Source = Json.Document(Web.Contents(""&url&"/TagsData/"&tagid&"/After?subscription-key="&key&"&after="&after&"&count="&count&"")),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"tag", "data"}, {"tag", "data"}),
    #"Removed Columns" = Table.RemoveColumns(#"Expanded Column1",{"tag"}),
    #"Expanded data" = Table.ExpandListColumn(#"Removed Columns", "data"),
    #"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"d", "v"}, {"d", "v"}),
    #"Changed Type" = Table.TransformColumnTypes(#"Expanded data1",{{"d", type datetime}, {"v", type number}})
in
    #"Changed Type"

 

 

 

 The body parameter "after" sets the datetime after which you want data from. The parameter "count" sets the number of rows, and is limited to 1000.

 

One challenge is that the API is not paginated, so I will need to keep track of timestamps and number of returned rows for each call.

 

I have been working with PowerBI professionally for the last year, but my Power Query scripting skills are not good enough for this challenge. Thanks in advance for any help.

 

Cheers, Mike

4 REPLIES 4
JosephDodd
Frequent Visitor

@mahoneypat Hey Pat, can't you use the theories and code in this community post to update the Salesforce Report connector to automatically call the next set of rows until it returns blank?

mahoneypat
Employee
Employee

Instead of pagination, you can also use List.Numbers along with Skip or Offset in the web calls to overcome the row limit.  This video walks through it.

Power BI - Tales From The Front - REST APIs - YouTube

 

Regards,

Pat





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


@mahoneypat Thanks for that wonderful video. I have implemented skip and offset method while pulling data from API. Earlier it used to limit 100 records but using this I am able to load complete data. Next challenge is refresh doesn't work in Power BI Service. Returns error saying This dataset contains dynamic data source. 

Any suggestions so that refresh works..Please note I already tried approach mentioned in below link

https://blog.crossjoin.co.uk/2016/08/23/web-contents-m-functions-and-dataset-refresh-errors-in-power...

but then after adding skip and top as query parameters.. it pulls same set of records everytime.. so this is not working for me. Do you suggest any other option which will resolve dynamic data source issue in Service?

Jimmy801
Community Champion
Community Champion

Hello @h4tt3n 

 

there are a lot of examples of how to do in this forum.

They use all List.Generate where you have a parameter when to stop to generate to api calls like giving an error or empty page.

just start out to search for api List.Generate.

If you have any specific question feel free to come back


If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

March 2024 PBI Gallery Image

Power BI Monthly Update - March 2024

Check out the March 2024 Power BI update to learn about new features.

Top Solution Authors
Top Kudoed Authors