Forum Discussion

alexbjorlig's avatar
alexbjorlig
Helper IV
2 years ago

Help with $top, when migrating from OData to web source

For reasons related to query-folding not working with OData, I'm migrating a dataflow to using  Json.Document(Web.Contents(BaseUrl, [Query = CombinedParams])) Everything works great. However,...
Need Help
alexbjorlig's avatar
alexbjorlig
Helper IV
2 years ago

Ok, I will try to explain better Anonymous .

I'm trying to pull data from a web-api with pagination. To fetch this data, I have written a recursive M function (it looks at nextCursor and calls itself when there is anextCursor). It works and can pull in all the data with no problem.

However, when opening up the dataflow in the Power Query web editor, I notice that it's loading all data from the API, millions of rows (where the OData connector injects a $top 1000) 😲 I would not expect that, in the web editor preview I just need a preview of the data - not a million rows.

So my question is when fetching data from a web source, is it possible to detect if the query should limit itself to only 1000 rows (just like the OData connector is doing today already)?