Forum Discussion

alexbjorlig's avatar
alexbjorlig
Helper IV
2 years ago

Help with $top, when migrating from OData to web source

For reasons related to query-folding not working with OData, I'm migrating a dataflow to using 

Json.Document(Web.Contents(BaseUrl, [Query = CombinedParams]))

Everything works great.

However, when using OData queries are automatically limited with $top, when using the Query Model Editor.

Can I somehow replicate this - is there a way to detect if we are running in preview mode?
Need Help

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi alexbjorlig 

     

    "Can I somehow replicate this - is there a way to detect if we are running in preview mode?", I don't quite understand what you mean by this statement, could you please explain it a bit more? Thank you for your time in advance.

     

    Best Regards,
    Yulia Xu

  • alexbjorlig's avatar
    alexbjorlig
    Helper IV
    2 years ago

    Ok, I will try to explain better Anonymous .

    I'm trying to pull data from a web-api with pagination. To fetch this data, I have written a recursive M function (it looks at nextCursor and calls itself when there is anextCursor). It works and can pull in all the data with no problem.

    However, when opening up the dataflow in the Power Query web editor, I notice that it's loading all data from the API, millions of rows (where the OData connector injects a $top 1000) 😲 I would not expect that, in the web editor preview I just need a preview of the data - not a million rows.

    So my question is when fetching data from a web source, is it possible to detect if the query should limit itself to only 1000 rows (just like the OData connector is doing today already)?