Forum Discussion
Managing pagination for a Web API connector used in a dataflow gen2
Hi pmscorca ,
Thanks for reaching out to the Microsoft fabric community forum.
Power Query Function (Manual Pagination)
Create a recursive Power Query function that:
Makes the initial API call
Checks for @odata.nextLink
Continues fetching until no more pages remain
You’ll need to use Json.Document(Web.Contents(...)) and loop through pages using List.Generate or similar constructs.
This method works well but requires careful error handling and throttling logic.Use
Data Pipeline Approch
Data Pipelines in Fabric handle pagination automatically for many connectors, including Business Central. If your use case allows, this is the simpler and more scalable route.
Pipelines abstract away the paging logic
You can dump the full dataset into a Lakehouse or Warehouse
Then use Dataflow Gen2 to transform it downstream
Solved: Power Query Rest API Paging with @odata.nextLink - Microsoft Fabric Community
TripPin 5 - Paging - Power Query | Microsoft Learn
Solved: Fabric Dataflow Gen2's Business Central connector ... - Microsoft Fabric Community
We truly appreciate your continued engagement and thank you for being an active and valued member of the community.
If you're still experiencing any challenges, please don’t hesitate to reach out we’d be more than happy to assist you further.
We look forward to hearing from you.
Best regards,
Lakshmi
Ok, but in some scenarios it could be more useful a Dataflow gen2 , on other ones a Data pipeline.
So, it should be appreciated to use a managed pagination for a Web API connctor in a Dataflow gen2 avoinding the manual pagination.
Hi pmscorca ,
Thanks for the follow-up questions
In many scenarios, Dataflow Gen2 is the preferred choice for lightweight ingestion and transformation workflows, especially when direct integration with Lakehouses is needed. However, when consuming data from paginated REST APIs, the lack of native managed pagination becomes a significant limitation.
Currently, users must implement manual pagination using recursive Power Query M scripts, which can be complex, error-prone, and difficult to maintain particularly for APIs using cursor based tokens, header-driven continuation, or dynamic termination conditions.
This appears to be a current limitation in Power BI. I would recommend submitting this scenario as a feature request on the official Power BI Ideas forum. The Product Team actively monitors suggestions there, and items with strong community support are often considered for future enhancements.
You can submit your idea here
We appreciate your collaboration and support!
Best Regards,Lakshmi.Hi, thanks for your reply.
I've created this idea --> Automatic pagination management for a dataflow gen2 that uses the Web API connector as a source
Please to vote it, thanks
