Managing pagination for a Web API connector used in a dataflow gen2

[pmscorca]

Hi,

I need to create a dataflow gen2 that has to use a Web API connector to read Dynamics 365 Business Central data.

This connector requires to manage the pagination else it allows to read only the first 20.000 rows.

I've read that I need to manage the pagination by a Power Query function, while with a data pipeline isn't required a such management.

Does it exist a more easy manner in order to use a Web API connector in a dataflow gen2 to read an entire BC table?

Many thanks

[v-lgarikapat]

Hi @pmscorca , 

Thanks for reaching out to the Microsoft fabric community forum.

Power Query Function (Manual Pagination)

Create a recursive Power Query function that:

Makes the initial API call

Checks for @odata.nextLink

Continues fetching until no more pages remain

You’ll need to use Json.Document(Web.Contents(...)) and loop through pages using List.Generate or similar constructs.

This method works well but requires careful error handling and throttling logic.Use

Data Pipeline  Approch 

Data Pipelines in Fabric handle pagination automatically for many connectors, including Business Central. If your use case allows, this is the simpler and more scalable route.

Pipelines abstract away the paging logic

You can dump the full dataset into a Lakehouse or Warehouse

Then use Dataflow Gen2 to transform it downstream

Solved: Power Query Rest API Paging with @odata.nextLink - Microsoft Fabric Community

TripPin 5 - Paging - Power Query | Microsoft Learn

Solved: Fabric Dataflow Gen2's Business Central connector ... - Microsoft Fabric Community

We truly appreciate your continued engagement and thank you for being an active and valued member of the community.

If you're still experiencing any challenges, please don’t hesitate to reach out  we’d be more than happy to assist you further.

We look forward to hearing from you.

Best regards,
Lakshmi

[pmscorca]

Ok, but in some scenarios it could be more useful a Dataflow gen2 , on othen ones a Data pipeline.

So, it shoul be appreciated to use a managed pagination for a Web API connctor in a Dataflow gen2 avoinding the manual pagination.