Enable pagination and parameterization in Dataflow Gen2 for REST API

[fbcideas_migusr]

We have a number of customers who are using online services. As we supply data warehouses for these customers, getting data from these online services is part of our daily routine.

Luckily most online services offer a REST Api to get the data from. Now the smaller API calls usually have one 'page' of data and we can use the dataflow to get the data, unnest the Json data and process it. But larger datasets usually have pagination built in.

Currently the Dataflow Gen2 doesn't support pagination.

To be able to read data from an API, we'd skip over to the mapping data flow in Data Factory. This works very well but isn't available in Fabric anymore. So we need to build notebooks to read data from the API, unnest it and write into the OneLake. But it's all manual work whereas the dataflows automatically detect the nesting and unnest the data. Something that's not only very cool to demo to customers but also saves a huge amount of work when getting data from the API. The amount of work I've done to get data from an API into a normalized format to be able to process it, more than I'd care to know ;).

Not only would it really help to be able to page through the results, some API's require parameters to limit the dataset or even to start getting data. For example, one API I'm using requires a specific ID as a parameter before it starts to return data.

It's not that we can't work without this functionality, but having it would seriously help! And as mentioned, really fits into the philosophy of Fabric in making data easily available in a low-code/no-code environment.

As an example: http://ergast.com/api/f1/2008/5/results This endpoint offers both pagination and parameterized input.