How to copy Data from Rest API with pagination?

[shun888]

Hi, 

I am trying to copy data using Data Pipelines from a Rest API into a dataware house. I am stuck at the pagination part. I am unable to get the pipeline to run through all data.

Here is a sample response from the API. Note that it gives a url for the next page and returns null when it is the last page:

{

"count": 656,

"next": "https://hi.abc.com/v3/customer/core/users/?limit=100&offset=100",

"previous": null,

"results": [

The question is how do I configure the set up so that it takes the "next": url and iterates it until it is null?

I tried the following set up and it did not work out. 

[v-cboorla-msft · Accepted solution]

Hi @shun888 

Thanks for using Microsoft Fabric Community.

As I understand that you are looking for how to use the pagination rules in Microsoft fabric for the data pipeline using the copy data activity from an API.

APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.

For more information please refer to this documentation: Implementing Pagination with the Copy Activity in Microsoft Fabric

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

[nik1031]

What if my pagination is done via body within a POST request where I need to have "after" parameter? If I use "pagination rules" where I add queryparameters and set "after" to $.paging.next.after, how do I put this value back into my request? I've been digging around for this everywhere and could not find anything. The solution offered in ADF did not work

[v-cboorla-msft]

Hi @shun888 

Glad that your query got resolved.

Please continue using Fabric Community for any help regarding your queries.

Thanks.