topic Re: How to copy Data from Rest API with pagination? in Pipelines

How to copy Data from Rest API with pagination?

shun888 — Mon, 26 Feb 2024 15:48:56 GMT

Hi,

I am trying to copy data using Data Pipelines from a Rest API into a dataware house. I am stuck at the pagination part. I am unable to get the pipeline to run through all data.

Here is a sample response from the API. Note that it gives a url for the next page and returns null when it is the last page:

{

"count": 656,

"next": "https://hi.abc.com/v3/customer/core/users/?limit=100&offset=100",

"previous": null,

"results": [

The question is how do I configure the set up so that it takes the "next": url and iterates it until it is null?

I tried the following set up and it did not work out.

Re: How to copy Data from Rest API with pagination?

v-cboorla-msft — Tue, 27 Feb 2024 06:24:30 GMT

Hi shun888

Thanks for using Microsoft Fabric Community.

As I understand that you are looking for how to use the pagination rules in Microsoft fabric for the data pipeline using the copy data activity from an API.

APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.

For more information please refer to this documentation: Implementing Pagination with the Copy Activity in Microsoft Fabric

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

Re: How to copy Data from Rest API with pagination?

v-cboorla-msft — Tue, 27 Feb 2024 14:34:50 GMT

Hi shun888

Glad that your query got resolved.

Please continue using Fabric Community for any help regarding your queries.

Thanks.

Re: How to copy Data from Rest API with pagination?

nik1031 — Thu, 21 Nov 2024 17:03:43 GMT

What if my pagination is done via body within a POST request where I need to have "after" parameter? If I use "pagination rules" where I add queryparameters and set "after" to $.paging.next.after, how do I put this value back into my request? I've been digging around for this everywhere and could not find anything. The solution offered in ADF did not work