Forum Discussion

shun888's avatar
shun888
Frequent Visitor
2 years ago
Solved

How to copy Data from Rest API with pagination?

Hi, 

 

I am trying to copy data using Data Pipelines from a Rest API into a dataware house. I am stuck at the pagination part. I am unable to get the pipeline to run through all data.

Here is a sample response from the API. Note that it gives a url for the next page and returns null when it is the last page:

 

{

"count": 656,

"next": "https://hi.abc.com/v3/customer/core/users/?limit=100&offset=100",

"previous": null,

"results": [

 

The question is how do I configure the set up so that it takes the "next": url and iterates it until it is null?

 

I tried the following set up and it did not work out. 

 

 

General Question
Need Help
pipelines
Tutorial Requests
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi shun888 

     

    Thanks for using Microsoft Fabric Community.

    As I understand that you are looking for how to use the pagination rules in Microsoft fabric for the data pipeline using the copy data activity from an API.

    APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.

    For more information please refer to this documentation: Implementing Pagination with the Copy Activity in Microsoft Fabric

     

    I hope this information helps. Please do let us know if you have any further questions.

     

    Thanks.

3 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi shun888 

     

    Thanks for using Microsoft Fabric Community.

    As I understand that you are looking for how to use the pagination rules in Microsoft fabric for the data pipeline using the copy data activity from an API.

    APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.

    For more information please refer to this documentation: Implementing Pagination with the Copy Activity in Microsoft Fabric

     

    I hope this information helps. Please do let us know if you have any further questions.

     

    Thanks.

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Microsoft Employee
      2 years ago

      Hi shun888 

       

      Glad that your query got resolved.

      Please continue using Fabric Community for any help regarding your queries.

       

      Thanks.

    • nik1031's avatar
      nik1031
      Frequent Visitor
      1 year ago

      What if my pagination is done via body within a POST request where I need to have "after" parameter? If I use "pagination rules" where I add queryparameters and set "after" to $.paging.next.after, how do I put this value back into my request? I've been digging around for this everywhere and could not find anything. The solution offered in ADF did not work