Forum Discussion

shun888's avatar
shun888
Frequent Visitor
2 years ago
Solved

How to copy Data from Rest API with pagination?

Hi,    I am trying to copy data using Data Pipelines from a Rest API into a dataware house. I am stuck at the pagination part. I am unable to get the pipeline to run through all data. Here is a ...
General Question
Need Help
pipelines
Tutorial Requests
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi shun888 

     

    Thanks for using Microsoft Fabric Community.

    As I understand that you are looking for how to use the pagination rules in Microsoft fabric for the data pipeline using the copy data activity from an API.

    APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.

    For more information please refer to this documentation: Implementing Pagination with the Copy Activity in Microsoft Fabric

     

    I hope this information helps. Please do let us know if you have any further questions.

     

    Thanks.

v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi shun888 

 

Thanks for using Microsoft Fabric Community.

As I understand that you are looking for how to use the pagination rules in Microsoft fabric for the data pipeline using the copy data activity from an API.

APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.

For more information please refer to this documentation: Implementing Pagination with the Copy Activity in Microsoft Fabric

 

I hope this information helps. Please do let us know if you have any further questions.

 

Thanks.

nik1031's avatar
nik1031
Frequent Visitor
1 year ago

What if my pagination is done via body within a POST request where I need to have "after" parameter? If I use "pagination rules" where I add queryparameters and set "after" to $.paging.next.after, how do I put this value back into my request? I've been digging around for this everywhere and could not find anything. The solution offered in ADF did not work