topic Copy Data Pagination - Offset in Request Body in Pipelines

Copy Data Pagination - Offset in Request Body

Anonymous — Wed, 19 Feb 2025 09:30:58 GMT

Hi all,

I am using the Copy Data activity to bring in a full data table into my bronze layer in Fabric. I am connecting to an API where the pagination method is using an offset in the request body. I can get the total number of pages (and therefore the max value of the offset figure and more) in the table easily using a POST call and a define variable activity. Additionally, each POST call returns the 'pages remaining' in the return body. What I need to know is can we paginate through the records using the Copy Data activity in the request body?

Let me know if I need to provide any further information.

Thanks,

Lewis

Re: Copy Data Pagination - Offset in Request Body

Anonymous — Thu, 20 Feb 2025 07:55:18 GMT

Hi Anonymous ,

If the API uses the offset in the request body for paging, you can implement the paging logic in the following ways:
Use the Define Variable activity to initialize the offset (e.g. offset = 0). Use the Until activity to loop through calls to the API until all page data has been fetched.

For more details, you can refer to below document:

Set Variable activity - Microsoft Fabric | Microsoft Learn

Until activity - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.