Copy Data Pagination - Offset in Request Body

[Jameopw]

Hi all,

I am using the Copy Data activity to bring in a full data table into my bronze layer in Fabric. I am connecting to an API where the pagination method is using an offset in the request body. I can get the total number of pages (and therefore the max value of the offset figure and more) in the table easily using a POST call and a define variable activity. Additionally, each POST call returns the 'pages remaining' in the return body. What I need to know is can we paginate through the records using the Copy Data activity in the request body?

Let me know if I need to provide any further information.

Thanks,

Lewis

[v-kongfanf-msft · Accepted solution]

Hi @Jameopw ,

If the API uses the offset in the request body for paging, you can implement the paging logic in the following ways:
Use the Define Variable activity to initialize the offset (e.g. offset = 0). Use the Until activity to loop through calls to the API until all page data has been fetched.

For more details, you can refer to below document:

Set Variable activity - Microsoft Fabric | Microsoft Learn

Until activity - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.