Forum Discussion
Copy Data Pagination - Offset in Request Body
Hi all,
I am using the Copy Data activity to bring in a full data table into my bronze layer in Fabric. I am connecting to an API where the pagination method is using an offset in the request body. I can get the total number of pages (and therefore the max value of the offset figure and more) in the table easily using a POST call and a define variable activity. Additionally, each POST call returns the 'pages remaining' in the return body. What I need to know is can we paginate through the records using the Copy Data activity in the request body?
Let me know if I need to provide any further information.
Thanks,
Lewis
Hi Anonymous ,
If the API uses the offset in the request body for paging, you can implement the paging logic in the following ways:
Use the Define Variable activity to initialize the offset (e.g. offset = 0). Use the Until activity to loop through calls to the API until all page data has been fetched.For more details, you can refer to below document:
Set Variable activity - Microsoft Fabric | Microsoft Learn
Until activity - Microsoft Fabric | Microsoft Learn
Best Regards,
Adamk KongIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
1 Reply
Hi Anonymous ,
If the API uses the offset in the request body for paging, you can implement the paging logic in the following ways:
Use the Define Variable activity to initialize the offset (e.g. offset = 0). Use the Until activity to loop through calls to the API until all page data has been fetched.For more details, you can refer to below document:
Set Variable activity - Microsoft Fabric | Microsoft Learn
Until activity - Microsoft Fabric | Microsoft Learn
Best Regards,
Adamk KongIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.