topic Re: Copying Paginated data Next page function not working in Data Engineering

Copying Paginated data Next page function not working

Housden996 — Thu, 28 Nov 2024 10:52:55 GMT

I am workin with a paginated API and it uses unique urls to pull in the following page instead of a sequential page number.

My copy function works but it isnt pulling in the following page, my following copy funcion is working as follows:

Pagination rules:

NOTE: I fixest the end condition so its now empty and it still doesnt work.

The trailing end of the JSON whic contains the next URL which i need to read is as follows:

Any idea what I am doing wrong here? do I need to get this as a variable and pass into a following copy function or should this just work?

Thanks

Re: Copying Paginated data Next page function not working

Anonymous — Fri, 29 Nov 2024 01:38:11 GMT

HI Housden996,

According to your description and snapshot, it seems like your requirement is mroe realted ot azure data factory pipeline invoke api. For this scenario, I'd like to suggest you post to that forum to get better support.

Azure Data Factory - Microsoft Q&A

Regards,

Xiaoxin Sheng

Re: Copying Paginated data Next page function not working

Housden996 — Fri, 29 Nov 2024 08:52:00 GMT

Thank you I thought this would be the solution, wanted to try here before diving into a new community. 🙂

Re: Copying Paginated data Next page function not working

Anonymous — Wed, 04 Dec 2024 02:07:50 GMT

HI Housden996,

In fact, paginated API should base on its definitions. (normally it checks the request header, but it also saw some API checked on request body) You should modify the function and request to send the correspond contents based on the API document and definitions.

For example:

Some of API returned page number in the response, some of them show you the next id, and also it can be directly returned the next link URL.

Regards,
Xiaoxin Sheng