topic Re: Copy Activity REST API - Paging in Pipelines

Copy Activity REST API - Paging

bcarlson_f — Fri, 11 Apr 2025 20:10:15 GMT

I am having a problem with paging in ADF. What i need to do is to make an initial call to a REST API, and then use "offset" that comes from the body as a parameter in subsequent calls. I can get it working, as in it doesn't fail, but it doesn't pull back the correct data. I think what it's doing is removing my other parameters. How do I get this scenario working?

First call should be:

"https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200&offset="

"https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200"

I've tested and it doesn't matter which.

Subsequent calls should be:

"https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200&offset="

+ the offset from the results body.

First, I don't know how to verify because the copy activity input just says "type": "RestSource", but I think what ADF is doing is to send:

"https://api.hubapi.com/email/public/v1/events?offset=" & the offset from the results,

i.e. removing the "limit=1000&startTimestamp=1744243200"

Even though I can't figure out how to see the URL in ADF, I can duplicate the results by using a REST testing tool. The problem is that data is coming back that occurs before the specified timestamp.

I've followed the documentation in this page: https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support (Example 1) but I get an error about invalid character, 7b. I've also tried about 100 different combinations from other examples I've found on YT and Google, but can't find anything that's exactly this besides the MS documentation linked.

Here's one example of something that works, but pulls the wrong data:

baseUrl = "https://api.hubapi.com/email/public/v1/"

relativeUrl = "events?limit=1000&startTimestamp=1744243200&offset=" (Adding "{offset}" as the example suggests causes the 7b error)

Any help would be appreciated. Even a link to something that goes deep into explaining what's going on behind the scenes with this so I can troubleshoot. Unfortunately, most examples just explain at a high level and are not giving me what I need to troubleshoot this, such as seeing the actual URL that ADF is sending.

It's troubling because I know exactly what I need to do, but not how to make ADF do it.

Thanks.

Re: Copy Activity REST API - Paging

lbendlin — Sat, 12 Apr 2025 16:29:59 GMT

7b is "left curly bracket". You don't seem to specify an actual offset value.

Re: Copy Activity REST API - Paging

bcarlson_f — Sat, 12 Apr 2025 21:42:32 GMT

Thanks. That specific error comes from when I try to follow the example in the linked ADF Documentation page. If you take a look at example 1, they show an image of the configuration that has a curly bracket. When I try that though, I get the error. The examples I have don't have it since it didn't work.

Re: Copy Activity REST API - Paging

v-sathmakuri — Wed, 16 Apr 2025 16:05:13 GMT

Hi bcarlson_f ,

Thank you for reaching out to Microsoft Fabric Community.

As this is microsoft fabrics forum, i would suggest raising a ticket is ADF forum https://techcommunity.microsoft.com/category/azuredatabases/discussions/azuredatafactory

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!

Thank you!!

Re: Copy Activity REST API - Paging

bcarlson_f — Thu, 17 Apr 2025 13:55:40 GMT

Thanks. I've posted over there. For me, the breadcrumbs say this is a Data Factory forum but I do see fabric in the URL.