User Profile
bcarlson_f
Regular Visitor
Joined 1 year ago
User Widgets
Contributions
Copy Activity REST API - Paging
I am having a problem with paging in ADF. What i need to do is to make an initial call to a REST API, and then use "offset" that comes from the body as a parameter in subsequent calls. I can get it working, as in it doesn't fail, but it doesn't pull back the correct data. I think what it's doing is removing my other parameters. How do I get this scenario working? First call should be: "https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200&offset=" or "https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200" I've tested and it doesn't matter which. Subsequent calls should be: "https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200&offset=" + the offset from the results body. First, I don't know how to verify because the copy activity input just says "type": "RestSource", but I think what ADF is doing is to send: "https://api.hubapi.com/email/public/v1/events?offset=" & the offset from the results, i.e. removing the "limit=1000&startTimestamp=1744243200" Even though I can't figure out how to see the URL in ADF, I can duplicate the results by using a REST testing tool. The problem is that data is coming back that occurs before the specified timestamp. I've followed the documentation in this page: https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support (Example 1) but I get an error about invalid character, 7b. I've also tried about 100 different combinations from other examples I've found on YT and Google, but can't find anything that's exactly this besides the MS documentation linked. Here's one example of something that works, but pulls the wrong data: baseUrl = "https://api.hubapi.com/email/public/v1/" relativeUrl = "events?limit=1000&startTimestamp=1744243200&offset=" (Adding "{offset}" as the example suggests causes the 7b error) Any help would be appreciated. Even a link to something that goes deep into explaining what's going on behind the scenes with this so I can troubleshoot. Unfortunately, most examples just explain at a high level and are not giving me what I need to troubleshoot this, such as seeing the actual URL that ADF is sending. It's troubling because I know exactly what I need to do, but not how to make ADF do it. Thanks.
