Forum Discussion

bcarlson_f's avatar
bcarlson_f
Regular Visitor
1 year ago
Solved

Copy Activity REST API - Paging

I am having a problem with paging in ADF. What i need to do is to make an initial call to a REST API, and then use "offset" that comes from the body as a parameter in subsequent calls. I can get it working, as in it doesn't fail, but it doesn't pull back the correct data. I think what it's doing is removing my other parameters. How do I get this scenario working?

 

First call should be:

"https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200&offset="
or
"https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200"
I've tested and it doesn't matter which.
 
Subsequent calls should be:
"https://api.hubapi.com/email/public/v1/events?limit=1000&startTimestamp=1744243200&offset="
+ the offset from the results body.
 
First, I don't know how to verify because the copy activity input just says "type": "RestSource", but I think what ADF is doing is to send:
"https://api.hubapi.com/email/public/v1/events?offset=" & the offset from the results,
i.e. removing the "limit=1000&startTimestamp=1744243200"
Even though I can't figure out how to see the URL in ADF, I can duplicate the results by using a REST testing tool. The problem is that data is coming back that occurs before the specified timestamp.
 
I've followed the documentation in this page: https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support (Example 1) but I get an error about invalid character, 7b. I've also tried about 100 different combinations from other examples I've found on YT and Google, but can't find anything that's exactly this besides the MS documentation linked.
  
Here's one example of something that works, but pulls the wrong data:
baseUrl = "https://api.hubapi.com/email/public/v1/"
relativeUrl = "events?limit=1000&startTimestamp=1744243200&offset=" (Adding "{offset}" as the example suggests causes the 7b error)

 

Any help would be appreciated. Even a link to something that goes deep into explaining what's going on behind the scenes with this so I can troubleshoot. Unfortunately, most examples just explain at a high level and are not giving me what I need to troubleshoot this, such as seeing the actual URL that ADF is sending.
 
It's troubling because I know exactly what I need to do, but not how to make ADF do it.
 
Thanks.
Need Help
Pipeline

4 Replies

  • lbendlin's avatar
    lbendlin
    Super User
    1 year ago

    7b is "left curly bracket".  You don't seem to specify an actual offset value.

    • bcarlson_f's avatar
      bcarlson_f
      Regular Visitor
      1 year ago

      Thanks. That specific error comes from when I try to follow the example in the linked ADF Documentation page. If you take a look at example 1, they show an image of the configuration that has a curly bracket. When I try that though, I get the error. The examples I have don't have it since it didn't work.

  • v-sathmakuri's avatar
    v-sathmakuri
    Community Support
    1 year ago

    Hi bcarlson_f ,

     

    Thank you for reaching out to Microsoft Fabric Community.

     

    As this is microsoft fabrics forum, i would suggest raising a ticket is ADF forum https://techcommunity.microsoft.com/category/azuredatabases/discussions/azuredatafactory 

     

    If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

     

    Thank you!!

    • bcarlson_f's avatar
      bcarlson_f
      Regular Visitor
      1 year ago

      Thanks. I've posted over there. For me, the breadcrumbs say this is a Data Factory forum but I do see fabric in the URL.