REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

[younghoon_kim]

Hi. I've set up Copy Activity to pull data from an API, and having a hard time to paginate.

API looks like this :

https://serviceportal.telenorconnexion.com/iot/api/subscriptions/details?{some parameters}

This call's response contains "next" of next url to call, which looks like this :

{
  ~~
  "next": "/details?q=CO~~~",
  ~~
}

   -> just the end part of first call.

so "next" doesn't give FULL url to call. 

So I've created Connection to this : https://serviceportal.telenorconnexion.com/iot/api/subscriptions

and set Relative URL to details?{some parameters} , and the first call goes fine. 

I've done my research to paginate "next", and AbsoluteUrl looks like my solution, so I set up like this:

Then this error pops up:

ErrorCode=RestCallFailedWithServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with server error, status code 503 ServiceUnavailable, please check your activity settings and remote server issues.
Request URL: https://serviceportal.telenorconnexion.com/details?{some_parameter~~} 

It pulls "next" value correctly, but call is being made to https://serviceportal.telenorconnexion.com/ , not https://serviceportal.telenorconnexion.com/iot/api/subscriptions that I've set the connection to. 

Looks like in pagination, it pulls baseUrl(not using the connection is set to) from Connection somehow, then use to paginate. 

I've tried making connection to https://serviceportal.telenorconnexion.com/  and start with 

/iot/api/subscriptions/details/?q=~~~ in Relative URL, but shows same error. I can't find a way to put "/iot/api/subscriptions" in front of response's "next". 

Anyone can help?

[v-sdhruv]

Hi @younghoon_kim ,
Just wanted to check if you got a chance to review the suggestions provided and whether that helped you resolve your query

[younghoon_kim]

API provider seems to be unable to change API unfortunately..

[v-sdhruv]

Hi @younghoon_kim ,
Just wanted to check if you got a chance to review the suggestions provided and whether that helped you resolve your query?

[v-sdhruv]

Hi @younghoon_kim ,
If your API can be configured to return the full URL in "next" like:
Then you can use:

"paginationRules": {
  "absoluteUrl": "$.next"
}

If the API only returns a root-relative path like 

/details?q=CO~~~"

 then you cannot use absolute url directly. Instead, Use python to resolve the relative "next" using  url join:

from urllib.parse import urljoin
base = "https://serviceportal.telenorconnexion.com/iot/api/subscriptions/"

Hope this helps!

[Vinodh247]

When you choose AbsoluteUrl in the pagination rules, the REST connector assumes that the "next" value returned from the API is the full URL to call next. It ignores both:

• The "Base URL" you configured in the linked service
• Any Relative URL you configured in the Copy Activity

Since your "next" only contains the tail path, the connector starts building the request from the domain root, which explains why /iot/api/subscriptions disappears. Given your "next" is /details?... and you always need /iot/api/subscriptions before it, the RelativeUrl + concat() expression approach is the cleanest and does not require redesigning the pipeline.

"paginationRules": {
    "absoluteUrl": null,
    "relativeUrl": "@concat('/iot/api/subscriptions', json(activity('CopyDataFromAPI').output.response)['next'])"
}

Where:

• Replace CopyDataFromAPI with your activity name

• Ensure the json() path correctly extracts "next"

 If you want, I can give you the exact JSON snippet for the Copy Activity Source that will prepend /iot/api/subscriptions on every "next" call so the connector never drops that segment again.

Please 'Kudos' and 'Accept as Solution' if this answered your query.

[younghoon_kim · Accepted solution]

can't apply your suggestion.

@concat('/iot/api/subscriptions', json(activity('copyFromAPIToLakehouseBronzePagination').output)['next'])

The output of activity 'copyFromAPIToLakehouseBronzePagination' can't be referenced since it is either not an ancestor to the current activity or does not exist