topic Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL. in Pipelines

REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

younghoon_kim — Tue, 12 Aug 2025 07:05:19 GMT

Hi. I've set up Copy Activity to pull data from an API, and having a hard time to paginate.

API looks like this :

https://serviceportal.telenorconnexion.com/iot/api/subscriptions/details?{some parameters}

This call's response contains "next" of next url to call, which looks like this :

{ ~~ "next": "/details?q=CO~~~", ~~ }

-> just the end part of first call.

so "next" doesn't give FULL url to call.

So I've created Connection to this : https://serviceportal.telenorconnexion.com/iot/api/subscriptions

and set Relative URL to details?{some parameters} , and the first call goes fine.

I've done my research to paginate "next", and AbsoluteUrl looks like my solution, so I set up like this:

Then this error pops up:

ErrorCode=RestCallFailedWithServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with server error, status code 503 ServiceUnavailable, please check your activity settings and remote server issues. Request URL: https://serviceportal.telenorconnexion.com/details?{some_parameter~~}

It pulls "next" value correctly, but call is being made to https://serviceportal.telenorconnexion.com/ , not https://serviceportal.telenorconnexion.com/iot/api/subscriptions that I've set the connection to.

Looks like in pagination, it pulls baseUrl(not using the connection is set to) from Connection somehow, then use to paginate.

I've tried making connection to https://serviceportal.telenorconnexion.com/ and start with

/iot/api/subscriptions/details/?q=~~~ in Relative URL, but shows same error. I can't find a way to put "/iot/api/subscriptions" in front of response's "next".

Anyone can help?

Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

Vinodh247 — Fri, 15 Aug 2025 14:23:54 GMT

When you choose AbsoluteUrl in the pagination rules, the REST connector assumes that the "next" value returned from the API is the full URL to call next. It ignores both:

The "Base URL" you configured in the linked service
Any Relative URL you configured in the Copy Activity

Since your "next" only contains the tail path, the connector starts building the request from the domain root, which explains why /iot/api/subscriptions disappears. Given your "next" is /details?... and you always need /iot/api/subscriptions before it, the RelativeUrl + concat() expression approach is the cleanest and does not require redesigning the pipeline.

"paginationRules": { "absoluteUrl": null, "relativeUrl": "@concat('/iot/api/subscriptions', json(activity('CopyDataFromAPI').output.response)['next'])" }

Where:

Replace CopyDataFromAPI with your activity name
Ensure the json() path correctly extracts "next"

If you want, I can give you the exact JSON snippet for the Copy Activity Source that will prepend /iot/api/subscriptions on every "next" call so the connector never drops that segment again.

Please 'Kudos' and 'Accept as Solution' if this answered your query.

Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

younghoon_kim — Mon, 18 Aug 2025 03:38:52 GMT

can't apply your suggestion.

@concat('/iot/api/subscriptions', json(activity('copyFromAPIToLakehouseBronzePagination').output)['next'])

The output of activity 'copyFromAPIToLakehouseBronzePagination' can't be referenced since it is either not an ancestor to the current activity or does not exist

Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

v-sdhruv — Fri, 22 Aug 2025 12:25:20 GMT

Hi younghoon_kim ,
If your API can be configured to return the full URL in "next" like:
Then you can use:

"paginationRules": { "absoluteUrl": "$.next" }

If the API only returns a root-relative path like

/details?q=CO~~~"

then you cannot use absolute url directly. Instead, Use python to resolve the relative "next" using url join:

from urllib.parse import urljoin base = "https://serviceportal.telenorconnexion.com/iot/api/subscriptions/"

Hope this helps!

Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

v-sdhruv — Mon, 25 Aug 2025 11:26:48 GMT

Hi younghoon_kim ,
Just wanted to check if you got a chance to review the suggestions provided and whether that helped you resolve your query?

Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

v-sdhruv — Thu, 28 Aug 2025 11:54:40 GMT

Hi @younghoon_kim ,
Just wanted to check if you got a chance to review the suggestions provided and whether that helped you resolve your query

Re: REST API connection pagination inside Copy Activity Source, AbsoluteUrl points to wrong URL.

younghoon_kim — Mon, 01 Sep 2025 00:57:49 GMT

API provider seems to be unable to change API unfortunately..