pulling data from a REST API via Linked Service & Copy object

KBD — Thu, 26 Jun 2025 21:05:58 GMT

Hello All:

I have been tasked with pulling data from a REST API , munging it and making it available for reporting (Power BI).

I have a the calls required working in POSTMAN and via Python from my desktop.

Now I need to move this processing to Azure.

I have gotten a couple of things to work.

Referring to the above:

Use a Web object to get a session token - required for all subsequent calls proof I have correct URL , User Id & Password.

Use Set objects to set variables with values from the results. Session Token is important.

Use Copy object to write out results

Web objects to simple calls to REST API to prove session token returned is valid.

This works fine. Progress

The REST copy data object is failing. I want to use Copy data because I read it can handle pagination.

Some of the result data sets require multiple calls. Larger than the record limit set on the API.

There are parameters on the Copy object for pagination. But we are not there yet.

POSTMAN generates this python code:

import requests import json url = "https://software.restapi.us/platformapi/core/system/applicationinfo/version" payload = {} headers = { 'Accept': 'application/json,text/html,application/xhtml+xml,application/xml', 'Content-Type': 'application/json', 'Authorization': 'XXXXX session-id=bigreddog5E383AAB611A6510' } response = requests.request("GET", url, headers=headers, data=payload) print(response.text)

this works. The REST API expects those headers.

This is what I created in data factory

Linked Service

{ "name": "SoftwareREST", "properties": { "annotations": [], "type": "RestService", "typeProperties": { "url": "https://software.restapi.us/platformapi/core/", "authenticationType": "Anonymous" } }, "Accept": "application/json,text/html,application/xhtml+xml,application/xml", "Content-Type": "application/json" }

Data Set

Default value is:

@concat('XXXXX session_id=', variables('XXXXXSessionToken') )

{ "name": "XXXXXRESTDataSetVersion_In", "properties": { "linkedServiceName": { "referenceName": "XXXXXREST", "type": "LinkedServiceReference" }, "parameters": { "Authorization": { "type": "string", "defaultValue": "@concat('XXXXX session_id=', variables('XXXXXSessionToken') )" } }, "annotations": [], "type": "RestResource", "typeProperties": { "relativeUrl": "system/applicationinfo/version" }, "schema": [] } }

When run I get the following error:

Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Rest call failed with client error, status code 401 Unauthorized, please check your activity settings.
Request URL: https://XXXXX.RESTAPI.us/platformapi/core/system/applicationinfo/version.
Response: ,Source=Microsoft.DataTransfer.ClientLibrary,'

Question:

How do I configure this call using the data factory objects?

What should the Linked Service look like? and what should the Source Data Set look like?

Many thanks for your kind attention to this matter.

Best Regards

Re: pulling data from a REST API via Linked Service & Copy object

KBD — Thu, 03 Jul 2025 15:27:40 GMT

In the end I did not do the parameterization on the linked service,

Set the Authentication Type on the linked service to Anonymous.

Then on the Copy Object in Additional Headers set the session key there:

Dynamic content looks like this:

WafaStudies out on YouTube has some videos which provide valuable pointers

topic pulling data from a REST API via Linked Service & Copy object in Pipelines

pulling data from a REST API via Linked Service & Copy object

Re: pulling data from a REST API via Linked Service & Copy object