Web API: Headers in Web-Activity vs. Headers in Copy data-Activity

chris__1 — Wed, 05 Jul 2023 13:44:13 GMT

Hello,

I want to query an API with Data Pipelines.

The API expects to first generate a Token and then use the Token as Authorization to make the desired query.

First part works fine - I get a valid Token.

As next step I first tried using the Copy data Activity. I set up the connection (is working), set connection type to http and POST method, file format json. This should all be fine. The problem seems to be the "Additional headers" part and the "Request body" part.

Looking at the Code of the Copy data Activity I get this. (See part starting at "additionalHeaders" - this is the problem):

"typeProperties": { "source": { "type": "JsonSource", "storeSettings": { "type": "HttpReadSettings", "requestMethod": "POST", "additionalHeaders": { "value": "\"Authorization\": {\n \"value\": \"@concat('Bearer ', variables('v_token'))\",\n \"type\": \"Expression\"\n },\n \"Content-Type\": \"application/json\",\n \"Accept\": \"application/json\"", "type": "Expression" }, "requestBody": "{\n \"sql\": \"SELECT * FROM table WHERE Date='2023-07-03'\"\n}", "requestTimeout": "" }, "formatSettings": { "type": "JsonReadSettings" }, "datasetSettings": { "annotations": [], "type": "Json", "typeProperties": { "location": { "type": "HttpServerLocation" } },

The Data Pipeline seems to be adding Backslashes \ I can't seem to get rid of!! This problem seems to exist in ADF as well: https://stackoverflow.com/questions/72095530/trouble-accessing-json-from-web-api-call-in-adf-and-inserting-in-sql-table/72128255#72128255

If use the Web-Activity I can use the dedicated fields for Headers - this gives me this Code which is working fine and the API replies accordingly:

"name": "Web2", "type": "WebActivity", "dependsOn": [ { "activity": "var for token", "dependencyConditions": [ "Succeeded" ] } ], "policy": { "timeout": "0.12:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "typeProperties": { "method": "POST", "headers": { "Authorization": { "value": "@concat('Bearer ', variables('v_token'))", "type": "Expression" }, "Content-Type": "application/json", "Accept": "application/json" }, "body": { "sql": "SELECT * FROM table WHERE Date='2023-07-03'" } }, "externalReferences": { "connection": "ed851474-77e8-41e5-917d-33171377856b" } }

However those dedicated Header Fields exist in the Web-Activity but not in the Copy-data activity!

So my question is: how do I have to enter the text in the Copy data activity to get the request formatted correctly for the API?

I could use the Web activity however I not sure how I would get the data to the Datawarehouse as the Web activity has not Destination section!

Any help is highly appreciated!

Re: Web API: Headers in Web-Activity vs. Headers in Copy data-Activity

GraceGu — Fri, 07 Jul 2023 01:55:15 GMT

Seems you are putting bearer token in additionalHeaders of Http connector , try additionalHeaders value in a whole string like @concat('Authorization: Bearer ',@variables('v_token'))

If you are using REST connector, payload is like below, where you can add fields.

Re: Web API: Headers in Web-Activity vs. Headers in Copy data-Activity

chris__1 — Fri, 07 Jul 2023 09:51:54 GMT

Thank you GraceGu ! Worked exactly as discribed when using the REST connector:

topic Re: Web API: Headers in Web-Activity vs. Headers in Copy data-Activity in Pipelines

Web API: Headers in Web-Activity vs. Headers in Copy data-Activity

Re: Web API: Headers in Web-Activity vs. Headers in Copy data-Activity

Re: Web API: Headers in Web-Activity vs. Headers in Copy data-Activity