topic Re: Copy Data Activity with REST API POST - Issue with request body and escape characters in Data Engineering

Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Wed, 30 Jul 2025 18:42:59 GMT

I am trying to use a copy data activity to get data from a REST API using a POST call and copy it to a lakehouse. The problem is that the copy data activity adds escape characters to the double quotes \" which makes the body unreadable by the API. I tried to use variables and concatenating and replacing \" with " but nothing seems to work.

Is there a solution to this issue?

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

apturlov — Thu, 31 Jul 2025 00:20:30 GMT

@Anonymous Can you post a screenshot of your COPY activity configuration for the body of your POST request? It sounds from your description that something is missing or misconfigured.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

lbendlin — Thu, 31 Jul 2025 01:29:23 GMT

How does your accept header look like?

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Thu, 31 Jul 2025 13:09:43 GMT

This is my configuration, the web connection works with other API calls with GET and the same exact call I am doing here works with Postman.

As you can see in the input screenshot, the activity adds a bunch of escape characters.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Thu, 31 Jul 2025 13:11:15 GMT

Here is my configuration

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

apturlov — Thu, 31 Jul 2025 18:09:04 GMT

@Anonymous I don't think that COPY activity did anything wrong. The encoding on the wire is correct and is based on how the REST connector handles content when the content-type is application/json. On the other hand, response 406 does not indicate an incorrect request, which would be 400 Bad Request. I guess, something is wrong based on 406 response status, but I would not blindly blame the COPY activity. To prove that, I did a quick experiment with a public REST API using a POST with a JSON body. As you would guess, the experiement was successful. See below the results:

As you can notice the encoding in the activity input is exactly the same as in your case, which is how a well-formed JSON looks like on the wire, and the response is also correct.

However, getting back to 406 response, it usually means that the request is missing an accept header that should indicate a content type that the response should be generated for. It seams that your API is very picky about that header and in your configuration it's missing. Try to add accept: application/json header to your COPY activity if you expect a JSON response, or another proper type, and verify if it works correctly this time.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Thu, 31 Jul 2025 18:24:10 GMT

Hello, thank you for your response. I tried adding accept but it says it's set to application/json by default anyway. I still get the same error.

These headers work in postman for the same request

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

apturlov — Thu, 31 Jul 2025 19:51:45 GMT

It is correct that the Accept header is not usually needed when the content-type header is specified. In my previous example above I did not need the Accept header. From information you provided so far it's difficult to diagnose the precise problem. You body and header look absolutely correctly from the activity input screenshot. For your original question about the body encoding the answer is yes, the encoding of the body is correct. But you say that the API returns an error. Can you post the output from your activity run or maybe a response that you get when you test with other tools like Postman?

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

v-venuppu — Mon, 04 Aug 2025 13:07:53 GMT

Hi @Anonymous ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @apturlov @lbendlin for the prompt response.

Can you please share the details as requested by @apturlov , so that it will be helpful for us to solve the issue.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Mon, 04 Aug 2025 14:18:41 GMT

Hello, here is the output from postman when I use the correct body and when I use the body from the copy activity with escape characters.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Mon, 04 Aug 2025 14:21:31 GMT

Hello, yes I just replied to @apturlov with the output I see in Postman for the same call

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

apturlov — Tue, 05 Aug 2025 16:28:06 GMT

Hi @Anonymous, thanks for your reply. I don't think the ouput from Postman is helpful with debugging issue in Fabric. I was asking about the output from your pipeline run in Fabric to see specifically what error was returned by the API.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

Anonymous — Tue, 05 Aug 2025 17:10:03 GMT

Hi, yes this is the error given by the copy activity in the pipeline. The same call works using a Web Activity but I still want to use copy activity because the web activity fails when the response is larger than 4MB

Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 406 NotAcceptable, please check your activity settings.
Request URL: https://*************/rest/2.0/outputModule/export/json.
Response: {"statusCode":406,"titleMessage":"Not Acceptable","userMessage":"HTTP 406 Not Acceptable"},Source=Microsoft.DataTransfer.ClientLibrary,'

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

apturlov — Wed, 06 Aug 2025 18:47:33 GMT

@Anonymous I agree that using a Copy activity is a better choice in this case compared to a Web activity. But something just does not add up in this picture: Web activity can execute the call, but the Copy activity fails. Both activities should be using the same REST connector with the same request settings. Can you compare settings of Web and Copy activities and make sure both are configured the same way? You could also paste a schreenshot of your Web activity settings here for a second check.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

v-venuppu — Mon, 11 Aug 2025 07:45:19 GMT

Hi @Anonymous ,

Have you compared the settings..?Can you also please paste a schreenshot of your Web activity settings here for a second check as requested by @apturlov .

Thank you.

Re: Copy Data Activity with REST API POST - Issue with request body and escape characters

v-venuppu — Thu, 14 Aug 2025 06:08:48 GMT

Hi @Anonymous ,

May I ask if you have resolved this issue? If not, kindly share the requested details.

Thank you.