Copy Data Activity with REST API POST - Issue with request body and escape characters

[youssef_o]

I am trying to use a copy data activity to get data from a REST API using a POST call and copy it to a lakehouse. The problem is that the copy data activity adds escape characters to the double quotes \" which makes the body unreadable by the API. I tried to use variables and concatenating and replacing \" with " but nothing seems to work.

Is there a solution to this issue?

[v-venuppu]

Hi @youssef_o ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @apturlov @lbendlin for the prompt response.

Can you please share the details as requested by @apturlov , so that it will be helpful for us to solve the issue.

[apturlov]

It is correct that the Accept header is not usually needed when the content-type header is specified. In my previous example above I did not need the Accept header. From information you provided so far it's difficult to diagnose the  precise problem. You body and header look absolutely correctly from the activity input screenshot. For your original question about the body encoding the answer is yes, the encoding of the body is correct. But you say that the API returns an error. Can you post the output from your activity run or maybe a response that you get when you test with other tools like Postman?

[apturlov]

@youssef_o I don't think that COPY activity did anything wrong. The encoding on the wire is correct and is based on how the REST connector handles content when the content-type is application/json. On the other hand, response 406 does not indicate an incorrect request, which would be 400 Bad Request. I guess, something is wrong based on 406 response status, but I would not blindly blame the COPY activity. To prove that, I did a quick experiment with a public REST API using a POST with a JSON body. As you would guess, the experiement was successful. See below the results:

As you can notice the encoding in the activity input is exactly the same as in your case, which is how a well-formed JSON looks like on the wire, and the response is also correct.

However, getting back to 406 response, it usually means that the request is missing an accept header that should indicate a content type that the response should be generated for. It seams that your API is very picky about that header and in your configuration it's missing. Try to add accept: application/json header to your COPY activity if you expect a JSON response, or another proper type, and verify if it works correctly this time. 

[youssef_o]

Hello, thank you for your response. I tried adding accept but it says it's set to application/json by default anyway. I still get the same error.

These headers work in postman for the same request 

[lbendlin]

How does your accept header look like?

[youssef_o]

Here is my configuration

[apturlov]

@youssef_o Can you post a screenshot of your COPY activity configuration for the body of your POST request? It sounds from your description that something is missing or misconfigured.

[youssef_o]

This is my configuration, the web connection works with other API calls with GET and the same exact call I am doing here works with Postman.

As you can see in the input screenshot, the activity adds a bunch of escape characters.