Copy Data Activity with REST API POST - Issue with request body and escape characters

[youssef_o]

I am trying to use a copy data activity to get data from a REST API using a POST call and copy it to a lakehouse. The problem is that the copy data activity adds escape characters to the double quotes \" which makes the body unreadable by the API. I tried to use variables and concatenating and replacing \" with " but nothing seems to work.

Is there a solution to this issue?

[apturlov · Accepted solution]

@youssef_o  I agree that using a Copy activity is a better choice in this case compared to a Web activity. But something just does not add up in this picture: Web activity can execute the call, but the Copy activity fails. Both activities should be using the same REST connector with the same request settings. Can you compare settings of Web and Copy activities and make sure both are configured the same way? You could also paste a schreenshot of your Web activity settings here for a second check.

[v-venuppu]

Hi @youssef_o ,

May I ask if you have resolved this issue? If not, kindly share the requested details.

Thank you.

[v-venuppu · Accepted solution]

Hi @youssef_o ,

Have you compared the settings..?Can you also please paste a schreenshot of your Web activity settings here for a second check as requested by @apturlov .

Thank you.

[v-venuppu]

Hi @youssef_o ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @apturlov @lbendlin for the prompt response.

Can you please share the details as requested by @apturlov , so that it will be helpful for us to solve the issue.

[youssef_o]

Hello, yes I just replied to @apturlov with the output I see in Postman for the same call

[apturlov]

It is correct that the Accept header is not usually needed when the content-type header is specified. In my previous example above I did not need the Accept header. From information you provided so far it's difficult to diagnose the  precise problem. You body and header look absolutely correctly from the activity input screenshot. For your original question about the body encoding the answer is yes, the encoding of the body is correct. But you say that the API returns an error. Can you post the output from your activity run or maybe a response that you get when you test with other tools like Postman?

[youssef_o]

Hello, here is the output from postman when I use the correct body and when I use the body from the copy activity with escape characters.

[apturlov]

Hi @youssef_o, thanks for your reply. I don't think the ouput from Postman is helpful with debugging issue in Fabric. I was asking about the output from your pipeline run in Fabric to see specifically what error was returned by the API.

[youssef_o]

Hi, yes this is the error given by the copy activity in the pipeline. The same call works using a Web Activity but I still want to use copy activity because the web activity fails when the response is larger than 4MB

Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 406 NotAcceptable, please check your activity settings.
Request URL: https://*************/rest/2.0/outputModule/export/json.
Response: {"statusCode":406,"titleMessage":"Not Acceptable","userMessage":"HTTP 406 Not Acceptable"},Source=Microsoft.DataTransfer.ClientLibrary,'

[apturlov · Accepted solution]

@youssef_o  I agree that using a Copy activity is a better choice in this case compared to a Web activity. But something just does not add up in this picture: Web activity can execute the call, but the Copy activity fails. Both activities should be using the same REST connector with the same request settings. Can you compare settings of Web and Copy activities and make sure both are configured the same way? You could also paste a schreenshot of your Web activity settings here for a second check.

[apturlov]

@youssef_o I don't think that COPY activity did anything wrong. The encoding on the wire is correct and is based on how the REST connector handles content when the content-type is application/json. On the other hand, response 406 does not indicate an incorrect request, which would be 400 Bad Request. I guess, something is wrong based on 406 response status, but I would not blindly blame the COPY activity. To prove that, I did a quick experiment with a public REST API using a POST with a JSON body. As you would guess, the experiement was successful. See below the results:

As you can notice the encoding in the activity input is exactly the same as in your case, which is how a well-formed JSON looks like on the wire, and the response is also correct.

However, getting back to 406 response, it usually means that the request is missing an accept header that should indicate a content type that the response should be generated for. It seams that your API is very picky about that header and in your configuration it's missing. Try to add accept: application/json header to your COPY activity if you expect a JSON response, or another proper type, and verify if it works correctly this time. 

[youssef_o]

Hello, thank you for your response. I tried adding accept but it says it's set to application/json by default anyway. I still get the same error.

These headers work in postman for the same request 

[lbendlin]

How does your accept header look like?

[youssef_o]

Here is my configuration

[apturlov]

@youssef_o Can you post a screenshot of your COPY activity configuration for the body of your POST request? It sounds from your description that something is missing or misconfigured.

[youssef_o]

This is my configuration, the web connection works with other API calls with GET and the same exact call I am doing here works with Postman.

As you can see in the input screenshot, the activity adds a bunch of escape characters.