How does a pipeline developer inspect details when encountering 400/4xx and 500/5xx errors

[dbeavon3]

I'm trying to put on my "low-code hat", in order to help a co-worker with a "copy-activity" in fabric pipelines.

The source is a web request.  It this request is sent to an on-premise http service via a gateway:

The http interface in question is one that was built prior to the invention of low-code tooling like ADF and Fabric.  It puts the technical error details into the body of the response.  For a large variety of HTTP errors (400/4xx/500/5xx), the "status" line will not be finely-polished for the needs of a "low-code" web consumer.  Whereas status line will NOT be very user-friendly or meaningful, the body of the 400 response will compensate for that.  It will certainly have enough detail for someone to understand the  reason why things are failing. 

Unfortunately I have not found any option in the ADF U/I that allows anyone to see the body of an error response.  Here is what is shown in ADF: 

If I wasn't a "low-code" developer, I would quickly side-step these ADF limitations.  That could be possible by running the same web request from Postman or some other well-built software development tool. 

Fiddler is another tool that might be useful in this situation .  It would allow inspection of all the traffic going to and from the web server in question.

Another potential solution (if I was an administrator) is to enable the full logging in the "on-prem enterprise gateway".  I suspect that the body of an HTTP 400 error might show up in the integration-runtime logs of that gateway (although this should be tested to be sure).  Navigating those gateway logs is not that convenient so I would only do this as a last resort, and only if I was a gateway admin.

I only provide these workarounds as examples of what a normal developer might require to troubleshoot a 400 error.  But these options are not necessarily appropriate for a "low-code" developer in ADF.  The main reason is because these options require administrator access to the gateway, or they require more sophisticated developer tools to be installed outside of the bounds of the Fabric environment. 

Question:

Can someone tell us what these low-code developers should do, in order to see the HTTP response body for a 400 status code?  I've seen various  other low-code products, outside of ADF, where the body was presented for troubleshooting.   For example I have seen tools that give the option to temporarily treat 400 errors as if they were successful.  The benefit of handling it in that way is that the output can be redirect to a sink, where it can be inspected at will.  Unfortunately I have not yet found any mechanism for this in ADF copy activities yet!  Any tips would be appreciated.

[dbeavon3]

I found a suggestion online that did not work.

The suggestion can be found here:
https://learn.microsoft.com/en-us/answers/questions/778258/getting-web-activity-server-response-erro...

The proposal involved having a failure step, after the web copy-activity, and assigning a variable like so:

Unfortunately the output that is retrieved is no better than before.  It does not contain the detailed body that was provided in the contents of the HTTP 400 response.  So I'm still looking for an approach that will actually work .

[dbeavon3 · Accepted solution]

OK, now I figured out how that other suggestion was supposed to work.  You have to begin with a "REST" source type, not HttpServer.

When using "REST" source types, then the connection and activity will reflect that.  See below.

At this point, it allows us to access the full body of the failure like so:

@string(activity('ABC').output)

Moreover, we don't actually need to set a variable, since the U/I now displays the full body from the 400-response failures.

It isn't clear to me WHY the full failure body is retrieved via "REST" copy-activities but not "HttpServer" copy-activities.  If anybody has any insight into the differences, please let me know!

FYI, We aren't currently using REST by default because it wasn't enabling us to make binary-to-binary copies for some reason.  Whereas HttpServer allows these copy activities and seemed more straightforward.  In any event, it does NOT seem like low-code users of ADF should be forced to sacrifice the content-body of these 400 errors, when they are selecting between the sources of type "REST" and "HttpServer".  I'd love to know why this issue is happening.