Getting REST response headers

[AdamFry]

Hi there!  I have an API that I would like to use in a data pipeline but this API requires me to call a generic URL with the request that I'd like to make and in the response headers, it provides me with a redirect URL.  In postman, it looks like this: 

Since the censored part of the above image is the only thing that changes, you can't really see the difference but I do a GET to the URL at the top and in the response headers, I get a "Location" property back that gives me the URL that I should use to make my request.  

I am able to successfully make this call via PostMan and then take the redirect URL that's provided in the response header and use it in a fabric data pipeline copy activity and make the REST call to get the data for that request.  However, I'd like to perform this call in my pipeline and pass the redirect URL to the next step in the process so that I can ensure that I'm always hitting the right URL if the vendor changes the redirect URL.  But I am unable to see any way to get a handle on the response header. 

If there's good documentation on how to do this, I'd be so grateful for anyone who could point me in the right direction.  If there's another, better approach to handle this situation, I'd love to hear about that as well.  Thanks in advance!

[Kaviraj11]

This will probably help you with that - Azure Data Factory - Copy Activity - Source API Request save header response - Stack Overflow

[v-cboorla-msft]

Hi @AdamFry 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[AdamFry]

Thank you for checking back on this.  I am trying to exercise the web activity to get the API reponse headers and I wonder if I'm overcomplicating this whole thing.  Based on what I'm seeing, I'm wondering if the fabric REST and/or WEB connection handles URL redirects automatically?  

The API I'm calling requires me to get a URL redirect by calling one URL and on that call, I get an HTTP 302 response with a "Location" field in the response header that provides me the URL that I should call to get my data.  Then I need to call another URL to get a token and pass that into the 3rd request at the redirected URL.  

When I try using the web connection and point to the first URL with anoynmous authentication, I get a 401 Unauthorized back saying the client request has not been completed because it lacks valid authentication credentials for the requested endpoint(url).  

However, when I make this same call via Postman with no authentication provided, I get the HTTP 302 and the redirect URL in the response headers.  

As a test, back in Fabric, I tried adding my token URL and API credentials to the web connection and then instead of getting the HTTP 302 and the redirect URL in the response headers, I got the content that I normally would get on the final call.  This makes me think that Fabric is automatically handling the redirect and I don't need to bother with the web call and parsing the redirect URL from the response and then making another request to get the data.  

Is there any documentation that describes this behavior?  I just want to confirm that the REST connection is actually automatically following the redirect and if that's the case, that's awesome and I can simplify my pipeline.

[v-cboorla-msft]

Hi @AdamFry 

Apologies for the issue you are facing.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.
Appreciate your patience.

Thanks.

[Kaviraj11]

Hi!

I have been in the similar situation but different scenarios. So, we stored the total count that comes in the header into the warehouse using Copy Data and use the 'LookUp' to query(SQL) the values from the table in warehouse. Next, we use 'Set Variable' to pass on the value into the Variables declared and further utilized the variable in the next step of the pipeline.

I hope it's helpful.

[AdamFry]

Hi Kaviraj, thank's for the suggestion!  I would be happy to do something similar, storing the header using copy data and then doing a lookup to read that value but what I'm not understanding is how did you get a handle of the REST response header to store it?  When I preview data or write the response to a file, I am just seeing the response body.