Forum Discussion
Helper VNewbie help with Pipeline copy from Web API that uses header token
Helper Vhi DavidCousins , trying to follow your suggestion but not making much progress. I set up a pipeline and add a copy activity, then go to the Sources tab:
When I click to add the new connection, there isn'ta WEB connector offered, only HTTP or REST:
With either of those two options, I don't have the opportunity to enter the header token, and therefore can't get past connector creation.
Am I missing something? I did notice that I can add a WEB activity directly in to the pipeline, and using it I can enter the header info and it seems to return data - but I'm not sure at that point how to "hook it up" to the copy activity so it can be used as a source.
Sorry for the newbie questions, thank you for the assistance!
Scott
I wasnt as clear as I could have been, so no worries! I can't add images sadly, but heres some instructions:
- In your pipeline create string Variables called bearer_type and bearer_token and set the values as your "some_api_token" and "hard coded" token, respectively
- In a copy activity for source create a HTTP source against the base URL for your API. In your example it's https://someurl.com/
- Use Anonymous for the authentication of this connection. If this URL isnt publically pingable then this will fail and you won't be able to do this until On-Prem data gateways are enabled for data factory to use.
- In the copy activity set the connection type to REST
- In the Relative URL build whatever string you need. Your example would be:
- /api/nameofapitouse
- Under advanced set the request method to GET
- Create 1 Additional headers
- Header Name = Authorization
- Value (use dynamic content) =
@concat(
variables('bearer_type')
, ' '
,variables('bearer_token')
)
- Now setup your sink
- Scott_Powell
Unfortunately this does indeed fail at step 3. I did see an article posted here https://blog.fabric.microsoft.com/en-us/blog/data-pipelines-tutorial-ingest-files-into-a-lakehouse-from-a-rest-api-with-pagination-ft-aveva-data-hub?ft=All: that might show another way to do this - if so I'll update here.
Thanks!
Scott