Forum Discussion

Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
3 years ago

Newbie help with Pipeline copy from Web API that uses header token

Hi, I'm (completely) new to data pipelines - I come from a Power BI world so I know dataflows not pipelines.   I'm trying to use a COPY activity to copy data from a web API. From PowerShell I'm abl...
General Question
Need Help
pipelines
Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
3 years ago

Hi GraceGu , I had actually seen (and tried) that Stack Overflow post with no success. I'm unable to create a dummy connection using anonymous/none type - when I try, it fails to authenticate and won't let me continue - so the dummy connection never gets created. Screenshot below. 

 

Since I'm unable to get the dummy connection to save, I'm not able to move forward to the point where I can add the header info. Any suggestions?

 

Thanks,

Scott

 

 

 

 

 

 

DavidCousins's avatar
DavidCousins
Frequent Visitor
3 years ago

Hey Scott,

 

for the copy activity you need to use a WEB connector. Give it the base URL and no authentication. Then in the copy activity itself on the source tab, you can define the connection type as REST. Complete the rest of the URL in the Relative URL field and then you open the advanced options and add your Authorization header.

 

It stumped me for an hour when I first encountered it, it's really not intuitive. The REST connector is only useful in very limited scenarios, for anything needing a bearer token it has to be WEB. Especially frustrating considering all of the MS REST APIs actually need bearers, so you can't use the connector for anything they produce 😄

  • Scott_Powell's avatar
    Scott_Powell
    Icon for Helper V rankHelper V
    3 years ago

    hi DavidCousins , trying to follow your suggestion but not making much progress. I set up a pipeline and add a copy activity, then go to the Sources tab:

     

    When I click to add the new connection, there isn'ta WEB connector offered, only HTTP or REST:

    With either of those two options, I don't have the opportunity to enter the header token, and therefore can't get past connector creation.

     

    Am I missing something? I did notice that I can add a WEB activity directly in to the pipeline, and using it I can enter the header info and it seems to return data - but I'm not sure at that point how to "hook it up" to the copy activity so it can be used as a source.

     

     

     

    Sorry for the newbie questions, thank you for the assistance!

    Scott

    • DavidCousins's avatar
      DavidCousins
      Frequent Visitor
      3 years ago

      I wasnt as clear as I could have been, so no worries! I can't add images sadly, but heres some instructions:

       

      1. In your pipeline create string Variables called bearer_type and bearer_token and set the values as your "some_api_token" and "hard coded" token, respectively
      2. In a copy activity for source create a HTTP source against the base URL for your API. In your example it's https://someurl.com/
      3. Use Anonymous for the authentication of this connection. If this URL isnt publically pingable then this will fail and you won't be able to do this until On-Prem data gateways are enabled for data factory to use.
      4. In the copy activity set the connection type to REST
      5. In the Relative URL build whatever string you need. Your example would be:
        1. /api/nameofapitouse
        2. Under advanced set the request method to GET
        3. Create 1 Additional headers
        4. Header Name = Authorization
        5. Value (use dynamic content) = 

       

      @concat(
    variables('bearer_type')
    , ' '
    ,variables('bearer_token')
    )

       

      • Now setup your sink