Forum Discussion

Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
3 years ago

Newbie help with Pipeline copy from Web API that uses header token

Hi, I'm (completely) new to data pipelines - I come from a Power BI world so I know dataflows not pipelines.   I'm trying to use a COPY activity to copy data from a web API. From PowerShell I'm abl...
General Question
Need Help
pipelines
GraceGu's avatar
GraceGu
Icon for Microsoft Employee rankMicrosoft Employee
3 years ago

1st step, get bearer token with a Web Activity.

2nd step, add output from the 1st step into authorization header of copy source connector (source connection is a "dummy" connection when created with anonymous/none authetication type, runtime get authenticated with bearer token header) . 

You can refer to How to pass bearer token API in Azure Data Factory - Stack Overflow. 

  • Scott_Powell's avatar
    Scott_Powell
    Icon for Helper V rankHelper V
    3 years ago

    Hi GraceGu , I had actually seen (and tried) that Stack Overflow post with no success. I'm unable to create a dummy connection using anonymous/none type - when I try, it fails to authenticate and won't let me continue - so the dummy connection never gets created. Screenshot below. 

     

    Since I'm unable to get the dummy connection to save, I'm not able to move forward to the point where I can add the header info. Any suggestions?

     

    Thanks,

    Scott

     

     

     

     

     

     

    • Scott_Powell's avatar
      Scott_Powell
      Icon for Helper V rankHelper V
      3 years ago

      p.s. and here's the section where you actually create the source connection. As you can see, until you pick a valid connection, the section where you can enter the additional header info doesn't even show.

       

      Thx,

      Scott

       

       

      • ajarora's avatar
        ajarora
        Icon for Microsoft Employee rankMicrosoft Employee
        3 years ago

        This should work. Can you please troubleshoot if the issue is due to the specific URL you are providing, start with anonymous and any other public endpoint say https://wikipedia.org and see if that works. If you need better support, please file a support ticket for the connection creation issue, as this is a supported scenario.

    • DavidCousins's avatar
      DavidCousins
      Frequent Visitor
      3 years ago

      Hey Scott,

       

      for the copy activity you need to use a WEB connector. Give it the base URL and no authentication. Then in the copy activity itself on the source tab, you can define the connection type as REST. Complete the rest of the URL in the Relative URL field and then you open the advanced options and add your Authorization header.

       

      It stumped me for an hour when I first encountered it, it's really not intuitive. The REST connector is only useful in very limited scenarios, for anything needing a bearer token it has to be WEB. Especially frustrating considering all of the MS REST APIs actually need bearers, so you can't use the connector for anything they produce 😄

      • Scott_Powell's avatar
        Scott_Powell
        Icon for Helper V rankHelper V
        3 years ago

        hi DavidCousins , trying to follow your suggestion but not making much progress. I set up a pipeline and add a copy activity, then go to the Sources tab:

         

        When I click to add the new connection, there isn'ta WEB connector offered, only HTTP or REST:

        With either of those two options, I don't have the opportunity to enter the header token, and therefore can't get past connector creation.

         

        Am I missing something? I did notice that I can add a WEB activity directly in to the pipeline, and using it I can enter the header info and it seems to return data - but I'm not sure at that point how to "hook it up" to the copy activity so it can be used as a source.

         

         

         

        Sorry for the newbie questions, thank you for the assistance!

        Scott