Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago
Solved

How to securely store and use token and secret for API request in Power Query M ?

Hello all,

 

I have successfully connected to an online service with an API request with token and secret, see example below. However, I believe storing the token and secret in the Power Query M code (or in a parameter) is not very secure, so I am looking for a more secure option. Does anyone know what is the best solution here? Can I use Azure Key Vault (if yes, how?). Or is there another way to do this securely? Many thanks!

 

let
  url = "https://url_of_service/",
  token = "<MY_TOKEN>",
  secret = "<MY_SECRET>",
  authString = token & ":" & secret,
  authBytes = Text.ToBinary(authString),
  encodedAuth = "Basic " & Binary.ToText(authBytes, BinaryEncoding.Base64),
  body = "{ ""profiles"": [ { ""id"": ""<PROFILE1>"", ""id2"": ""<PR1>"" }, { ""id"": ""<PROFILE2>"", ""id2"": ""<PR2>"" } ], ""date_start"": ""2023-01-01"", ""date_end"": ""2023-01-31"", ""metric"": ""metric1"", ""dimensions"": [ {""type"": ""date.day""}, {""type"": ""id""} ]}",
  options = [
        Headers = [
            #"Authorization" = encodedAuth,
            #"Content-Type" = "application/json; charset=utf-8"
        ],
        Content = Text.ToBinary(body)
    ],
  response = Web.Contents(url, options),
  data = Json.Document(response),
  #"Converted to table" = Record.ToTable(data)
in
data

 

 

 

  • artemus's avatar
    artemus
    3 years ago

    Oh, sorry about that... didn't realize that POST didn't support authentication in Web.Contents. I don't know any use of keyvault, and you would likely run into the same issues you face here using it.

     

    I think the only option you would have is to write your own connector, which is given much more freedom on how it operates. Note, that if you write your own connector you cannot publish your report online unless you use a gateway (in which case you might as well just store the secret on the gateway machine and configure access to that speratly). Also, note that this does not work with importing data into Excel.

15 Replies

  • ams1's avatar
    ams1
    Responsive Resident

    Hi Anonymous ,

     

    Great question - indeed using hardcoded credentials in the query is NOT secure.

     

    There is a quick-and-maybe-not-so-dirty workarround:

     

    Context:

     

    If you do...

     

    let
        Source = Web.Contents("http://localhost:5000/test", [ApiKeyName="howdy"])
        //                                                  ^^^^^^^^^^^^^^^^^^^^ this is just the NAME of the key (NOT the key secret)
    in
        Source

     

    ...and then store the credentials securely in the PowerQuery "credentials manager"...

    ...PowerQuery will send to the webserver the following GET web request:

     

                                     vvvvv - notice the key SECRET from credentials manager
    http://localhost:5000/test?howdy=12345
                               ^^^^^ - notice the key name from ApiKeyName

     

     

    So PowerQuery sends the API key name and secret as GET query parameters.

     

    Problem:

    Now MOST of us need that api key secret inside the Header (NOT as query parameters).

     

    A solution:

    Well, one way to get that secret is to have a SUPER-SIMPLE web server somewhere (local, intranet, lambda, azure function etc.) that just extracts the secret from the query parameter and just sends it back in the response - I can provide ex. the python code to run on localhost if you want.

     

    With that "intermediary boomerang server" ğŸ˜Š, your code would look like:

     

     

    let
        // below is the web request that uses the token from PowerQuery credentials manager
        // sends it as query parameter to the "boomerang" server which just sends it back in the response 
        token = Web.Contents("http://localhost:5000/boomerang", [ApiKeyName="howdy"]), // 1st request with apy key credentials
        actualWebRequstResponse = Web.Contents( // 2nd request with annonymous credentials
            "http://api_that_uses_header_authorization",
            [
                Headers = [
                    // below i use the token from the intermediary
                    Authorization = Text.FromBinary(token) // if the token request returns just the key as text
                ],
                // POST requests also work with apy key this way!
                Content = Text.ToBinary("how are you?")
            ]
        )
    in
        actualWebRequstResponse

     

     

    Why do we need to go to these extra lengths to use the PowerQuery credentials manager for Header authorization (majority of APIs)? I don't know - I'd actually be curious to get an answer.

     

    The thing is there is a workarround which will be as secure as the boomerang server.

     

    FYI: Anonymous 

     

    Please mark also this as answer if it helped.

    • Dops0's avatar
      Dops0
      New Member

      ams1 Ha! I just did exactly this before seeing your post. This method works fine in PBI Desktop. Unfortunately, the Web API Credential method is not supported in PowerBi Service even though it is 2023 and the whole world is using REST APIs with Authentication! 

       

      I really hope I am wrong here and someone will correct me 🙂

    • Anonymous's avatar
      Anonymous
      Not applicable

      ams1 thanks for your additional input! I indeed need that api key secret inside the Header. Your proposal sounds like it could work, but at this point I don't know if we can set up a web server for this. I am just thinking out loud... could I not extract the secret from the query parameter in the query itself and then add it to the Header?

      • ams1's avatar
        ams1
        Responsive Resident

        Anonymous indeed, there is the extra hassle of the boomerang server - the code itself is SUPER simple and runs in milliseconds, you just need somewhere secure to serve it (like I've said - a lambda/azure function would be more than enough).

         

        The ideea is that instead of running 1 x request and have PowerQuery handle the authentication, you first run a request that gets you everything* you need (token, credentials etc) to authorize the 2nd request.

        *in this regard you could also refresh the token/get a new one, do ouath etc. - with additional code on the boomerang server side -> including getting things from Azure Key Vault.

         

        Note that as far as I know PowerQuery does NOT "see" the "compiled" web request (and thus is not able to extract the web query parameters), only the webserver does - so that's why you need it.

         

        P.S.: Please mark my previous reply as solution if it helped (and maybe kudo just to raise awareness - hopefully we'll get this feature) ğŸ˜‰

  • artemus's avatar
    artemus
    Microsoft Employee

    Your best bet it to add the option: ApiKeyName = "Authorization". However, you would need to encode the encoded authorization by hand. When you are prompted for credentials, select ApiKey for login type.

    let
      url = "https://url_of_service/",
      body = "{ ""profiles"": [ { ""id"": ""<PROFILE1>"", ""id2"": ""<PR1>"" }, { ""id"": ""<PROFILE2>"", ""id2"": ""<PR2>"" } ], ""date_start"": ""2023-01-01"", ""date_end"": ""2023-01-31"", ""metric"": ""metric1"", ""dimensions"": [ {""type"": ""date.day""}, {""type"": ""id""} ]}",
      options = [
            Headers = [
                #"Content-Type" = "application/json; charset=utf-8"
            ],
            Content = Text.ToBinary(body),
            ApiKeyName = "Authorization"
        ],
      response = Web.Contents(url, options),
      data = Json.Document(response),
      #"Converted to table" = Record.ToTable(data)
    in
    data

     

    • Anonymous's avatar
      Anonymous
      Not applicable

      Thanks! Would this work with a scheduled refresh?

      • artemus's avatar
        artemus
        Microsoft Employee

        It would last as until the token expires. When this happens you will get an email and need to provide a new updated token.