Forum Discussion

Soldexio's avatar
Soldexio
Helper I
7 years ago
Solved

Scheduled Refresh Custom Function

Hi everyone,

 

I'm trying to generate a scheduled refresh for my data sources, however every data source uses our own API endpoints. These are accessible outside the organization, however require certain authentication and parameters. Therefore, I have defined a custom function as follows:

 

(param1, param2) =>
let
Source = Json.Document(Web.Contents("URL" & "ParamX=" & param1 & "&ParamY=" & param2, [Headers=[Authorization="Bearer " & #"Authorization Token"]]))
in
Source

However, when I try to schedule a refresh like this, it says the data source is unknown and cannot be scheduled as a refresh. Is there any way to get this working or specify this function in a different way to have it working?

 

The parameters come from a different custom function and those cannot be specified as a parameter or static variable. Sometimes 700 times param1 & param 2 will go through.

 

Please let me know as currently manual refreshes are a pain!

 

14 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi Soldexio,

     

    AFAIK, current power bi service still not support custom function, please convert your query to parameterized query instead custom function.

     

    Remove top row and define param1, param2 as query parameters with accurate parameter type:

    let
    Source = Json.Document(Web.Contents("URL" & "ParamX=" & param1 & "&ParamY=" & param2 , [Headers=[Authorization="Bearer " & #"Authorization Token"]]))
    in
    Source

     

    BTW, current query parameters with 'any' type not allow to changed on power bi service side.

     

    Regards,

    Xiaoxin Sheng

    • Soldexio's avatar
      Soldexio
      Helper I

      Hi Anonymous,

       

      Alright good to know I can translate each query to a parameterized query. However, I did this test for one of the queries and when I try to schedule a refresh now I get:

       

      Failed to update data source credentials: The credentials provided for the Web source are invalid. (Source at <URL>.)Hide details

      Activity ID:b46998c5-dd70-47c2-ac52-2c7095e5dbec
      Request ID:4d4f0423-8fbe-3ad3-c487-4914e391ca35
      Status code:400
      Time:Tue Aug 21 2018 08:51:03 GMT+0200 (Central European Summer Time)
      Version:13.0.6358.131
      Cluster URI:https://wabi-europe-north-b-redirect.analysis.windows.net

       

      The data is loading perfectly within Power BI never have any problems, but when I try to schedule a refresh it fails. Any tips to this?

      • Soldexio's avatar
        Soldexio
        Helper I

        Hi @v-shex-msft,

         

        I did some testing with webhook tester. The request that the Power BI service is sending, is not sending the authorization token properly. 

         

        hostwebhook.site
        user-agentMicrosoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
        authorizationBearer
        content-length(empty)
        content-type

        (empty)

         

        Any idea to why this is and how to fix this? Because this is the reason every request on my side is failing when I translate it to parameterized query. The request within Power BI is as follows:

         

        Headers

        connectionKeep-Alive
        hostwebhook.site
        accept-encodinggzip, deflate
        user-agentMicrosoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
        accept*/*
        authorizationBearer 12345
        content-length(empty)
        content-type(empty)

         

        Any help here would be welcome!

    • Soldexio's avatar
      Soldexio
      Helper I

      Anonymous the suggestion you make to use it in a query parameter, how do you that?

       

      Currently I use an API query to receive IDs that need to be used in the second query

       

      So for example: 

       

      Source = Json.Document(Web.Contents("URL" & "ParamX=" & param1, [Headers=[Authorization="Bearer " & #"Token"]]))

      That results in IDs that need to be used in the second query. Basicly for each ID from the above code, do the code below:

       

      Source = Json.Document(Web.Contents("URL2" & "&ParamY=" & param2 , [Headers=[Authorization="Bearer " & #"Token"]]))

      Can you tell me how to do that using query parameters?

       

      Because the following code still fails in the web refresh as it cannot register my credientials:

       

       Source = Json.Document(Web.Contents("URL" & "url_part_1", [Headers=[Authorization="stringtoken"]])),
          

      It says it cannot be validated in the online environment, while locally it works perfectly fine.

       

      The other question also remains on why the scheduled refresh is not working!

      • Anonymous's avatar
        Anonymous
        Not applicable

        HI Soldexio,

         

        If your query need to invoke multiple times based on result data, I think this scenario not suitable to transform to parameterized query.

         

        It means you need to manually write custom steps to invoke api url with different result data.

         

        Regards,

        Xiaoxin Sheng

    • Soldexio's avatar
      Soldexio
      Helper I

      Hi cwebb, Chris,

       

      Thank you for that post and elaboration, that is amazing and I will look in this and this seems to be the solution! Do you know if this also works when you need to pass Authorization towards the URL in order to validate the URL?

       

      Kind regards,

       

      Dennis

      • cwebb's avatar
        cwebb
        Advocate V

        Ah, I didn't see that you are doing authentication using a header - no, I don't think it will work (see the comment by danzrust on the blog post). Sorry!

         

        Chris

    • Oscar_Mtz_V's avatar
      Oscar_Mtz_V
      Kudo Commander

      Hi, cwebb great post, still I have a small issue, my code has two parameters, one part of the URL, the other is a query, thanks to your post I solved the second, still canĀ“t find the right code for the first one.

       

      Parameters are player and season.

       

      Original code:

       

      (Season,Player)=>
      
      let
          Source = Web.Page(Web.Contents("http://www.espnfc.com/player/" & Player & "?season="&Season)),
          Data3 = Source{3}[Data],
          #"Changed Type" = Table.TransformColumnTypes(Data3,{{"Team", type text}, {"Oppo", type text}, {"Date", type date}, {"Comp", type text}, {"Res", type text}, {"Appear", type text}, {"g", Int64.Type}, {"a", Int64.Type}, {"sh", Int64.Type}, {"sg", Int64.Type}, {"fc", Int64.Type}, {"fs", Int64.Type}, {"yc", Int64.Type}, {"rc", Int64.Type}})
      in
          #"Changed Type"

        

      New code:

       

      "Season" parameter is working, "Player" is the one I'm having the problem with, can't find the right code.

       

      (Season,Player)=>
      
      let
          Source = Web.Page(Web.Contents("http://www.espnfc.com/player/" & Player,
              [Query=[season=Season]]
              )),
          Data3 = Source{3}[Data],
          #"Changed Type" = Table.TransformColumnTypes(Data3,{{"Team", type text}, {"Oppo", type text}, {"Date", type date}, {"Comp", type text}, {"Res", type text}, {"Appear", type text}, {"g", Int64.Type}, {"a", Int64.Type}, {"sh", Int64.Type}, {"sg", Int64.Type}, {"fc", Int64.Type}, {"fs", Int64.Type}, {"yc", Int64.Type}, {"rc", Int64.Type}})
      in
          #"Changed Type"

       

      Your support is appreciated.

       

      Regards,

       

      Oscar