Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
5 years ago

Implementing "Value.WaitFor" in PowerBI DataFlow

Hi experts!

 

In my dataflow that consumes a REST API I come across HTTP 429 "Too many requests" errors which break the dataflow refresh.

 

I have copied all queries to my PowerBI Desktop and implemented the 'Value.WaitFor' to overcome the issues with the 'HTTP 429' Status.

 

Finally, I used the code supplied here: https://gist.github.com/CurtHagenlocher/68ac18caa0a17667c805 

 

So, now I have moved the queries back to the dataflow, The preview of the Queries works fine. 

 

But when I am Saving the DataFlow, the message

 

Can't save dataflow

One or more entities references a dynamic data source.

 

 

I really hope that someone here has a workaround or a proper solution. This is a major blocker for us.

 

Thanks for reading!

 

Best,

 

Denis

Dataflow
Need Help

3 Replies

    • Anonymous's avatar
      Anonymous
      Not applicable
      5 years ago

      Hi v-easonf-msft !

      Thanks but I am already using the RelativePath and Query options together with Web.Contents. 

      That's my big frustration. The base URL is always the same (server name) but still, the error suggests that I am using a dynamic URL.

      As you can see in my screenshot, the Web.Contents method always gets the same parameter value:

       

       

      (url as text, optional options as record) => 

let
    WaitFor = 
        (producer as function, interval as function, optional count as number) as any =>
                let
                    list = List.Generate(
                        () => {0, null},
                        (state) => state{0} <> null and (count = null or state{0} < count),
                        (state) => if state{1} <> null
                            then {null, state{1}}
                            else {1 + state{0}, Function.InvokeAfter(() => producer(state{0}), interval(state{0}))},
                        (state) => state{1})
                in
                    List.Last(list)

in

WaitFor(
        (i) =>
            let
                options2 = if options = null then [] else options,
                options3 = if i=0 then options2 else options2 & [IsRetry=true],
                result = Web.Contents(#"Jira Endpoint", options3 & [ManualStatusHandling={429}]),
                buffered = Binary.Buffer(result), /* avoid risk of double request */
                status = if buffered = null then 0 else Value.Metadata(result)[Response.Status],
                actualResult = if status = 429 then null else buffered
            in
                actualResult,
        (i) => #duration(0, 0, 0, i*0.1), 10
        )

       

      Best, Denis