Forum Discussion

2366's avatar
2366
Helper I
3 years ago

Inefficient Dataflow default orchestration

Hi All,


I have setup a dataflow C which is sourced from A and B where linked entities is turned off by disable load. The transformation steps I have implemented in C takes about 40 minutes to run. I am using power automate to load A and B and trigger C.
however I now wanted to implement a another transformation logic on C by merging another table from D which has now made the data flow run for more than 10 hours.


I turned on linked entities in C and the refresh takes about 30 minutes now.

 

I understand, powerbi is now using enhanced compute engine, hence the refresh is faster but now the problem I have run into is, C is refreshed 3X times everyday, whenever I refresh A, B and D which is inefficient because my overall runtime is 1.5 hours.
ideally it should be just 40 minutes because A, B, and D load within 8 minutes and C runs for about 30 minutes with enhanced compute engine. 

 

Shouldn't there be a way, I can configure the dataflow to retain enhanced compute engine but only refresh the final table after all my source dataflows are refreshed?

 

thanks

 

Dataflow

3 Replies

  • bcdobbs's avatar
    bcdobbs
    Community Champion
    3 years ago

    Bit of a workaround however...

    If you edit the power query in advanced editor for C you will see it references the linked dataflows by GUID (long unique I'd). If you manually edit the code so it navigates by the name of the power query the auto refresh doesn't kick in. Away from a computer at the moment but can supply an example if needed.

    • 2366's avatar
      2366
      Helper I
      3 years ago

      Thank you bcdobbs! this is something I have already done by creating parameters for Data flow id which breaks the auto refresh but it just reloads the linked table all over again.

      Does anyone ever feel like this is a really inefficient design of dataflows? It makes no sense for the target table to refresh multiple times every time the upstream data source is refreshed unless all the linked sources are refreshed

  • ysusanto's avatar
    ysusanto
    New Member
    3 years ago

    Hi, I was facing the same issue, what works for me was in the combined table, I unchecked the "Enabled Load" option within the Properties, and schedule refresh time on the Settings.