Data flow refresh order

[Drunken_Sailor]

Hello,
In a workspace, I have serval dataflows When dataflow 1 is refrershed, it also refreshes the next dataflow (2).  And when Dataflow 2 starts it starts dataflows 3a, 3b, and 3c.

I assumed that none of the the 3 (a, b and c) would start until 2 is done. and 2 wont start until 1 is done.  But when I look at the end refresh times of each
1) 13:07:48
2) 13:08:10
3a) 13:07:49
3b) 13:08:15
3c) 13:07:49

The 3s are small API calls, so they are very quick anyway.  Actually 3a and 3c, wont find any data at the moment, as they are just setup for future use.  But I still dont understand how they end before number 2 does.

Thanks in advance

[Drunken_Sailor]

Thanks to all for trying to provide a solution. However, after further investigation, what I have learned is that when dataflow 1 is triggered, either manually, scheduled, API  or Power Automate, Dataflows 2 and 3 (all of them) are also triggered in PARRALEL and not sequentially.  Basically, in my scenario, 3 finishes before 2 and 1.  Which makes no sense for me. There also seems to be no option to turn this "feature" off. In addition, the queries within the dataflow seem to run in parallel too even though some have dependecies on each other. 
The only way I see making this work is to come up with a convoluted mix of Dataflows and semantic models and use power automate to "orchestrate" the timing.  

The only way I see making this work is to come up with a convuluted mix of Dataflows and semantic models and use power automate to "orchestrate" the timing.  I hope someone can tell me I am wrong.

[TomMartens]

Hey @Drunken_Sailor ,

I'm wondering if you have Power BI Premium?
Chaining, is exactly what we are doing by using linked entities. Most of the time to create staging/transforming/loading chains, this works perfecly. So I'm wondering if you are using linked entites, but as I already mentioned - this is a Premium feature (but also available with Premium Per User).

This article explains the premium features of dataflows in more detail: Premium features of dataflows - Power BI | Microsoft Learn

Regards,

Tom

[Drunken_Sailor]

Hi Tom, Premium per user is what I have.  Take a look at the refresh log of some of your linked queries within a dataflow. Look at the start times and you'll probably notice that the start at the same time. For example. If query A gets data from another flow, and B uses A to get some data from another source, and C then uses B to get soemthing from soemwhere else, when the dataflow is started, A status is "skipped", not sure why it says that, but okay it gets the latest info, and B and C start at the same time.  Now B takes longer to run as it looks at something and then filters out the unwanted.  C should then pick up after that and just get the rest of the data that is required.  However, C finishes, according to the log, before B.  Just strange. 

My workaround was to move some of the steps to a semantic model. Then use Power Automate to trigger the semantic mode, when the first dataflow finishes and then "delay"  some minutes and start antoher step.  Its not optimal, but it works.

[SaiTejaTalasila]

Hi,

Using power automate is the perfect option for your requirement.

[Drunken_Sailor]

Sorry, not really. I'll get two triggers then, which seem redundant.

[v-zhengdxu-msft]

Hi @Drunken_Sailor 

You can refer to the following official documentation for a description of the refresh order:

Understand and optimize dataflows refresh - Power BI | Microsoft Learn

After you delete the dataflow, if your dataflow contains many queries or some queries depend on others. If you cancel a refresh before you start evaluating a query that loads data into a target, no changes will occur to the data in the target of that query. There are a few instrumentation tools you can use to further observe the refresh order of your dataflow:
powerquery-docs/powerquery-docs/dataflows/monitor-dataflow-refreshes.md at main · MicrosoftDocs/powe...

Best Regards

Zhengdong Xu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Drunken_Sailor]

I read that too, which is what led me to orgnaizing the dataflows in such a way.  They used the term "chaining" which led me to believe it was sequential... but it isnt.

[Daryl-Lynch-Bzy]

Hi @Drunken_Sailor , are you using Scheduled Refresh to trigger the Dataflows.  I.e. start them all at 08:30.  If so, I would suggest switching to use Power Automate.  There is a Action (Start Dataflow Refresh) and Trigger (When Dataflow Refresh Ends) connectors that will allow you to run the Dataflows in a specific order or in parallel.  Here is video to get you started.
How to Refresh a Dataflow Using Power Automate 

[Drunken_Sailor]

Dataflow A is scheduled to start at 8:30.  That then triggers B and C to start at the same time.  If instead I use Power Automate to trigger A, i get the same result. B and C will also get triggered at the same time.

Sure, I could then use "when a dataflow refresh completes" to trigger B (which then triggers C) when A is finisshed, but that just means B gets refressed twice. If I then use another flow to trigger c when B is done, C get s triggered three times.  
In this scenario, it is not a big deal, as the refresh cycles are only a few seconds, but I have others that take 2 to 3 hours each. They make API calls to a service that also limits how many you can do concurrently. Hence why I need these to be in a series and not in parrallell.