Coordinating data flows and PBI loads

[Anonymous]

Hi, I've got what I believe to be a pretty simple scenario, but I haven't been able to figure out how to get the Power BI service to run some stuff in the right order.

In general, I have:

- 3 "ingest" dataflows that pull down data

- 1 "compute" dataflow that uses linked entities from the ingest flows to create several entities

- A power BI dataset I want to then load with the results of the "compute" entity

What I want to do sounds simple:

a. First refresh the 3 ingest dataflows

b. Once all three ingest dataflows are done, then refresh the "compute" dataflow

c. Once the compute dataflow is complete, refresh the Power BI dataset

It feels to me like this should be super simple...but I don't know how to do this. Any advice?

Thanks,
Scott

[GilbertQ]

Hi there

What you would need to do, is to schedule them in the order as you have shown?

[Anonymous]

Hi @GilbertQ , that's not really a very workable solution in our case. I'm really looking for something dependancy based vs. just guessing / hoping how far apart to schedule jobs so that things like jobs running long or not being kicked off on time in the Power BI service (which happens ALL the time...I often get jobs running a full hour behind) don't hose my dependencies and cause data to be incorrect.

Matthew Roche goes to great length to expound the virtuals of dataflows, so I assumed as a complete dataflow newbie that I was missing something obvious. But I'm sensing from your response that I'm not, and that basic functionality like job dependencies isn't available.

I've thought of combining my 3 "ingest" dataflows into a single one, which would eliminate the issue of trying to make sure 3 input dataflows finish before the compute dataflow refreshes. But the performance in the web is so bad when you start getting many entities in a single dataflow that I've deliberately avoided that...because it can take a good 15 to 30 minutes for a dataflow to do whatever it's "evaluating entities to make sure the load will work properly" step to complete so the job will save. Plus, although this would help get the dataflows synchronized, it still doesn't I think help with kicking off the Power BI dataset refresh once the dataflows are complete.

Maybe I need to look into Power Automate...not sure if that would allow this or not...

Thanks,

Scott