Dataflow loading data - mechanism?

[rdnguyen]

Hi Everyone,

Would anyone shares to me some insight of how dataflow loading data works? My question is to help me understand if I have Dataflow A is the source, and Dataflow B is the connected as having a reference to A directly. A and B are residing in different workspace.

Now if I refresh A and B (on auto schedule) every half an hour;

when B refreshes, my first thought is B gonna take the partition of data half an hour ago (30 mins latency); 

it seems to work that way but intermittenly, B became inactive in loading data from A; say got lagged for some days if I din't figure out and refresh it manually.

Any suggestion are welcome.

[rdnguyen]

I see, the problem is B was stalled and no longer load data at a point of time; rather than the latest completed result set of A. So, here I set up a test before and after the choke point between A and B; A reflected data is 676 on the report; but B reflect another number eventhough B load data directly from A as is.

[rdnguyen]

To an extent, if A is loading while B is finished pulling data from A and assuming B just have partial data from A at that refresh, would B be stalled next refresh regarding the issue someway? Is there a mechanism that stop B from loading the next time?

I was stuck as I set A and B both load every half an hour and suddenly think B failed to load its data due to this overlapping auto schedule.

[collinq]

Hey @rdnguyen ,

If one is running while the other is starting, you will have issues.  At the moment, there is no way within Power BI to kick of B when A ends (or to confirm that A is done before B starts).  I think that you may be able to do this with Power Automate and doing something like "when A ends, kick of B".  And then, have A start again when B is done.  Either use Power Automate or move your timing out where they don't conflict (and then review that refresh duration again now and then to confirm that you are not slowly creeping into the same problem again).

[BryanCarmichael]

Actually now you can trigger dataflows of other dataflow completion by using a datapipeline

[collinq]

Hi @rdnguyen ,

I don't think that this is quite how it works.  If you have Dataflow A running then the results and then Dataflow B uses Dataflow A my understanding is that Dataflow B will be using the results of Dataflow A AT THAT MOMENT .  Therefore, if A is still running and/or is hung up then B will get a result set that is impartial - or B could be hung up as well.

My solution is that if B is using A, you must make sure that A is fully complete or B will not be correct.

[jeffshieldsdev]

To clarify, B won't get a partial A result set, it'll get the last refreshed complete A result set.