Fastest possible access time for power platform dataflows? Gen2 DF in 2024

[dbeavon3]

It is taking a minimum of about ~10 seconds to get a tiny table out of power platform dataflows.

In the status message it says : "waiting for powerplatformdataflows", then "waiting for lakehouse".

Assuming it is a table of one record, it seems very extreme to be forced to wait 10 seconds for it.  It really impacts productivity during development (inner loop) if there is 10 seconds of overhead on every single table, regardless of how small.  I can compile a very large c# application in that amount of time.  Why can't I get a single record out of azure any faster than this?

Here is a sample of the PQ.  We have premium P1.

Any tips would be appreciated.

[GilbertQ]

Hi @dbeavon3 

While I can understand it feels like 10 seconds is a long time, you have to understand that even though you are using premium capacity, it still requires time to get the resources to then process what you need to be done.

[dbeavon3]

@GilbertQ 

If I had told you it took 60 seconds to retrieve a single 1KB row of data, would you have believed me?    Or 600 seconds?  I think 10 seconds overhead per table is crazy, and is hard to find a way to short-circuit that.  Ideally there would be a "mock fabric" or "sync fabric" or something else that we could run locally on the desktop (something incredibly fast, instead of incredibly slow).


Do you know if there is any way to dig into these delays to isolate where they are coming from?   Another thing I noticed is that the delays are far longer (eg. an additional ~10 seconds) when refreshing the final dataset, than they are when interacting with the gen2 dataflow in the "transform/editing" window.

I'm guessing there are some workarounds, assuming I can isolate the source of the delays.  Perhaps workarounds are avaialble without even needing to leaving Fabric.

... But as a last resort, customers might consider sending the results of our dataflows to an independent ADLS storage account, for the sake of fast and reliable retrieval:

https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-azure-data-lake-stora...

By setting some portions of Fabric aside, I'm certain that I can get predictable performance behavior by ingesting from a normal ADLS storage account.