Forum Discussion

NotebookEnjoyer's avatar
NotebookEnjoyer
Icon for Advocate III rankAdvocate III
11 months ago

Inexplicable DataflowsStagingWarehouse Load on Capacity

Hello, in our daily load, we use Dataflows Gen2 for only a miniscule percentage of the data, and really only for copying the data (and removing 1 column maybe, nothing fancy). Nonetheless, Dataflows...
data engineering
sviesa's avatar
sviesa
New Member
1 day ago

Has anyone been able to determine the source of this issue? A seemingly identical issue started for us on a random Monday and it's continued since then. I'm working with Microsoft support but we haven't found a cause yet.

Using the Fabric Capacity Metrics app and the built in time detail feature one of the stagingwarehousefordataflows warehouse objects is being hit 24/7 with a mix of "OneLake Write via Redirect",  "OneLake Write via Proxy", and "High Scale Dataflow Compute - Warehouse Query" system activities hitting the warehouse object every minute - sometimes every 30 seconds.

Nothing in our pipeline changed when this activity started. We have 3 dataflows that run within 5 minutes. Our usage went up around 10% - which is a 150% increase for us.