Forum Discussion
Inexplicable DataflowsStagingWarehouse Load on Capacity
Hello,
in our daily load, we use Dataflows Gen2 for only a miniscule percentage of the data, and really only for copying the data (and removing 1 column maybe, nothing fancy). Nonetheless, DataflowsStagingWarehouse and StagingWarehouseForDataflows use some 15 % of our capacity. In both cases, the relevant activity ist "High Scale Dataflow Compute - Warehouse Query".
Since the default Capacity Metrics App is rather trash, I can't correlate load peaks to specific times (without manually screening each and every timepoint detail, anyway). My best hunch is that this activity reflects queries from models to the lakehouse via the SQL endpoint, because in the Staging Lakehouses the main activity is "SQL Endpoint Query". But that would make no sense, right? After all, it's a staging warehouse for dataflows, it's right in the name!
So my main question is this: Why are the Staging Warehouses using this much capacity, although there is only insignificant use of Dataflows? And/or how can I find out/verify what is going on?
Thanks for any help!
7 Replies
- csm
Advocate I
We've recently started seeing entries for "High Scale Dataflow Compute - Warehouse Query" in the Capacity Metric App under Background records. These were not present a couple months ago and we had a good baseline on the workspace's total compute. Our dataflows have not changed but suddenly we're seeing a 6% increase in base capacity used under this workspace, attributed to these high-scale entries.
Within this workspace, no dataflows have staging enabled (all queries are italicized), but we are using SQL data destinations so that may be the underlying usage. I initially thought the billing was just broken out differently in the metric app, but the % base capacity used by the entire workspace has increased by the amount from this line. It doesn't show under the tooltip for each dataflow either, as NotebookEnjoyer experienced. The app is no help to narrow this down further.
- AnonymousNot applicable
Hi NotebookEnjoyer ,
Thanks for reaching out to the Microsoft fabric community forum.
High Scale Dataflow Compute: You're charged when staging is enabled, based on Lakehouse (Staging storage) and Warehouse (Storage Compute) SQL engine consumption duration.
For high-scale scenarios when staging is turned on queries run on the Lakehouse or Warehouse SQL engine. Each second of compute time uses 6 CU seconds, so longer queries consume more. This could be the reason for high capacity consumption.
If your dataflow uses staging, to find out how much High Scale compute you used, open the Fabric Capacity Metrics App and filter by your Dataflow’s name. Right-click the name, look for High Scale compute in the list of operations, and check the duration.
I hope this information helps. Please do let us know if you have any further queries.
Thank you- NotebookEnjoyer
Advocate III
As far as I know, my dataflows don't even use staging (all queries are italicized), which would explain why there is no high-scale compute in the Metrics App Tooltip:
And to reiterate: I'm not worried about the capacity consumption of the dataflows themselves, but about the consumption of the staging warehouse.
- AnonymousNot applicable
Hi NotebookEnjoyer ,
If you believe values are being calculated incorrectly in Fabric Capacity Metric App, Please consider reaching out to Microsoft Support. You can provide them with all the troubleshooting steps you've already taken, which will help them understand the issue better and provide a resolution. They might be able to identify something specific about your admin account setup or provide a solution that isn't immediately obvious.
Below is the link to create Microsoft Support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thank you
- AnonymousNot applicable
Hi NotebookEnjoyer ,
Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.Thank you for your understanding and assistance.
- sviesaNew Member
Has anyone been able to determine the source of this issue? A seemingly identical issue started for us on a random Monday and it's continued since then. I'm working with Microsoft support but we haven't found a cause yet.
Using the Fabric Capacity Metrics app and the built in time detail feature one of the stagingwarehousefordataflows warehouse objects is being hit 24/7 with a mix of "OneLake Write via Redirect", "OneLake Write via Proxy", and "High Scale Dataflow Compute - Warehouse Query" system activities hitting the warehouse object every minute - sometimes every 30 seconds.Nothing in our pipeline changed when this activity started. We have 3 dataflows that run within 5 minutes. Our usage went up around 10% - which is a 150% increase for us.
- jubinsoni
Resolver II
Hi sviesa,
Your detail about activity every 30-60 seconds, 24/7, while your three dataflows only run for about 5 minutes, is the key clue. Dataflow refreshes can't produce that pattern, so something other than the refreshes is talking to the staging warehouse. While your support case runs, I would narrow it down with a few elimination tests.
1. Prove it is (or isn't) the dataflows. Pause all three dataflow schedules for a few hours and watch the staging warehouse item in the Capacity Metrics app:
schedules paused, activity continues -> not the refreshes: something else is polling schedules paused, activity stops -> tied to the dataflows: go to step 22. Turn off staging where you don't need it. For simple copy-and-drop-a-column queries landing in a Lakehouse, staging adds a warehouse round trip without benefit. In each dataflow, right-click each query and clear Enable staging, then republish and compare the next day's CU on the staging items.
3. Look for anything querying through the staging artifacts. The "High Scale Dataflow Compute - Warehouse Query" activity every minute looks like a consumer rather than a writer. Things worth checking:
semantic models / reports connected to a dataflow (Dataflows connector) with frequent refresh or DirectQuery Power Automate / Logic Apps flows polling dataflow output external tools (Excel, notebooks) connected to the dataflow monitoring scripts calling dataflow or workspace APIsConsumers connected through the dataflow connector read via the staging warehouse, so a single report on a short refresh interval, or a DirectQuery model, would produce exactly this "every minute" pattern.
4. Give support the timeline. The fact that it started on a specific Monday with no pipeline change is useful evidence:
staging warehouse item name + workspace ID exact start date of the extra activity operation names + CU per hour from Capacity Metrics result of the "schedules paused" test aboveSo my overall approach would be:
pause dataflow schedules -> does the activity stop? -> disable staging on simple queries -> find consumers reading through the dataflow connector -> share the timeline and test results with supportThe pause test alone will tell you within a few hours whether you are looking for a refresh problem or a hidden consumer.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.