Inexplicable DataflowsStagingWarehouse Load on Capacity

[NotebookEnjoyer]

Hello,

in our daily load, we use Dataflows Gen2 for only a miniscule percentage of the data, and really only for copying the data (and removing 1 column maybe, nothing fancy). Nonetheless, DataflowsStagingWarehouse and StagingWarehouseForDataflows use some 15 % of our capacity. In both cases, the relevant activity ist "High Scale Dataflow Compute - Warehouse Query".

Since the default Capacity Metrics App is rather trash, I can't correlate load peaks to specific times (without manually screening each and every timepoint detail, anyway). My best hunch is that this activity reflects queries from models to the lakehouse via the SQL endpoint, because in the Staging Lakehouses the main activity is "SQL Endpoint Query". But that would make no sense, right? After all, it's a staging warehouse for dataflows, it's right in the name!

So my main question is this: Why are the Staging Warehouses using this much capacity, although there is only insignificant use of Dataflows? And/or how can I find out/verify what is going on?

Thanks for any help!

[v-nmadadi-msft]

Hi @NotebookEnjoyer ,
Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.

Thank you for your understanding and assistance.

[v-nmadadi-msft]

Hi @NotebookEnjoyer  ,
Thanks for reaching out to the Microsoft fabric community forum.

High Scale Dataflow Compute: You're charged when staging is enabled, based on Lakehouse (Staging storage) and Warehouse (Storage Compute) SQL engine consumption duration.
For high-scale scenarios when staging is turned on queries run on the Lakehouse or Warehouse SQL engine. Each second of compute time uses 6 CU seconds, so longer queries consume more. This could be the reason for high capacity consumption.
If your dataflow uses staging, to find out how much High Scale compute you used, open the Fabric Capacity Metrics App and filter by your Dataflow’s name. Right-click the name, look for High Scale compute in the list of operations, and check the duration.

I hope this information helps. Please do let us know if you have any further queries.
Thank you

[NotebookEnjoyer]

As far as I know, my dataflows don't even use staging (all queries are italicized), which would explain why there is no high-scale compute in the Metrics App Tooltip:

And to reiterate: I'm not worried about the capacity consumption of the dataflows themselves, but about the consumption of the staging warehouse.

[v-nmadadi-msft]

Hi @NotebookEnjoyer ,

If you believe values are being calculated incorrectly in Fabric Capacity Metric App, Please consider reaching out to Microsoft Support. You can provide them with all the troubleshooting steps you've already taken, which will help them understand the issue better and provide a resolution. They might be able to identify something specific about your admin account setup or provide a solution that isn't immediately obvious. 

Below is the link to create Microsoft Support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thank you