Forum Discussion
Resolver IWhy is there CU usage while no activity running?
Community ChampionWhen we use the Multi Metric Ribbon Chart to drill down to a specific day and a specific hour (for example 1 am), and then click on the specific hour on the x-axis of the Multi Metric Ribbon Chart, this crossfilters the Items (14 days) visual so that it changes name to Items (1 day).
However, when we select a specific day and hour combination in the Multi Metric Ribbon Chart, we are actually able to see the list of which artifacts are consuming CU (s) in that specific hour, by looking at the Items (1 day) visual.
Which artifacts in your capacity are using CU (s) at that time? What type of items do you have in the Items (1 day) visual when you use the Multi Metrics Ribbon Chart to cross-filter down to a specific day and hour? For example, 1 am on Oct 13. This is very useful to understand what is going on.
The two visuals mentioned above show the actual, raw consumption. So these are the visuals which are easiest to interpret.
The CU (%) Over Time visual shows the smoothed consumption, so it is a bit more difficult to interpret. However, if the CU (%) Over Time visual is rising in the specified hour, there must be some artifacts in the Item (1 day) visual that tells which artifacts are the cause of the rise.
The drill-through page (Timepoint) also shows smoothed consumption mainly, so that page is also more difficult to interpret. Also note that a single refresh of a semantic model will trigger several background operations. So it's better to use the main page's Multi Metric Ribbon Chart to cross-filter the Items (14 days) visual in order to identify which artifacts are the cause of the CU (s) consumption. And use the tooltip in the Items (14 days) visual to see which kind of operations are consuming CU (s) for your items.
Resolver II did extensive analysis of these background operations.
they starts as soon as I pause and resume my instance, before conducting any type of activity.
and like I said, I can go up to 200 000 background operations!!!!
and the system is counting terminated activities in the background % usage for days.
I was able to track a background activity related to a simple SQL query consuming my CU for more than 24h!!!
like currently I have 0.625CU consumed by a process terminated yesterday (a pipeline), the dataflow consumed 1800CU, paid yesterday.
so this background activity will cost me 53 568CU per 24h for something which cost only 1800CU during its execution and was burndown in seconds.
and when I'll pause my instance, MS will charge me for these CUs!!!!
I'm curious to know if you see the same behaviors. a standard process staying in background operation for 1 day or more. and a consumption during this time greater than the activity itself.
- frithjof_v
All background operations consume CU (s) for 24 hours after they have ended. This is smoothing. We don't actually pay for the operation while it is running, but we start paying for it once it has ended. It's like paying down a loan over a 24 hour period, in 30-second installments. There are 2880 installments on each loan. I.e. (24 hours x 60 minutes/hour x 60 seconds/minute) / (30 seconds/installment).
Interactive operations are usually paid down over a 5 minute downpayment period, i.e. ten 30-second installments.
This makes the CU % Over Time visual (and the Timepoint page) a bit complicated.
If we use the Multi Metric Ribbon Chart and the Items (14 days) visuals instead, we don't need to worry about this, because those two visuals show the raw, unsmoothed consumption (down to the hour when the operation was actually executed). In these two visuals, we can see the CU (s) usage when it actually happened. So I highly recommend using those two visuals.
Most of the other visuals just show the payment for the usage, after the operation has actually ended. An example of this is the Timepoint page.
In general, a background operation will enter the Timepoint page after the End time of the operation, and it will stay on the Timepoint page until 24 hours after the End time of the operation.
For the interactive operations, they will enter the Timepoint page after the End time of the operation, and will usually stay on the Timepoint page until 5 minutes after the End time of the operation.
If we are experiencing more than 24 hours to pay down a background operation after it has ended, I think it must be because we have been using more than 100% of our capacity for some longer time intervals (or, we had some very high peaks far above 100%). It means we have been accumulating overages, and we need free funds (idle CU (s)) to pay down the overage. This means we will need to consume less than 100% for some time period, so we can use the idle CU (s) to pay down the overages. I think we can view the overages as deferred payments (carryforward), so some operations might be fully paid down later than 24 hours after they finished. However, as long as we just stay over 100% in the CU % Over Time visual for shorter periods, we shouldn't have any issues.
This GIF illustrates the concepts: FabricBurstingSmoothing5.gif (1920×1080) (dataplatformblogcdn.azureedge.net)
Please note, if we have spent above 100% for sufficiently long time, or sufficiently far above 100%, that our accumulated overages equal 100% utilization for 10 minutes (i.e. our overages cannot possibly be paid down within 10 minutes), we will experience throttling on interactive operations. Similarly, if our accumulated overages equal 100% utilization for 60 minutes, interactive operations will be rejected. And if our accumulated overages equal 100% utilization for 24 hours, then also background operations will get rejected. This is the mechanism for stopping us from overconsuming, and while I haven't experienced throttling on my trial capacity, I guess it can be really bad on a production capacity. So throttling and rejection are the situations we really want to avoid.
https://learn.microsoft.com/en-us/fabric/enterprise/throttling#future-smoothed-consumption
- Jerome22
I understand all of this.
But I'm not overconsuming
the problem is: the cumulative smoothing is about 10 times the cost of the initial activity.
even interactive activities stay in background for 24hours (any single SQL query...)
add to that activities coming from nowhere, like a lakehouse not used for the past weeks start consuming CUs for no reason.
and boom, you have 200 000+ background activities for 24hours consuming more than the available capacity.
its really crazy.
I hope that Microsoft will solve that, as they recognize the issue.
wait and see... but they have to solve this ASAP, has this cost us a lot of money
- frithjof_v
SQL queries are counted as background queries and are smoothed for 24 hours. https://learn.microsoft.com/en-us/fabric/enterprise/fabric-operations#data-warehouse
"the cumulative smoothing is about 10 times the cost of the initial activity." Can you describe how you found this number?
"add to that activities coming from nowhere, like a lakehouse not used for the past weeks start consuming CUs for no reason." What kind of transactions are you being billed for? Is it OneLake read transactions? This can be found in the Items (14 days) visual. Is the Lakehouse being queried? E.g. are some Power BI reports using the Lakehouse data, or some other processes using the Lakehouse data?
"I did extensive analysis of these background operations.
they starts as soon as I pause and resume my instance, before conducting any type of activity." What kind of operations are these? Is it SQL queries, dataflow gen2, spark notebook, pipeline, etc.?
"like currently I have 0.625CU consumed by a process terminated yesterday (a pipeline), the dataflow consumed 1800CU, paid yesterday." Background operations are smoothed for 24 hours after they finish. It's normal that operations from yesterday are still showing in the Timepoint page.
"and a consumption during this time greater than the activity itself." Can you describe what you mean by greater than the activity itself? How do you know the consumption of the activity itself? And where do you see that you are smoothed for more consumption than the activity itself?