Why is there CU usage while no activity running?

[Jerome22]

I'm checking the capacity report to try to understand the usage.

but I found that I have a constant 12CU(s) consumed all the time.

in the middle of the night, when I have nothing running, CU(s) are consumed and the capacity report is not able to identify what is consuming this.

I'm using F2 for now and 12CU(s) = 21%!!!

today there was near no activity (3 at 10AM (for a total of 4min) and 1 at 3PM  (for 5 seconds)) and the throttling counter is increasing the day long for no reason.

this causes capacity errors everywhere.

so did the CU(s) calculation can be trusted?

How can we validate what Microsoft is billing us?

because for now I think it's completely broken.

can we have a real time counter of the current CU used and which resource use it?

[frithjof_v]

Please watch this YouTube video about the Fabric Capacity Metrics App. Hopefully you will be able to find the answer to the issue after watching it:

Tips on Using the Fabric Capacity Metrics App

https://youtu.be/EuBA5iK1BiA?si=8hm207pOqcrn6pt7

I think what confuses many users, is the fact that some visuals in the Metrics app show the smoothed consumption (while other visuals show the raw consumption). Please watch the video, it explains it quite well.

Also, please note that when you pause a capacity, the remaining cumulative overages and smoothed operations on your capacity are summed, and added to your Azure bill. https://learn.microsoft.com/en-us/fabric/enterprise/pause-resume

[Jerome22]

yes, I know the remaining CUs are billed.

but for now there is a bug, the system still counting terminated activities and background activities still appeared without any reason.

so my background CU is going higher and higher and never down like we can see anywhere else.

the extra CU I pay are just few minutes of CUs, so normally these CU should be recovered during IDLE time without any issue, but it's not the case.

for example, during 1am and 5am there is no activity in fabric, this mean that any terminated background activity can use this available capacity to "reimburse" Microsoft. but I'm seeing an increase in usage... 

in the video you sent, we can clearly see that the blue chart (background) is going up than down and than back to 0 most of the time, even after there was big front activities.

also in the video the number of background activities is pretty low, few hundreds.

in my case this value jump to hundred of thousands !!! yes more than 200 000 background activities, during IDLE time. really crazy.

and I'm not doing something complicated or resource intensive. (compared to the video)

well... there are issues and MS has to solve this. or improve the way background activities are "billed"

[frithjof_v]

When we use the Multi Metric Ribbon Chart to drill down to a specific day and a specific hour (for example 1 am), and then click on the specific hour on the x-axis of the Multi Metric Ribbon Chart, this crossfilters the Items (14 days) visual so that it changes name to Items (1 day).

However, when we select a specific day and hour combination in the Multi Metric Ribbon Chart, we are actually able to see the list of which artifacts are consuming CU (s) in that specific hour, by looking at the Items (1 day) visual.

Which artifacts in your capacity are using CU (s) at that time? What type of items do you have in the Items (1 day) visual when you use the Multi Metrics Ribbon Chart to cross-filter down to a specific day and hour? For example, 1 am on Oct 13. This is very useful to understand what is going on.

The two visuals mentioned above show the actual, raw consumption. So these are the visuals which are easiest to interpret.

The CU (%) Over Time visual shows the smoothed consumption, so it is a bit more difficult to interpret. However, if the CU (%) Over Time visual is rising in the specified hour, there must be some artifacts in the Item (1 day) visual that tells which artifacts are the cause of the rise.

The drill-through page (Timepoint) also shows smoothed consumption mainly, so that page is also more difficult to interpret. Also note that a single refresh of a semantic model will trigger several background operations. So it's better to use the main page's Multi Metric Ribbon Chart to cross-filter the Items (14 days) visual in order to identify which artifacts are the cause of the CU (s) consumption. And use the tooltip in the Items (14 days) visual to see which kind of operations are consuming CU (s) for your items.

[Jerome22]

I did extensive analysis of these background operations.

they starts as soon as I pause and resume my instance, before conducting any type of activity.

and like I said, I can go up to 200 000 background operations!!!!

and the system is counting terminated activities in the background % usage for days.

I was able to track a background activity related to a simple SQL query consuming my CU for more than 24h!!! 

like currently I have 0.625CU consumed by a process terminated yesterday (a pipeline), the dataflow consumed 1800CU, paid yesterday. 

so this background activity will cost me 53 568CU per 24h for something which cost only 1800CU during its execution and was burndown in seconds.

and when I'll pause my instance, MS will charge me for these CUs!!!!

I'm curious to know if you see the same behaviors. a standard process staying in background operation for 1 day or more. and a consumption during this time greater than the activity itself.

[frithjof_v]

All background operations consume CU (s) for 24 hours after they have ended. This is smoothing. We don't actually pay for the operation while it is running, but we start paying for it once it has ended. It's like paying down a loan over a 24 hour period, in 30-second installments. There are 2880 installments on each loan. I.e. (24 hours x 60 minutes/hour x 60 seconds/minute) / (30 seconds/installment).

Interactive operations are usually paid down over a 5 minute downpayment period, i.e. ten 30-second installments.

This makes the CU % Over Time visual (and the Timepoint page) a bit complicated.

If we use the Multi Metric Ribbon Chart and the Items (14 days) visuals instead, we don't need to worry about this, because those two visuals show the raw, unsmoothed consumption (down to the hour when the operation was actually executed). In these two visuals, we can see the CU (s) usage when it actually happened. So I highly recommend using those two visuals.

Most of the other visuals just show the payment for the usage, after the operation has actually ended. An example of this is the Timepoint page.

In general, a background operation will enter the Timepoint page after the End time of the operation, and it will stay on the Timepoint page until 24 hours after the End time of the operation.

For the interactive operations, they will enter the Timepoint page after the End time of the operation, and will usually stay on the Timepoint page until 5 minutes after the End time of the operation.

If we are experiencing more than 24 hours to pay down a background operation after it has ended, I think it must be because we have been using more than 100% of our capacity for some longer time intervals (or, we had some very high peaks far above 100%). It means we have been accumulating overages, and we need free funds (idle CU (s)) to pay down the overage. This means we will need to consume less than 100% for some time period, so we can use the idle CU (s) to pay down the overages. I think we can view the overages as deferred payments (carryforward), so some operations might be fully paid down later than 24 hours after they finished. However, as long as we just stay over 100% in the CU % Over Time visual for shorter periods, we shouldn't have any issues.

This GIF illustrates the concepts: FabricBurstingSmoothing5.gif (1920×1080) (dataplatformblogcdn.azureedge.net)

Please note, if we have spent above 100% for sufficiently long time, or sufficiently far above 100%, that our accumulated overages equal 100% utilization for 10 minutes (i.e. our overages cannot possibly be paid down within 10 minutes), we will experience throttling on interactive operations. Similarly, if our accumulated overages equal 100% utilization for 60 minutes, interactive operations will be rejected. And if our accumulated overages equal 100% utilization for 24 hours, then also background operations will get rejected. This is the mechanism for stopping us from overconsuming, and while I haven't experienced throttling on my trial capacity, I guess it can be really bad on a production capacity. So throttling and rejection are the situations we really want to avoid.

https://learn.microsoft.com/en-us/fabric/enterprise/throttling#future-smoothed-consumption

[Jerome22]

I understand all of this.

But I'm not overconsuming

the problem is: the cumulative smoothing is about 10 times the cost of the initial activity.

even interactive activities stay in background for 24hours (any single SQL query...)

add to that activities coming from nowhere, like a lakehouse not used for the past weeks start consuming CUs for no reason.

and boom, you have 200 000+ background activities for 24hours consuming more than the available capacity.

its really crazy.

I hope that Microsoft will solve that, as they recognize the issue.

wait and see... but they have to solve this ASAP, has this cost us a lot of money

[frithjof_v]

SQL queries are counted as background queries and are smoothed for 24 hours. https://learn.microsoft.com/en-us/fabric/enterprise/fabric-operations#data-warehouse

"the cumulative smoothing is about 10 times the cost of the initial activity." Can you describe how you found this number?

"add to that activities coming from nowhere, like a lakehouse not used for the past weeks start consuming CUs for no reason." What kind of transactions are you being billed for? Is it OneLake read transactions? This can be found in the Items (14 days) visual. Is the Lakehouse being queried? E.g. are some Power BI reports using the Lakehouse data, or some other processes using the Lakehouse data?

"I did extensive analysis of these background operations.

they starts as soon as I pause and resume my instance, before conducting any type of activity." What kind of operations are these? Is it SQL queries, dataflow gen2, spark notebook, pipeline, etc.?

"like currently I have 0.625CU consumed by a process terminated yesterday (a pipeline), the dataflow consumed 1800CU, paid yesterday."  Background operations are smoothed for 24 hours after they finish. It's normal that operations from yesterday are still showing in the Timepoint page.

"and a consumption during this time greater than the activity itself." Can you describe what you mean by greater than the activity itself? How do you know the consumption of the activity itself? And where do you see that you are smoothed for more consumption than the activity itself?

[Jerome22]

I sent a example:

I have 0.625CU consumed for 24h, related to a pipeline which consumed 1800CU. 

so this background activity will cost me 54 000CU per 24h for something which cost only 1800CU during its execution and was burndown in seconds.

0.625 * 60 * 60 * 24 = 54 000 CU in total

if the impact on my capacity is 0.625, this activity should consumed my capacity for 48 minutes

0.625 * 60 * 48 = 1800

else I expect, 1800 / (24 * 60 * 60) = 0.0208 CU consumed for 24h

for my lakehouse, yes, I had activities 10 days ago, nothing after that.

it's just an example, I'm seeing activities poping from everywhere.

remember: I'm reaching 200 000 + background activities!!! 

just by running few pipelines a day, nothing complicated. and the number of user is pretty low for now (mainly myself and few people running 5 reports in the day)

[frithjof_v]

On my trial capacity (I am the only user), I have 75 000 background activities being smoothed at the moment. It doesn't mean any of these background activities are running now. This is the number of background activities which ended during the last 24 hours. 

These 75 000 background activities are costing me 10% of my trial capacity.

"I have 0.625CU consumed for 24h, related to a pipeline which consumed 1800CU. 

so this background activity will cost me 54 000CU per 24h for something which cost only 1800CU during its execution and was burndown in seconds."

I am guessing the Total CU (s) is 1800CU.

0.625 CU (s) / 30 s * 60 s/min * 60 min/h * 24 h = 1800CU (s).

So I am guessing you will pay in total 1800CU (s).

I think this makes sense.

The Total CU (s) is the cost of the operation. I don't think you will pay 54000CU. I think you will pay 1800CU (s). What makes you think 54000CU is the amount you will pay?

I think these two visuals (red boxes) are the ones which provide the most useful information. Here we can find the actual resource usage, and what day and what hour the usage really happened. Yes, it will be billed over a 24 hour period after the usage actually happened. But these two visuals show when the usage actually happened, which I prefer.

[ajm45]

Hi @Jerome22,

I'm having the same issue.  I'm trialling Fabric (on an F8) and the week before last I saw that I seemed to be using a lot of CUs for background tasks when I had nothing running.  Last week I decided not to use Fabric at all to eliminate completely anything that I was doing, I had no scheduled jobs running and no queries being made to my Lakehouse’s, Warehouses or Dataflows, no refreshes of Dataflows or data models and I didn’t view any reports that would be using my Fabric data either.  The only thing I had was 5 mirrored dbs (all MS documentation that I've read states that these should not use CUs to keep in sync, should only be using CUs when you query them, unless I’ve misinterpreted anything) and I have shortcuts setup in my Lakehouse’s (again these should not be using CUs as no data is being copied, they are shortcuts).  As I said, none of my dataflows or pipelines that take the data from the Lakehouse, carry out transformationss and land the data in my Warehouse are running or scheduled to run, yet all last week every day I seemed to be using between 15% and 20% of my CUs on background tasks.  Last Wednesday I had supposedly used 96,441 CUs, on Thursday 140,000 and on Friday 110,000 all when I'm doing nothing and have nothing running.

I'm really liking Fabric and all the new capabilities that are available but I'm really not liking that 15 to 20% of my daily CUs on an F8 are being used when I'm not doing anything.  I'm already quite concerned that when I have been running jobs that they are using a lot of CUs and this is only a small amount of what I would like to have running on Fabric.

I hope that you got some answers or its resolved for you.  I will also be raising an MS support ticket.

[jasonn]

I am having the same issue. I thought it was normal for there to always be CU activity because I have never had anything but. Today I realized it didn't make sense that when there is nothing running there should be no or minimal CU activity. I run the dataflows twice a day and this is my CU usage... flatline.... ~20% usage on a 32.

I opened a support ticket as well with screen shots, my first interaction were links to two documents I've already read. I need an answer to - if no processes are running - what is the expected CU?

Anyone make any progress on this?

2408220040006359

[Jerome22]

Hi,

first... you are "lucky", your blue line is going down... mine is going up , never down! 🙂

and I have to pause and resume my instance to solve the problem (the big spikes are when I pause and resume)

I discussed with MS for my case, which is the same as yours.

but I don't understand clearly how the background activities are calculated.

it's far from a clear answer.

they agree that there is a problem and some activities should not be there in the CU usage.

but they were not able to provide any ETA to solve the issue.

did you get some info on your side?

[Jerome22]

MS is still unable to solve the problem for me.

and we are suffering billing issues, we are paying more than our capacity!

so check if it's the case for you too.

at least for us we are in F2 mode and some day we are paying 23$ isntead of 13$, not a so big issue, but still a concern. 

[Jerome22]

for info.

I increased my capacity to see how fast the system can recover. 

there was a drop just after this change in capacity, but not enough to make the environment useable again.

there is still a plenty of CU to burn down for no reason.

I finally pause my Fabric Capacity.

I got a spike of 112 000 CUs (or about 101 000% of CU usage)

Then I restart my capacity and everything was back to the normal. no more CU consumed for no reason.

and finally I run a simple pipeline which took 5min to copy some data.

and boom, again I have a constant CU usage of 31% !!!! (1h after I ran the pipeline and everything is IDLE)

the capacity metric report is definitly broken or the CU evaluation is broken.

I no longer have processes consuming my capacity for no reason.

[v-cboorla-msft]

Hi @Jerome22 

Thanks for using Microsoft Fabric Community.

Apologize for the issue you are facing. The best course of action is to open a support ticket and have our support team take a closer look at it.

Please reach out to our support team so they can do a more thorough investigation on why this it is happening: Microsoft Fabric Support and Status | Microsoft Fabric

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Meanwhile you can try the below steps that might help you.

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to Microsoft Fabric might be causing the issue. Clearing the cache removes this potentially problematic data.

Hard Refresh: A hard refresh bypasses the cached version of the webpage and forces the browser to download the latest version from the server. Press Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac).

I hope this information helps.

Thank you.

[Jerome22]

Hi,

For info, the dev teams is still not able to solve the issue after 3 weeks.

we also found that the price we are paying is more than the capacity we setup!

for now we are in F2 (plan to upgrade later with an agreement when the problem will be solved) but paying up to 50% more some days! (23 CAD$ instead of 13$)

I start to be very worried about the capacity of your teams to support Fabric. if the devs are not able to find the root cause for something like this, this means the product has a lot (too much) issues to solve everywhere else.

[Jerome22]

I opened a ticket 2406170040005044

finally I found that ALL the background jobs are all using CU and never really stopped.

I got spikes to more than 100 000 background operations!!!! for no reason

I just reached my 100% CU usage consumed by terminated background executions. this number is increasing and never decreasing.

because I reached 100% of CU consumed by background tasks, I can't burndown my standard activities anymore and my capacity usage % will increase until I pause the capacity.

there is defintly something wrong on the MS side there.

I sent the screenshots and informations to the support team.