topic Re: eventstream consumption (CU%) in Fabric platform

eventstream consumption (CU%)

joakimfenno — Fri, 27 Sep 2024 07:39:24 GMT

My fabric run out of capacity today

"Unable to complete the action because your organization's Fabric compute capacity has exceeded its limits. Try again later"

efter some troubleshooting and using teh fabric capacity metric app I understand that it is my event streams (2) that are the biggest consumers

The eventstream read messages from kafka.

I turned kafka (or the app that connect kafka to evenstream) off and expected the eventstream CU usage to go down but it did not go down at all

I have verified that no messages are processed through the eventstream

Can anyone explain this to me?

Re: eventstream consumption (CU%)

lbendlin — Fri, 27 Sep 2024 12:07:12 GMT

CUs are "smoothed out" over a 24 hr period. That means when you switch the consumer off your capacity will continue to be overloaded for many more hours.

What can you do about it? Move all workloads to a different capacity. Don't have one? Then you are stuck.

Re: eventstream consumption (CU%)

joakimfenno — Sun, 29 Sep 2024 10:37:17 GMT

I have waited more than 24 hours

It really seems like the event stream consumes a lot CU even when its not receiving any data

Re: eventstream consumption (CU%)

frithjof_v — Sun, 29 Sep 2024 12:34:08 GMT

Not all the visuals in the Capacity metrics app show smoothed CU consumption. Some visuals show the raw, unsmoothed CU consumption aggregated to a daily or hourly granularity.

Here is a great video about how to interpret the Capacity Metrics App:

https://youtu.be/EuBA5iK1BiA?si=Ibnz7o22ePEk-WMR

Re: eventstream consumption (CU%)

frithjof_v — Sun, 29 Sep 2024 12:26:29 GMT

Even with smoothing, you would see the consumption starting to decrease "immediately" if you have already had stable operation for more than a full 24 hours prior to shutting off the operation.

The smoothed consumption would gradually move towards zero, and would reach zero 24 hours after the operation was stopped.

I recommend this video about the Capacity metrics app, it made the Capacity metrics app a lot more useful to me after watching it: https://youtu.be/EuBA5iK1BiA?si=Ibnz7o22ePEk-WMR

Here is some information about Event Streams capacity consumption:

Microsoft Fabric event streams capacity consumption - Microsoft Fabric | Microsoft Learn

It seems strange that your Event Stream is still consuming CUs if there has been no data movement for more than 2 hours:

Re: eventstream consumption (CU%)

lbendlin — Sun, 29 Sep 2024 16:50:30 GMT

Even with smoothing, you would see the consumption starting to decrease "immediately" if you have already had stable operation for more than a full 24 hours prior to shutting off the operation.

That's a fallacy. The smoothing is even over the 24 hr period for each consumption event. There's nothing "decreasing immediately", especially not when you continue to have new consumption events.

Re: eventstream consumption (CU%)

frithjof_v — Sun, 29 Sep 2024 18:51:06 GMT

If we cancel a recurring activity, the impact of the smoothed CU consumption (red bars in my graphic below) will start to decrease almost immediately, as the smoothing period for the previous runs completes one by one.

The smoothed CU consumption stemming from the recurring activity will be fully reduced to zero 24 hours after cancelling the recurring activity.

Re: eventstream consumption (CU%)

joakimfenno — Mon, 30 Sep 2024 06:25:31 GMT

thanks for clarifying

I dont see that behaviour - it seems like the eventstream consumes a F4 capacity just by being active (and not processing any data)

Re: eventstream consumption (CU%)

frithjof_v — Mon, 30 Sep 2024 06:49:57 GMT

Sounds strange. I'd consider creating a support ticket for it. You could also ask if anyone over at Reddit has experienced anything similar: https://www.reddit.com/r/MicrosoftFabric/s/MyopVz2oZy

Re: eventstream consumption (CU%)

gustavoleo — Tue, 04 Mar 2025 16:19:48 GMT

Thanks for the heads up, please vote (1) Smoothing is not Smoothies - Microsoft Fabric Community