Confused by CU-seconds calculation

[JeffGray]

I have a pipeline with just one data copy activity.  It ran for 1053 seconds duration.  Pricing documentation says pipeline copy activity uses 1.5 CU, so I would expect 1053 x 1.5 = 1580 CU-seconds. I don't understand why the capacity monitor app is showing 6480 CU-s for this activity.  Can anyone explain how this calculation works?  It seems the copy used 4x what it should have given its duration!!!! 

[DanielAmbler]

@JeffGray Did you ever find anything helpful?  The answer above is exactly the same as what I got when I looked via co-pilot for some useful resources.

I have the same issue where pipelines that take 10 seconds consume 360 CUs - cant for the life in me find any correlation after working through the plethora of articles surrounding the mystery of the costing calculations so i'm somewhat dubious about any calculations that follow.

For all I know CUs could be a made up number 🙂

[Anonymous]

Hi @JeffGray 
Thanks for using Fabric Community.

The calculation of CU-seconds for a data copy activity in Microsoft Fabric can be influenced by several factors. Here are a few possibilities:

Parallelism: Microsoft Fabric may run multiple copies of the same activity in parallel to speed up the process. This could increase the total CU-seconds consumed.

Overhead Time: The CU-seconds calculation might include some overhead time for setting up and tearing down the activity, which would be in addition to the actual runtime of the activity.

Activity Type: Different types of activities in Microsoft Fabric consume different amounts of CUs. For example, the pricing documentation mentions that data movement Copy activities consume 1.5 CU per hour. If your activity involves more than just data movement, it could consume more CUs.

Pricing for data pipelines - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

[Anonymous]

Hi @JeffGray 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

[Anonymous]

Hi @JeffGray 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

[JeffGray]

Thanks for the follow-up.  I guess I’m not sure if the question is answered or not.  Is there any way to tell in the capacity monitor tool (or anywhere else) whether/when parallel processing occurred?

We’re trying to build a set of benchmark loads, so we can estimate the capacity/cost of scaling up to cover our whole environment. 1.5 vs. 6 is a huge difference!  If there’s a way to know for certain it was 4 processes running in parallel at ~1.5 each, with the duration thus reduced to roughly a quarter of what a single threaded process would take then I can use the CUs number. But if that’s not what it is, or I can’t be sure that’s what is causing it, then I can’t trust the number for scaling estimation purposes.

So I guess the key question is whether we can see anywhere how many jobs ran in parallel for a particular pipeline copy activity or data flow.

Thanks,

Jeff Gray

[Anonymous]

Hi @JeffGray 

In Microsoft Fabric, you can monitor the utilization of your resources using the Fabric Capacity Metrics App, also known as the metrics app. This app provides visibility into capacity resources and enables admins to see how much CPU utilization, processing time, and memory are utilized by data pipelines, dataflows, and other items in their Fabric capacity-enabled workspaces.

However, it might not provide detailed information about the number of parallel processes for a particular pipeline copy activity or data flow. The app breaks down the item type (semantic model, notebook, pipeline, and others), and helps you to identify items or operations that use high levels of compute.

Currently there is no option to see how many jobs ran in parallel for a pipeline.

However, your suggestion is definitely valuable! We use customer feedback like yours to prioritize future features. The more users who request the ability to customize backgrounds, the higher it moves on our list.

Appreciate if you could share the feedback on our Microsoft Fabric Ideas. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

I hope this information helps. If you have any further queries please do let us know.

Thanks

[Anonymous]

Hi @JeffGray 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks