Forum Discussion

JeffGray's avatar
JeffGray
Icon for Advocate I rankAdvocate I
2 years ago

Confused by CU-seconds calculation

I have a pipeline with just one data copy activity.  It ran for 1053 seconds duration.  Pricing documentation says pipeline copy activity uses 1.5 CU, so I would expect 1053 x 1.5 = 1580 CU-seconds. ...
General Question
Need Help
pipelines
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi JeffGray 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi @JeffGray 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

  • JeffGray's avatar
    JeffGray
    Icon for Advocate I rankAdvocate I
    2 years ago

    Thanks for the follow-up.  I guess I’m not sure if the question is answered or not.  Is there any way to tell in the capacity monitor tool (or anywhere else) whether/when parallel processing occurred?

     

    We’re trying to build a set of benchmark loads, so we can estimate the capacity/cost of scaling up to cover our whole environment. 1.5 vs. 6 is a huge difference!  If there’s a way to know for certain it was 4 processes running in parallel at ~1.5 each, with the duration thus reduced to roughly a quarter of what a single threaded process would take then I can use the CUs number. But if that’s not what it is, or I can’t be sure that’s what is causing it, then I can’t trust the number for scaling estimation purposes.

     

    So I guess the key question is whether we can see anywhere how many jobs ran in parallel for a particular pipeline copy activity or data flow.

     

    Thanks,

    Jeff Gray

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi JeffGray 

      In Microsoft Fabric, you can monitor the utilization of your resources using the Fabric Capacity Metrics App, also known as the metrics app. This app provides visibility into capacity resources and enables admins to see how much CPU utilization, processing time, and memory are utilized by data pipelines, dataflows, and other items in their Fabric capacity-enabled workspaces.

      However, it might not provide detailed information about the number of parallel processes for a particular pipeline copy activity or data flow. The app breaks down the item type (semantic model, notebook, pipeline, and others), and helps you to identify items or operations that use high levels of compute.

      Currently there is no option to see how many jobs ran in parallel for a pipeline.

      However, your suggestion is definitely valuable! We use customer feedback like yours to prioritize future features. The more users who request the ability to customize backgrounds, the higher it moves on our list.

       

      Appreciate if you could share the feedback on our Microsoft Fabric Ideas. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

       

      I hope this information helps. If you have any further queries please do let us know.

      Thanks

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi @JeffGray 
        We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
        Thanks