Forum Discussion

smeetsh's avatar
smeetsh
Icon for Continued Contributor rankContinued Contributor
1 year ago
Solved

CU Usage per pipeline run

Hi All,

 

We are running into our limit for our F4 SKU. I am trying to see if there is one, or a specific set of pipelines that over uses, causing our 24 hours background usage to creep up.

 

Is there a way i can see, or calculate, how much CU's a specific pipeline uses? I know the CU's are per 30 seconds, so if it used, for instance, 200 CU's and the pipeline runs 1 minute, I know that that pipeline consumed 100CU's per 30 seconds, which would be withing the F4 capacity (which is 120)


I dont see anyway in the reporting to do this.

Cheers

Hans

8 Replies

  • v-echaithra's avatar
    v-echaithra
    Icon for Community Support rankCommunity Support

    Hi smeetsh ,

    Thank you for reaching out to Microsoft Community.

    Azure Data Factory does not provide direct CU consumption metrics per pipeline, making granular tracking a challenge. However, you can approximate usage and implement monitoring strategies to optimize performance and avoid throttling.

    Step 1: Understand F4 Capacity Limits

    F4 Integration Runtime offers a maximum of 120 CU/sec, equivalent to: 7,200 CU/min and 432,000 CU/hr
    CU usage is calculated in 30-second intervals, with throttling enforced over a 24-hour sliding window.

    Step 2: Manually Estimate CU Usage per Pipeline

    While ADF doesn’t expose CU metrics per pipeline, you can estimate consumption using:
    Formula: CU Usage = Activity Duration (sec) × CU/sec (per activity type)

    To gain visibility into CU usage trends you can also enable ADF Diagnostic Logs and route them to Log Analytics.

    Navigate to: Azure Portal > ADF Resource > Monitoring > Integration Runtime > Metrics > CU Consumption
    Tag high-usage pipelines using naming conventions or metadata to aid in tracking and optimization.

    Hope this helps,
    Warm Regards
    Chaithra E.

  • v-echaithra's avatar
    v-echaithra
    Icon for Community Support rankCommunity Support

    Hi smeetsh ,

    Thank you wardy912  for your inputs.

    We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

    Thank you for your patience and look forward to hearing from you.

    Best Regards,
    Chaithra E.



  • smeetsh's avatar
    smeetsh
    Icon for Continued Contributor rankContinued Contributor

    v-echaithra The f4 SKU is not 120CU /second but per 30 seconds

     

    wardy912  We already use the fabric metrics app, but its in near impossible to see an individual run of a pipeline, to determine if a pipeline burts over the, in my case, 120CU per 30 seconds.  I have now resorted to rename a pipeline with like a date or a change i made, to see if my CU's per run become more efficient, which is messy and only works on pipelines that not in production yet. I have not tried FUAM yet.

    • v-echaithra's avatar
      v-echaithra
      Icon for Community Support rankCommunity Support

      Hi smeetsh ,

      You're absolutely right the f4 SKU is 120CU per 30 seconds.
      You can try using activity type estimates. You can combine activity duration + activity type + concurrency to build a rough profile. Additionally you can enable ADF Diagnostic Logs and send to Log Analytics. In Log Analytics, use KQL (Kusto Query Language) to query per pipeline run where can then estimate CU usage using rules of thumb for CU/sec by activity type. Hope you tried FUAM, that may give more granular, visually organized usage metrics across workloads.

      Thank you.

      • smeetsh's avatar
        smeetsh
        Icon for Continued Contributor rankContinued Contributor

        Will definitely try FUAM. just haven't been able to get around to it due to workload. 😄

  • v-echaithra's avatar
    v-echaithra
    Icon for Community Support rankCommunity Support

    Hi smeetsh ,

    We’d like to follow up regarding the recent concern. Have you got any chance to try FAUM. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

    Best Regards,
    Chaithra E.

    • smeetsh's avatar
      smeetsh
      Icon for Continued Contributor rankContinued Contributor

      I have looked into FUAM, but it does need some setting up on our end. It is on the to do list , like so many things LOL