Compare Pipeline Run Capacity Usage

[esjetmore]

Hello! I'm trying to compare runs of a pipeline to see how much of the Fabric capacity they use with each run. Is there a way to easily quantify and compare that? I have the Fabric Capacity Metrics report, but this seems organized around moments in time and not the pipeline run as a whole. Thanks for your help!

[v-ssriganesh]

Hi @esjetmore,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[esjetmore]

Thanks. None of these responses have helped me get any closer to quantifying my pipeline runs individually. The issue persists. 

[v-ssriganesh]

Hello @esjetmore,
If these solutions do not resolve the issue, please consider raising a Microsoft support ticket. You can create a Microsoft support ticket using the link below:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

If this helps, kindly Accept it as a solution and give a "Kudos" so other members can find it more easily.

Thank you.

[v-ssriganesh]

Hi @esjetmore,
Could you please confirm if this issue has been resolved? If it has, kindly mark it as the solution and share the details in the community to help other members.
Thank you.

[v-ssriganesh]

Hi @esjetmore,
Thank you for reaching out to the Microsoft Fabric community forum with your query.

The Fabric Metrics App doesn’t natively provide pipeline-level CU usage here are two alternative approaches to estimate pipeline-level capacity usage:

By connecting Power BI to the Fabric Metrics App dataset, you can create a comprehensive view that aligns CU usage trends with pipeline run timestamps. This approach allows you to apply DAX measures to quantify and compare the capacity usage for each pipeline run. This method provides a more granular and accurate analysis of CU usage over time.

(OR)

Even though per-run CU usage isn’t explicitly logged, Microsoft Fabric provides CU consumption rates per activity type, allowing you to estimate usage.

• Copy Activities → ~1.5 CU-hours per hour of execution.
• Orchestration Activities → ~0.0056 CU-hours per activity run.

By multiplying these values with pipeline execution durations, you can estimate relative capacity usage per run.

For your better understanding please refer the below link:
https://learn.microsoft.com/en-us/fabric/data-factory/pricing-pipelines#compute-estimated-costs-usin...

If this helps, then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.

Thank you.

[nilendraFabric]

The Microsoft Fabric Capacity Metrics App primarily organizes data around capacity usage over time, which can make it challenging to analyze pipeline runs as discrete entities.

The Fabric Capacity Metrics App uses smoothing to distribute the compute resource consumption of operations over a longer period (e.g., 24 hours). This means that the reported usage may not align precisely with the start and end times of a pipeline run, as the consumption is spread across time. This smoothing mechanism helps avoid spikes in capacity reporting but complicates associating exact resource usage with individual pipeline executions

To access the semantic model of the Microsoft Fabric Capacity Metrics App and create custom reports that aggregate usage data by pipeline runs instead of time intervals, you can follow these steps. This process involves connecting to the app's semantic model and leveraging tools like DAX Studio or Power BI to extract and manipulate the data.

https://pbi-guy.com/2024/03/30/how-to-extract-data-from-the-fabric-metrics-app-part-1/

https://pbi-guy.com/2024/04/24/how-to-extract-data-from-the-fabric-metrics-app-part-2/

Understand Compute Unit (CU) Consumption:

• Each pipeline activity type consumes Fabric Capacity Units (CUs). For example:
  • Data movement activities (e.g., Copy operations) consume 1.5 CU-hours per hour of execution.
  • Orchestration activities (e.g., non-copy operations) consume 0.0056 CU-hours per activity run

    Please see if this helps and accept the answer if this resolved the query.

[esjetmore]

Thank you, but that guide does not show how to organize the data around individual pipeline executions. It is still organized around time periods. 

[ObungiNiels]

Hi @esjetmore ,

unfortunately, I believe that the capacity metrics app as of now does not offer an easy way to achieve your desired comparison of individual pipeline runs. The app is inherently designed around temporal intervals and aggregates consumption by item over time. 

What was suggested at FabCon last year was, that in cases where you want to investigate isolated runs, you have the possibility to create a designated "break out" capacities for it. This can be an F2 since it doesn't need a lot of compute. To investigate your isolated workload you can do the following: 

• Create a new capacity (e.g. F2)
• Branch out to a new workspace, creating a copy of your current workspace
• Connect this workspace to your new capacity and make sure all scheduled workloads are disabled
• Run your pipeline on this separate workspace and observe the consumption in the capacity metrics app

Generally speaking, it makes a lot of sense to go wide in your capacity strategy, whether it is to detect high consumption workloads, isolate users who use up unproportionally much capacity or separate interactive and background loads if need be. Clinging to a single or few capacities will not provide sufficient flexibility when finding the capacity which suits your needs and therefore bears the risk of leading to higher costs in the long run.  

I hope this helps you monitor your capacity consumptions! 🙂