Forum Discussion
Compare Pipeline Run Capacity Usage
Hello esjetmore,
If these solutions do not resolve the issue, please consider raising a Microsoft support ticket. You can create a Microsoft support ticket using the link below:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticketIf this helps, kindly Accept it as a solution and give a "Kudos" so other members can find it more easily.
Thank you.
The Microsoft Fabric Capacity Metrics App primarily organizes data around capacity usage over time, which can make it challenging to analyze pipeline runs as discrete entities.
The Fabric Capacity Metrics App uses smoothing to distribute the compute resource consumption of operations over a longer period (e.g., 24 hours). This means that the reported usage may not align precisely with the start and end times of a pipeline run, as the consumption is spread across time. This smoothing mechanism helps avoid spikes in capacity reporting but complicates associating exact resource usage with individual pipeline executions
To access the semantic model of the Microsoft Fabric Capacity Metrics App and create custom reports that aggregate usage data by pipeline runs instead of time intervals, you can follow these steps. This process involves connecting to the app's semantic model and leveraging tools like DAX Studio or Power BI to extract and manipulate the data.
https://pbi-guy.com/2024/03/30/how-to-extract-data-from-the-fabric-metrics-app-part-1/
https://pbi-guy.com/2024/04/24/how-to-extract-data-from-the-fabric-metrics-app-part-2/
Understand Compute Unit (CU) Consumption:
- Each pipeline activity type consumes Fabric Capacity Units (CUs). For example:
- Data movement activities (e.g., Copy operations) consume 1.5 CU-hours per hour of execution.
- Orchestration activities (e.g., non-copy operations) consume 0.0056 CU-hours per activity run
Please see if this helps and accept the answer if this resolved the query.
Thank you, but that guide does not show how to organize the data around individual pipeline executions. It is still organized around time periods.
Hi esjetmore ,
unfortunately, I believe that the capacity metrics app as of now does not offer an easy way to achieve your desired comparison of individual pipeline runs. The app is inherently designed around temporal intervals and aggregates consumption by item over time.
What was suggested at FabCon last year was, that in cases where you want to investigate isolated runs, you have the possibility to create a designated "break out" capacities for it. This can be an F2 since it doesn't need a lot of compute. To investigate your isolated workload you can do the following:
- Create a new capacity (e.g. F2)
- Branch out to a new workspace, creating a copy of your current workspace
- Connect this workspace to your new capacity and make sure all scheduled workloads are disabled
- Run your pipeline on this separate workspace and observe the consumption in the capacity metrics app
Generally speaking, it makes a lot of sense to go wide in your capacity strategy, whether it is to detect high consumption workloads, isolate users who use up unproportionally much capacity or separate interactive and background loads if need be. Clinging to a single or few capacities will not provide sufficient flexibility when finding the capacity which suits your needs and therefore bears the risk of leading to higher costs in the long run.
I hope this helps you monitor your capacity consumptions! 🙂
