Fabric-Price higher then assumed

[marius1106]

Dear Microsoft,

my Azure Bill was higher then expected. I searched for a reason and saw that often there is another position which costs extra to the Capacity-runtime: "Data Movement" or "Spark Memory Optimized". 
For example in 08.01.2025 - 14.01.2025 I run the Fabric-Capacity 91 hours and paused it every day after work. So I assumed a price of 91h * 0,42€ (F2 Germany west central) = 38,43 €

But I had to pay 61,65€:

- So do I have to pay for Data Movement-Operation extra in generell? Because the azure price-calculator doesnt show a position like this.

- Or is the "data movement"-position on the bill the remaining cumulative overages and smoothed operations on my capacity that is added on top because I paused the capacity after work?

- I just want to understand the extra costs on top to my 38,43€ that I calculated, because I thought that the price for the fabric capacity is only dependant on the duration of running fabric capacity. 

Thank you 🙂

[v-karpurapud · Accepted solution]

Hello @marius1106 

Thank you for contacting the Microsoft Fabric community. We understand you are experiencing issues with Azure usage and billing. Let's address your queries one by one:

1. Data Movement Operations: When data is moved within Fabric, such as during Copy activities in Data Pipelines, it incurs additional costs. These actions use Capacity Units (CUs) and are billed separately from the base Fabric capacity cost.

2. Spark Memory Optimized Operations: These charges are related to the use of Spark for data processing. Running Spark jobs consumes additional CUs based on the memory and compute resources used.

• Data Movement Operations have separate charges and are not included in the base Fabric Capacity cost. The Azure Price Calculator may not show these costs, which depend on your Fabric usage, especially for large data movements in OneLake, Dataflows, or Lakehouse operations.

• Some data movement charges may arise from operations that continued even after you paused capacity. Data movement costs occur when Fabric transfers data between storage layers or processing engines. Tasks like ETL processes, queries, or data refreshes could have triggered these extra charges.

• Fabric Capacity charges cover only compute runtime, but Data Movement and Spark Memory Optimized operations add extra costs.Your bill exceeds 38.43€ because these operations use additional CUs, which are billed separately from the base capacity cost.

If my response has resolved your query, please mark it as the Accepted Solution to assist others. Additionally, a 'Kudos' would be appreciated if you found my response helpful.

Thank you!

[marius1106 · Accepted solution]

Thank you for your help!

How can I calculate the Data-movement and Spark costs?
 - Is there a metric like for example 4,90€ per TB Data movement or something?

- Or like for example 0,50€ per hour Spark runtime for cluster?

[v-karpurapud · Accepted solution]

Hi @marius1106 

You can estimate Data Movement and Spark costs by reviewing "Data Movement CU Consumption" and "Spark Execution CU Consumption" in Fabric Capacity Metrics App. Additionally, Azure Monitor logs (if enabled) can help analyze data movement-related CU usage, since Spark costs are based on memory and compute consumption rather than a per-hour/per-cluster model.

For more information on calculating these costs, refer to the following Microsoft documentation:

Microsoft Fabric - Pricing | Microsoft Azure

Pricing for data pipelines - Microsoft Fabric | Microsoft Learn

Billing and utilization reports in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

By understanding these metrics and utilizing the Fabric Metrics App, you can better estimate and manage costs for Data Movement and Spark operations.

If you find this post helpful, please mark it as an "Accept as Solution" and give a KUDOS.

Thank you!

[v-karpurapud · Accepted solution]

Hi @marius1106 

Yes, your understanding is correct, but there is a small clarification regarding how Data Movement charges apply.

If your F2 capacity costs €0.42 per hour, then each CU/h costs €0.21 (since F2 provides 2 CU/h). If your Data Movement activities consume 3,600 CUs, the additional cost depends on whether this consumption fits within your allocated capacity. If the 3,600 CUs are consumed within your allocated F2 capacity (2 CU/h), then there is no extra charge then it is covered by the base cost. If Data Movement operations exceed your allocated CUs, extra CUs are billed separately at pay-as-you-go rates.

Example Calculation:

• Base Cost: F2 capacity provides 2 CU/h at €0.42 per hour.
• Cost per CU: €0.21 per CU/h.
• Total CUs Consumed: 3,600 CUs
• Total Cost: 3,600 CUs * €0.21 per CU = €0.756

The additional cost would be calculated based on the pay-as-you-go rate for the excess CUs.


Thank You!

[p_jampana]

Hi @v-karpurapud ,

Thank you for the helpful insights and guidance on monitoring CU usage and optimising workloads is invaluable for managing costs effectively. I appreciate the resources shared.

[v-karpurapud · Accepted solution]

Hello @marius1106 

Thank you for contacting the Microsoft Fabric community. We understand you are experiencing issues with Azure usage and billing. Let's address your queries one by one:

1. Data Movement Operations: When data is moved within Fabric, such as during Copy activities in Data Pipelines, it incurs additional costs. These actions use Capacity Units (CUs) and are billed separately from the base Fabric capacity cost.

2. Spark Memory Optimized Operations: These charges are related to the use of Spark for data processing. Running Spark jobs consumes additional CUs based on the memory and compute resources used.

• Data Movement Operations have separate charges and are not included in the base Fabric Capacity cost. The Azure Price Calculator may not show these costs, which depend on your Fabric usage, especially for large data movements in OneLake, Dataflows, or Lakehouse operations.

• Some data movement charges may arise from operations that continued even after you paused capacity. Data movement costs occur when Fabric transfers data between storage layers or processing engines. Tasks like ETL processes, queries, or data refreshes could have triggered these extra charges.

• Fabric Capacity charges cover only compute runtime, but Data Movement and Spark Memory Optimized operations add extra costs.Your bill exceeds 38.43€ because these operations use additional CUs, which are billed separately from the base capacity cost.

If my response has resolved your query, please mark it as the Accepted Solution to assist others. Additionally, a 'Kudos' would be appreciated if you found my response helpful.

Thank you!

[marius1106 · Accepted solution]

Thank you for your help!

How can I calculate the Data-movement and Spark costs?
 - Is there a metric like for example 4,90€ per TB Data movement or something?

- Or like for example 0,50€ per hour Spark runtime for cluster?

[v-karpurapud · Accepted solution]

Hi @marius1106 

You can estimate Data Movement and Spark costs by reviewing "Data Movement CU Consumption" and "Spark Execution CU Consumption" in Fabric Capacity Metrics App. Additionally, Azure Monitor logs (if enabled) can help analyze data movement-related CU usage, since Spark costs are based on memory and compute consumption rather than a per-hour/per-cluster model.

For more information on calculating these costs, refer to the following Microsoft documentation:

Microsoft Fabric - Pricing | Microsoft Azure

Pricing for data pipelines - Microsoft Fabric | Microsoft Learn

Billing and utilization reports in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

By understanding these metrics and utilizing the Fabric Metrics App, you can better estimate and manage costs for Data Movement and Spark operations.

If you find this post helpful, please mark it as an "Accept as Solution" and give a KUDOS.

Thank you!

[marius1106]

So did I understand that correctly:

If I have a F2 Capacity which costs me 0,42€ per hour, than 1 CU/h is 0,21€ (0,42€ / 2) and if my Data Movement activities consume 3.600 CU(s) than it adds 0,21€ to the azure Bill?

[v-karpurapud · Accepted solution]

Hi @marius1106 

Yes, your understanding is correct, but there is a small clarification regarding how Data Movement charges apply.

If your F2 capacity costs €0.42 per hour, then each CU/h costs €0.21 (since F2 provides 2 CU/h). If your Data Movement activities consume 3,600 CUs, the additional cost depends on whether this consumption fits within your allocated capacity. If the 3,600 CUs are consumed within your allocated F2 capacity (2 CU/h), then there is no extra charge then it is covered by the base cost. If Data Movement operations exceed your allocated CUs, extra CUs are billed separately at pay-as-you-go rates.

Example Calculation:

• Base Cost: F2 capacity provides 2 CU/h at €0.42 per hour.
• Cost per CU: €0.21 per CU/h.
• Total CUs Consumed: 3,600 CUs
• Total Cost: 3,600 CUs * €0.21 per CU = €0.756

The additional cost would be calculated based on the pay-as-you-go rate for the excess CUs.


Thank You!

[marius1106]

Thank you, that helped a lot! 

- Is there a possibiliy to define that no extra CUs are used so that I stay at the base costs?

- And if I take a reserved capacity, are there also extra costs when my consumption doesnt fit within my allocated capacity?

[v-karpurapud · Accepted solution]

Hi @marius1106 

1. Preventing Extra CU Usage (Staying Within Base Cost)

• Microsoft Fabric does not provide a hard limit setting to automatically prevent extra CU consumption.

Utilize Fabric Capacity Metrics to monitor CU usage trends and adjust workloads accordingly. Setting up monitoring and alerts in the Azure portal can notify you when usage approaches the limit, allowing proactive management to avoid exceeding it.

To optimize CU consumption, follow these best practices:

• Reduce unnecessary data movement in Data Pipelines, Dataflows, and Spark jobs.
• Schedule workloads strategically to efficiently distribute CU consumption and prevent sudden spikes.

2. Reserved Capacity and Extra Costs

• Reserved Capacity offers a fixed CU allocation at a set price, with no additional charges if usage stays within this allocation.
• Exceeding the reserved CUs will result in extra CUs being billed at pay-as-you-go rates.
• Regularly monitor usage and optimize workloads to avoid extra costs.

For more details, please refer to the following Microsoft documentation:

Save costs with Microsoft Fabric Capacity reservations - Microsoft Cost Management | Microsoft Learn
 

By monitoring and optimizing your workloads, you can minimize extra CU usage and stay within the base cost of your reserved capacity.

Thak You!

[marius1106]

Perfect answers, you helped a lot! Thank you for your time!