Is the pricing of Fabric pipeline data copy activities incorrect?

[Isch]

Hi all, here are some tests and questions regarding the pricing of data copy activities within Fabric pipelines. 

tldr;
I'd like to know your opinion about the pricing of data copy activities. In my opinion, the consumption of pipeline data copy tasks is not billed correctly. Almost all operations are measured with the same number of CUs, independent of the duration and optimization. This leads into the conclusion that optimizing the data copy activities does not have a real impact to billing but reducing the number of elements absolutely has.

What did I test?

I did some investigations on a Fabric pipeline with a copy data task loading 12 tables from a test database to parquet. The copy data task is executed within a ForEach loop.

The target was to investigate how we can optimize the CU usage of copy data tasks.

What does Microsoft say?
According to the pricing page of fabric pipelines, the following statement is written for “Data Movement” tasks (copy data activity):
Data Movement service for Copy activity runs. You are charged based on the Capacity Units consumed during the Copy activity execution duration.

In the price breakdown on how the “Data movement” is charged, Microsoft states that the metrics are…
Based on Copy activity run duration in hours and the used intelligent optimization throughput resources

Source: Pricing for data pipelines - Microsoft Fabric | Microsoft Learn, 18.12.2024

But, what is “intelligent optimization”? According to Microsoft’s “Copy activity performance and scalability guide” there are several things to consider like the parallel copy for partitioned sources or the intelligent throughput optimization.

Source: Copy activity performance and scalability guide - Microsoft Fabric | Microsoft Learn

So I made three tests with different settings, changing the intelligent throughput optimization setting (use Max vs Auto) and comparing it against setting the batch count in the ForEach loop to 6.

The batch count has a lot of impact to the duration, but the “ITO” setting does not:

Let’s have a look at the consumed CUs in the Fabric Metrics App:

All pipelines are charged the same. If we go deeper into the detail, we can check how many CUs are used by the single activities. According to Microsoft’s pricing calculation, the duration of the operations is important for the calculation of the costs:

https://learn.microsoft.com/en-us/fabric/data-factory/pricing-pipelines

That means that the duration should have direct impact to the CU calculation and costs. If we have a look at the single operations, they all have 360 CUs, independent of the runtime:

Not what I expected.

Taking the next statement out of a blog post here, this is also what I assume for the calculation:

In my eyes:

1. 1.5 CU per hour gives 0,0004166 CU per second.
2. Say 30 s duration. 30 * 0,0004166 = 0,0125.
3. Now how many intelligent optimization throughput resources are used? Was set to auto, so unclear.
   But even assuming a maximum of 256, we only get 256 * 0,0125 =  3,2 CU (s). Far from listed 360!

Source: Solved: Minimum CU (s) billing per copy? Or am I just bad ... - Microsoft Fabric Community

Let’s have a look at a real-life scenario up and running at a customer. If we check the correlation between duration of the operation and CUs, we actually see that almost all data movement operations have 360 CUs!

Actually, 99% of the operations at the customer result in 360 CUs.

If I look at the duration at least the operations with higher CUs are also “long-running” ones:

Here we see another thing: It seems that the CUs are calculated in 360-steps (maybe this is linked to a time calculation in seconds somehow ((60*60)/10)?

Questions

• Is this calulation of consumption expected and intended (by Microsoft)?
• Am I doing the calculation right, or do I oversee something?
• What could a possible best-practice be if you have lot of tables to copy with a rather small amount of data?

Other sources

Confused by CU-seconds calculation - Microsoft Fabric Community

[lbendlin]

Since you have a Pro license you can open a Pro ticket at https://admin.powerplatform.microsoft.com/newsupportticket/powerbi

Somewhere they advise against having too many small files. But this sure looks like a bug.