Increase In Copy Activity CU Usage

[mheggie]

I have a metadata driven pipeline that I use to land on premise data into the Lakehouse.  This pipeline does so using a copy activity that runs with an Overwrite Schema operation.  I have an F2 capacity.

This process has been running unchanged for 3 months now with very consistent CU utlization ... until this morning the CU usage for the copy activity tasks was approximately 3 times what it has been.  In comparing to yesterday, the volumes are exactly the same.   The durations are almost exactly the same.  When I look in the Fabric Capacity Metrics app, what I've noticed is the data movement task is what has increased.   Yesterday, almost all of my processes had a CU usage of 360 each for data movement.  This morning, they have a CU usage of 1080 each.

Any idea why this might have jumped up so significantly?

[Tröttöröö]

Hi. Expecriencing the exact same issue since Saturday ~7.00am EEST. This seem to affect the minimum volume pipelines. Unbearable situation in production environments.

[mheggie]

Thanks for chiming in.  I was afraid it was just us.  Have you received any insight into what is causing this?

[frithjof_v]

Did you check the duration of the Copy Activity (Data Movement) in the Capacity Metrics App or the Data Pipeline run history?

Under normal conditions, the CU cost is 360 CU (s) per started minute.

If it runs for 1 second it will cost 360 CU (s), if it runs for 1 min 1 sec it will cost 720 CU (s) and if it runs for 2 min 1 sec it will cost 1080 CU (s).

https://www.reddit.com/r/MicrosoftFabric/s/0BRf1iuHoW

[mheggie]

Thanks for the question.  I did check all of the run times.  Almost all of the processes completed in less than a minute but the data movement charge was 1080.  For the few that completed in more than 1 minute but less than 2, the cost was 1440.

[mheggie]

This was an issue for us on Sunday and Monday.  It mysteriously fixed itself on Tuesday morning to align with exactly what you've described above.  (We didn't change anything.)

[v-dineshya]

Hi @mheggie ,

Thanks for the update. We are happy to hear that issue is resolved. Please do let us know if you have any further queries.

Regards,

Dinesh

[v-dineshya]

Hi @mheggie ,

Thank you for reaching out to the Microsoft Community Forum.

The sudden spike in CU usage for your copy activity's data movement tasks in Microsoft Fabric, despite unchanged data volumes and durations can be attributed to a few key factors.

Please check below things to fix the issue.

1. If your capacity was paused and resumed, Fabric may batch background operations like metadata processing or data movement immediately after resumption. This leads to CU spikes. Try to schedule pauses during low-usage periods and monitor post-resumption behavior.

2. Use the Health Page in Metrics App, Drill into the Compute and Storage pages to identify Background rejection rates, Throttling durations and Usage variance.

Please refer below Microsoft documents and communtity threads.

Solved: Lakehouse high CU usage, unknown source - Microsoft Fabric Community

Solved: Optimizing CU Usage in Microsoft Fabric - Microsoft Fabric Community

Solved: Re: Fabric Capacity Metrics: Interaction CU Freque... - Microsoft Fabric Community

Solved: Re: Increasing CU capacity usage running dataflows - Microsoft Fabric Community

Solved: Re: Fabric CU Usage - Microsoft Fabric Community

Solved: Discrepancy in CU Consumption Between Fabric Capac... - Microsoft Fabric Community

Understand the metrics app health page (preview) - Microsoft Fabric | Microsoft Learn

Introducing the Item History Page in Microsoft Fabric Capacity Metrics App (Preview) | Microsoft Fab...

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh

[mheggie]

Thanks for the suggestions.  I went through your list and found the following:

- We have not paused our capacity

- We have no background rejection or throttling.  We haven't exceeded our capacity.  We just observed a significant increase in the CU usage.

Any other ideas?

[v-dineshya]

Hi @mheggie ,

1.  Please monitor via Metrics App,  Use the Item History and Compute tabs to track trends and identify any new patterns.

2.  Batch or Consolidate Copy Activities,  If feasible reduce the number of individual copy activities by batching data or using wildcards.

3. In Metrics App’s Compute Page, Sort by CU usage and identify which pipeline or operation is consuming the most. Hover over the top items to see if the spike is due to data movement or refresh.

4. Confirm if your gateway was updated to version 3000.282. Review if any new features are enabled that weren’t before.

5. Check the pipeline is still using Append mode and not inadvertently switched to Incremental or Overwrite. Validate that no hidden transformations or schema validations were introduced.

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh

[mheggie]

Hi Dinesh - Thanks for the suggestions.  I checked them all.  This was an issue with every copy activity - and data movement was the operation where CU increase occurred.  Our gateway is up to date.  The pipeline has not changed.

Possible Good News:  This morning, the CU usage returned to normal without any intervention or changes on our part.  This makes me wonder if there was a bug.  I will continue to watch through the week to make sure the issue doesn't recur.