Forum Discussion

mheggie's avatar
mheggie
Frequent Visitor
1 year ago
Solved

Increase In Copy Activity CU Usage

I have a metadata driven pipeline that I use to land on premise data into the Lakehouse.  This pipeline does so using a copy activity that runs with an Overwrite Schema operation.  I have an F2 capac...
data pipeline
  • mheggie's avatar
    mheggie
    1 year ago

    Hi Dinesh - Thanks for the suggestions.  I checked them all.  This was an issue with every copy activity - and data movement was the operation where CU increase occurred.  Our gateway is up to date.  The pipeline has not changed.

     

    Possible Good News:  This morning, the CU usage returned to normal without any intervention or changes on our part.  This makes me wonder if there was a bug.  I will continue to watch through the week to make sure the issue doesn't recur.  

mheggie's avatar
mheggie
Frequent Visitor
1 year ago

Thanks for the suggestions.  I went through your list and found the following:

- We have not paused our capacity

- We have no background rejection or throttling.  We haven't exceeded our capacity.  We just observed a significant increase in the CU usage.

 

Any other ideas?

v-dineshya's avatar
v-dineshya
Community Support
1 year ago

Hi mheggie ,

 

1.  Please monitor via Metrics App,  Use the Item History and Compute tabs to track trends and identify any new patterns.

 

2.  Batch or Consolidate Copy Activities,  If feasible reduce the number of individual copy activities by batching data or using wildcards.

 

3. In Metrics App’s Compute Page, Sort by CU usage and identify which pipeline or operation is consuming the most. Hover over the top items to see if the spike is due to data movement or refresh.


4. Confirm if your gateway was updated to version 3000.282. Review if any new features are enabled that weren’t before.

 

5. Check the pipeline is still using Append mode and not inadvertently switched to Incremental or Overwrite. Validate that no hidden transformations or schema validations were introduced.

 

I hope this information helps. Please do let us know if you have any further queries.

 

Regards,

Dinesh

 

 

  • mheggie's avatar
    mheggie
    Frequent Visitor
    1 year ago

    Hi Dinesh - Thanks for the suggestions.  I checked them all.  This was an issue with every copy activity - and data movement was the operation where CU increase occurred.  Our gateway is up to date.  The pipeline has not changed.

     

    Possible Good News:  This morning, the CU usage returned to normal without any intervention or changes on our part.  This makes me wonder if there was a bug.  I will continue to watch through the week to make sure the issue doesn't recur.