Forum Discussion

harrybao0901's avatar
harrybao0901
Frequent Visitor
1 month ago
Solved

Fabric Copy Data activity Capacity Usage

Hi everyone,   Curerntly, I'm looking into the performance of the pipeline in Fabric and checking the CU via Capacity Metrics App. Why the DataMovement for job with duration is 44 is 1080, but for ...
data engineering
  • v-abhinavmu's avatar
    v-abhinavmu
    1 month ago

    Hi harrybao0901,

    Thanks for reaching out to the Microsoft Fabric Community forum. and thanks to sannavajjala for sharing valuable insights.

     

    According to the official Fabric documentation, Data Movement consumption for a Copy activity is based on:

    • Copy activity run duration
    • The intelligent optimization throughput resources used during the Copy activity execution

    The Copy activity also handles the copy process by connecting to the source, processing the data (such as serialization/deserialization, compression/decompression, column mapping, and data type conversions), writing the processed data to the destination, and providing monitoring with detailed logs and metrics.

     

    To compare the SQL Server → Snowflake and Files → Snowflake executions, you can review the Copy activity monitoring details, including metrics such as copyDuration, dataRead, dataWritten, throughput, usedDataIntegrationUnits, and usedParallelCopies.

     

    For more information, please refer to the below official documentation:

    Pricing for Pipelines - Microsoft Fabric | Microsoft Learn

    How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

    Monitor Copy activity - Microsoft Fabric | Microsoft Learn

     

    I hope this helps. Please feel free to reach out if you have any further questions.
    Thank you.

  • v-abhinavmu's avatar
    v-abhinavmu
    1 month ago

    Hi harrybao0901,

     

    Thanks for sharing your findings. According to the Pricing for Pipelines documentation, Copy activity capacity consumption is based on both the Copy activity run duration and the intelligent optimization throughput resources used during execution, rather than duration alone.

     

    To further analyze the differences between your runs, you can compare the Copy activity monitoring metrics such as copyDuration, throughput, usedDataIntegrationUnits, and usedParallelCopies, as these provide additional insight into the Copy activity execution.

     

    For more details, please refer to the below official documentation:

    I hope this helps. Please feel free to reach out if you have any further questions.
    Thank you.

     

harrybao0901's avatar
harrybao0901
Frequent Visitor
1 month ago

it's copy data activity. i copy around 30 tables from sql server to snowflake. the table size are not the same, some only 1K, 4K rows, some could reach 10K rows or more. but all CU are equivalent 1080.

 

And the case above (in the topic) is betwen files to snowflake and sql server to snowflake.

v-abhinavmu's avatar
v-abhinavmu
Community Support
1 month ago

Hi harrybao0901,

Thanks for reaching out to the Microsoft Fabric Community forum. and thanks to sannavajjala for sharing valuable insights.

 

According to the official Fabric documentation, Data Movement consumption for a Copy activity is based on:

  • Copy activity run duration
  • The intelligent optimization throughput resources used during the Copy activity execution

The Copy activity also handles the copy process by connecting to the source, processing the data (such as serialization/deserialization, compression/decompression, column mapping, and data type conversions), writing the processed data to the destination, and providing monitoring with detailed logs and metrics.

 

To compare the SQL Server → Snowflake and Files → Snowflake executions, you can review the Copy activity monitoring details, including metrics such as copyDuration, dataRead, dataWritten, throughput, usedDataIntegrationUnits, and usedParallelCopies.

 

For more information, please refer to the below official documentation:

Pricing for Pipelines - Microsoft Fabric | Microsoft Learn

How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

Monitor Copy activity - Microsoft Fabric | Microsoft Learn

 

I hope this helps. Please feel free to reach out if you have any further questions.
Thank you.

  • harrybao0901's avatar
    harrybao0901
    Frequent Visitor
    1 month ago

    i think it's based on Timepoint CU is more correct way to say it, as some Datamovement take 110s but only consume 1080 Total CU, but some having same 110s duration but consume 2160 total CU (I notice the Timepoint CU of these are not the same, DataMovements have higher Timepoint CU consuming more Total CU.)

    • v-abhinavmu's avatar
      v-abhinavmu
      Community Support
      1 month ago

      Hi harrybao0901,

       

      Thanks for sharing your findings. According to the Pricing for Pipelines documentation, Copy activity capacity consumption is based on both the Copy activity run duration and the intelligent optimization throughput resources used during execution, rather than duration alone.

       

      To further analyze the differences between your runs, you can compare the Copy activity monitoring metrics such as copyDuration, throughput, usedDataIntegrationUnits, and usedParallelCopies, as these provide additional insight into the Copy activity execution.

       

      For more details, please refer to the below official documentation:

      I hope this helps. Please feel free to reach out if you have any further questions.
      Thank you.

       

      • v-abhinavmu's avatar
        v-abhinavmu
        Community Support
        1 month ago

        Hi harrybao0901,

        I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. 


        Thank you.