Forum Discussion

christianunr's avatar
christianunr
Helper I
1 year ago
Solved

Copy Activity is very slow

I have a data pipeline that has a Copy Activity action. The data that is being pulled from our on-prem SQL Server is tiny. The biggest table is about 1,800. However, despite the small size, Copy Acti...
data pipeline
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi christianunr,

    Thank you for reaching out in Microsoft Community Forum.

    Yes, OPTIMIZE is a SQL command designed for Delta tables in Fabric Lakehouse. It helps improve query and write performance by compacting small files into larger ones, reducing metadata overhead
    .

    You can run it in a SQL notebook or a SQL query execution environment within Fabric:
    OPTIMIZE delta.`/lakehouse_path/table_name`

    Please continue using Microsoft community forum.

    If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

    Regards,
    Pavan.

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi christianunr,

Thank you for reaching out in Microsoft Community Forum.

Please follow below steps to resolve the error;

1. Please Optimize the Delta Table
OPTIMIZE delta.`/lakehouse_path/table_name`

2. Please Run VACUUM periodically to clean up old files and Use Append instead of Upsert if updates aren’t needed.

3. Try running the pipeline during off-peak hours or check SKU limitations.

4. Please make sure the Data Pipeline is running on a suitable Fabric SKU (higher tiers have better performance).

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

 

christianunr's avatar
christianunr
Helper I
1 year ago

Hi Anonymous - for option #1, is the OPTIMIZE a SQL command?

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi christianunr,

    Thank you for reaching out in Microsoft Community Forum.

    Yes, OPTIMIZE is a SQL command designed for Delta tables in Fabric Lakehouse. It helps improve query and write performance by compacting small files into larger ones, reducing metadata overhead
    .

    You can run it in a SQL notebook or a SQL query execution environment within Fabric:
    OPTIMIZE delta.`/lakehouse_path/table_name`

    Please continue using Microsoft community forum.

    If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

    Regards,
    Pavan.