Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Did you hear? There's a new SQL AI Developer certification (DP-800). Start preparing now and be one of the first to get certified. Register now

yexu

Boost Performance with Fast Copy in Dataflows Gen2 for Snowflake

Fast Copy in Dataflows Gen2 is a game-changer to enhance the performance and cost-efficiency of your Dataflows Gen2. By leveraging the same optimized backend as the Copy Activity in data pipelines, Fast Copy significantly reduces data processing time and enhances cost efficiency.

Fast Copy in Dataflows Gen2 (Generally Available) enabled by default in all newly created Dataflows Gen2—making it the recommended choice for your production workloads.

In this blog post, you’ll discover how Fast Copy dramatically improves performance and efficiency when loading data from Snowflake into Microsoft Fabric.

We’ll walk through a real-world example using Dataflows Gen2 to load 180 million rows from Snowflake into a Lakehouse table. By comparing performance before and after enabling Fast Copy, you’ll see the substantial impact it can make.

Case 1: Dataflow Gen2 without Fast Copy

Configurations Steps to reproduce this scenario:

  1. Create a table in Snowflake with a Sales Data dataset containing approximately 180 million rows, using a schema similar to the following example.

Tip: You can use any similar dataset, such as the NYC Taxi dataset.

Boost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_SnowflakeBoost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_Snowflake

  1. Create a Dataflow Gen2 to load data from Snowflake.
  2. Disable Fast Copy in the Options settings.
  3. Set Lakehouse as the output destination
  4. Publish and refresh the Dataflow Gen2.

Performance Result

The Dataflow Gen2 refresh took approximately 42 minutes to ingest 180M rows.

Boost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_SnowflakeBoost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_Snowflake

Case 2: Dataflow Gen2 with Fast Copy

Configurations Steps to reproduce this scenario:

  1. Create a Dataflow Gen2 to load same data from Snowflake. (Fast Copy was enabled by default).
  2. Set Lakehouse as the output destination.
  3. Publish and refresh the Dataflow Gen2.

Performance Result

With Fast Copy enabled, the same scenario completes in about 5 minutes.

Boost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_SnowflakeBoost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_Snowflake

Summary

This table compares the performance of the two scenarios:

Boost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_SnowflakeBoost_Performance_with_Fast_Copy_in_Dataflows_Gen2_for_Snowflake

With Fast Copy enabled in Dataflows Gen2, data processing times are significantly reduced. In this example, loading 180 million rows from Snowflake into a Lakehouse in Microsoft Fabric resulted in an 8X improvement in performance.

Give it a try yourself and experience the performance boost firsthand!

More resources