Fast Copy in Dataflow Gen2 is now General Available! This powerful feature enables rapid and efficient ingestion of large data volumes, leveraging the same robust backend as the Copy Activity in Data pipelines.
With Fast Copy, you can experience significantly shorter data processing times and improved cost efficiency for your Dataflow Gen2. Additionally, it boosts end-to-end performance by allowing you to use Fast Copy to ingest data into staging, and then seamlessly transform it at scale using SQL DW compute.
Fast Copy: Enhance Performance and Reduce Costs in Dataflow Gen2
Fast Copy leverages the same powerful backend as the copy activity in Data pipelines, utilizing a serverless architecture that enables parallelism to maximize data movement throughout. This approach fully utilizes network bandwidth and data store IOPS for optimal performance.
As detailed in this blog, using Fast Copy to load a 6 GB CSV file into a Lakehouse table in Microsoft Fabric resulted in an 8x increase in performance and a 3x reduction in cost in Dataflow Gen2.
Unlock the Power of Fast Copy: How to Get Started
When you create a new Dataflow Gen2, Fast Copy is automatically enabled. For data sizes over 100 MB or 5 million rows, Dataflow Gen2 seamlessly switches to the Fast Copy backend, no extra steps are needed during dataflow authoring.
After running a refresh, simply check the Refresh History to see if Fast Copy was used by reviewing the entity status.
Supported data sources
Fast Copy supports numerous source connectors, including ADLS Gen2, Blob storage, Azure SQL DB, On-Premises SQL Server, Oracle, Fabric Lakehouse, Fabric Warehouse, PostgreSQL, and Snowflake.
An on-premises gateway is also supported, allowing for high-performance data transfer from on-premises sources. For details, see Fast Copy with On-premises Data Gateway Support in Dataflow Gen2.
Start optimizing your data processing with Fast Copy in Dataflow Gen2 today!
Resources:
Docs: Fast copy in Dataflows Gen2 - Microsoft Fabric | Microsoft Learn