Forum Discussion
Less resource consumption
Hello ebjim
agreed DF Gen2 are resource intensive.MS are doing a bit to improve this
Enabling Fast Copy shifts workloads from Power Query’s mashup engine to the scalable backend of the pipeline Copy Activity. This reduces compute resource usage by up to 80% (e.g., 78 CU with Fast Copy vs. 394 CU without for a 5B-row workload) while accelerating refresh times by 10x compared to non-Fast Copy scenarios
Microsoft acknowledges that the lack of parallel file ingestion in Dataflow Gen2 contributes to inefficiency. This is listed as an area of improvement on the Fabric roadmap, which will further reduce resource overhead
Unlike Dataflow Gen1, Gen2 doesn’t require manual historical range definitions, reducing configuration complexity and potential errors
https://www.datalineo.com/post/accelerating-dataflows-gen2-with-fast-copy
Hello ebjim
agreed DF Gen2 are resource intensive.
MS are doing a bit to improve this
Enabling Fast Copy shifts workloads from Power Query’s mashup engine to the scalable backend of the pipeline Copy Activity. This reduces compute resource usage by up to 80% (e.g., 78 CU with Fast Copy vs. 394 CU without for a 5B-row workload) while accelerating refresh times by 10x compared to non-Fast Copy scenarios
Microsoft acknowledges that the lack of parallel file ingestion in Dataflow Gen2 contributes to inefficiency. This is listed as an area of improvement on the Fabric roadmap, which will further reduce resource overhead
Unlike Dataflow Gen1, Gen2 doesn’t require manual historical range definitions, reducing configuration complexity and potential errors
https://www.datalineo.com/post/accelerating-dataflows-gen2-with-fast-copy