Forum Discussion
Less resource consumption
- 1 year ago
Hello ebjim
agreed DF Gen2 are resource intensive.MS are doing a bit to improve this
Enabling Fast Copy shifts workloads from Power Query’s mashup engine to the scalable backend of the pipeline Copy Activity. This reduces compute resource usage by up to 80% (e.g., 78 CU with Fast Copy vs. 394 CU without for a 5B-row workload) while accelerating refresh times by 10x compared to non-Fast Copy scenarios
Microsoft acknowledges that the lack of parallel file ingestion in Dataflow Gen2 contributes to inefficiency. This is listed as an area of improvement on the Fabric roadmap, which will further reduce resource overhead
Unlike Dataflow Gen1, Gen2 doesn’t require manual historical range definitions, reducing configuration complexity and potential errors
https://www.datalineo.com/post/accelerating-dataflows-gen2-with-fast-copy
Hi ebjim,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.