topic Re: Less resource consumption in Dataflow

Less resource consumption

ebjim — Thu, 06 Mar 2025 18:31:57 GMT

Is there anything that the Fabric Team is doing to make Datafflow Gen2 less resource intensive? I understand that DF gen2 may never get to the usage level of notebooks, but right now, the difference is large enough for us to consider not using them.

Re: Less resource consumption

nilendraFabric — Thu, 06 Mar 2025 20:19:49 GMT

Hello ebjim

agreed DF Gen2 are resource intensive.

MS are doing a bit to improve this

Enabling Fast Copy shifts workloads from Power Query’s mashup engine to the scalable backend of the pipeline Copy Activity. This reduces compute resource usage by up to 80% (e.g., 78 CU with Fast Copy vs. 394 CU without for a 5B-row workload) while accelerating refresh times by 10x compared to non-Fast Copy scenarios

Microsoft acknowledges that the lack of parallel file ingestion in Dataflow Gen2 contributes to inefficiency. This is listed as an area of improvement on the Fabric roadmap, which will further reduce resource overhead

Unlike Dataflow Gen1, Gen2 doesn’t require manual historical range definitions, reducing configuration complexity and potential errors

https://www.datalineo.com/post/accelerating-dataflows-gen2-with-fast-copy

Re: Less resource consumption

v-ssriganesh — Wed, 09 Apr 2025 12:13:41 GMT

Hi ebjim,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to nilendraFabric for sharing valuable insights.

Could you please confirm if your query has been resolved by the provided solution? If so, please mark it as the solution. This will help other community members solve similar problems faster.

Thank you.

Re: Less resource consumption

v-ssriganesh — Sat, 12 Apr 2025 16:55:46 GMT

Hi ebjim,

May I ask if you have resolved this issue? If so, please mark it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Less resource consumption

v-ssriganesh — Wed, 16 Apr 2025 10:42:34 GMT

Hi ebjim,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: Less resource consumption

HimanshuS-msft — Sun, 20 Apr 2025 21:21:29 GMT

ebjim : I am curious and wanted to understand a bit more on the what do you mean by "resource intensive ". Can you please elaborate ?