Less resource consumption

[ebjim]

Is there anything that the Fabric Team is doing to make Datafflow Gen2 less resource intensive? I understand that DF gen2 may never get to the usage level of notebooks, but right now, the difference is large enough for us to consider not using them.

[HimanshuS-msft]

@ebjim : I am curious and wanted to understand a bit more on the what do you mean by "resource intensive ". Can you please elaborate ?

[v-ssriganesh]

Hi @ebjim,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

[v-ssriganesh]

Hi @ebjim,

May I ask if you have resolved this issue? If so, please mark it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-ssriganesh]

Hi @ebjim,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to @nilendraFabric for sharing valuable insights.

Could you please confirm if your query has been resolved by the provided solution? If so, please mark it as the solution. This will help other community members solve similar problems faster.

Thank you.

[nilendraFabric · Accepted solution]

Hello @ebjim 

agreed DF Gen2 are resource intensive. 

MS are doing a bit to improve this

Enabling Fast Copy shifts workloads from Power Query’s mashup engine to the scalable backend of the pipeline Copy Activity. This reduces compute resource usage by up to 80% (e.g., 78 CU with Fast Copy vs. 394 CU without for a 5B-row workload) while accelerating refresh times by 10x compared to non-Fast Copy scenarios

Microsoft acknowledges that the lack of parallel file ingestion in Dataflow Gen2 contributes to inefficiency. This is listed as an area of improvement on the Fabric roadmap, which will further reduce resource overhead

Unlike Dataflow Gen1, Gen2 doesn’t require manual historical range definitions, reducing configuration complexity and potential errors

https://www.datalineo.com/post/accelerating-dataflows-gen2-with-fast-copy