Forum Discussion
Is dataflowgen2 better in performance and refresh than dataflow gen 1?
Dear All,
Have a basic query. I am using Dataflow gen 1 (legacy) and fabric is not enabled to ingest data from sharepoint library to dataflow. It has huge data to refresh which takes more than 2 hours and stops.
I have these below questions:
1. Will there be any improvement in performance and refresh time if I switch to ms fabric dataflow gen 2?
2. Will there be any improvement if I move from pro shared workspace to premium workspace (i.e F64/P1)?
What will be the alternate option to solve this issue of huge dataflow refresh timing if I cannot further breakdown the data into chunks for dataflow. There is no complex transformation involved, it simple loading from source (sharepoint library to dataflow)
Thanks,
Pallavi
Hi pallavi_r ,
Thank you andrewsommer for the helpful response!1.Yes, switching to Dataflow Gen 2 may reduce refresh times compared to Gen 1, depending on data size, network, and source performance. To get the best results, ensure query folding is maintained and avoid heavy transformations in the query steps.
2.Moving to Premium Capacity adds longer refresh windows and provides more stable performance. The main benefit is avoiding refresh failures and handling larger datasets more reliably.
If you cannot break the data further, also consider Incremental Refresh in Gen 2 if your SharePoint data has a timestamp. This might reduce ongoing refresh times by 50–80%.
Please refer the attached documents for more understanding:
https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh
https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-understand-optimize-refresh
https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-overview
I hope this helps.If so,give us kudos and consoder accepting it as solution.
Regards,
Pallavi G.
3 Replies
Yes, you should see performance improvement after switching to a Dataflow Gen 1; the trade off is they consume more CUs. A dataflow Gen2 will most likely reduce refresh times.
Yes, moving to a premium capacity will also improve your refresh times and improve stability. The max refresh time for shared (pro) capacity is 2 hours; however it is 5 hours with a premium (P/F SKU).
If your SharePoint source includes a timestamp you could investigate incremental refresh to reduce refresh time.
Please mark this post as a solution if it helps you. Appreciate Kudos.
Hi andrewsommer ,
Thanks for reply. Did you mean moving to dataflow gen 2 will reduce the refresh time to some extent? For example if in dataflow gen 1, it takes 1 hour, then gen 2 refresh time will be less than 1 hour. This is something I want to know. And the same question is for moving to premium capacity. Will it significantly reduce the refresh time?
Thanks,
Pallavi
Hi pallavi_r ,
Thank you andrewsommer for the helpful response!1.Yes, switching to Dataflow Gen 2 may reduce refresh times compared to Gen 1, depending on data size, network, and source performance. To get the best results, ensure query folding is maintained and avoid heavy transformations in the query steps.
2.Moving to Premium Capacity adds longer refresh windows and provides more stable performance. The main benefit is avoiding refresh failures and handling larger datasets more reliably.
If you cannot break the data further, also consider Incremental Refresh in Gen 2 if your SharePoint data has a timestamp. This might reduce ongoing refresh times by 50–80%.
Please refer the attached documents for more understanding:
https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh
https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-understand-optimize-refresh
https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-overview
I hope this helps.If so,give us kudos and consoder accepting it as solution.
Regards,
Pallavi G.
