Forum Discussion
Dataflow Gen1 data refresh duration is allways more than 30 seconds
I understand, but rest assured, your queries are fine it’s just Gen1’s design.
It’s not necessarily a best practice to create many small Dataflows Gen1 if each contains only one or two very light tables, because each one will incur that same startup overhead.
Instead, you’ll get better performance by grouping related tables into a single dataflow so they refresh together within the same container session.
Hope it helps you !Best regards,Antoine
Super UserHello kadirkkkk1,
A bit weird but it may have an explication, the refresh duration in Dataflow Gen1 includes not just the query execution time, but also service overhead from the Power BI dataflow engine.
What you can do :
If you’re migrating or optimizing, consider Dataflow Gen2 in Fabric, which uses a newer backend with parallel refresh, lower overhead, and shorter startup times.
Microsoft is investing a lot in this new tool, you can easily migrate and optimize with Dataflow Gen2.
Source :
- https://learn.microsoft.com/fr-fr/fabric/data-factory/dataflows-gen2-overview
- https://learn.microsoft.com/fr-fr/fabric/data-factory/move-dataflow-gen1-to-dataflow-gen2
Hope it can help you !
Best regards,
Antoine
I have only PRO license so i can not use Dataflow Gen2
- AntoineW
Yes, you are right, you can't use Dataflow Gen2 'cause it's only available in a Fabric Capacity.
And so, you may have the refresh duration that is more than expected, due to starting server and dataflow engine, I know it's a bit frustrating but does it affect your usage? or have you encounter some issue when you execute your dataflow gen1 ?
No there is no problem in benefit of Gen1. It just confuse me like something is wrong in my queries. And it makes me avoid enabling too much table as each one takes at least 30 seconds
- AntoineW
I understand, but rest assured, your queries are fine it’s just Gen1’s design.
It’s not necessarily a best practice to create many small Dataflows Gen1 if each contains only one or two very light tables, because each one will incur that same startup overhead.
Instead, you’ll get better performance by grouping related tables into a single dataflow so they refresh together within the same container session.
Hope it helps you !Best regards,Antoine