Dataflow Gen1 data refresh duration is allways more than 30 seconds

[kadirkkkk1]

In dataflow gen1 when i look at refresh history i see each of my table takes at least 30 seconds of update time. It doesn't matter how light is the query is, it always takes tthat much time. If i have 5 very simple tables it takes 5 * 30 seconds = 2.5 minutes

If query takes 35 seconds in power query editor than it rounds up to 1 min in refresh history.

Isn't it weird, What's the reason behind this behaviour?

[v-echaithra]

Hi @kadirkkkk1 ,

We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.

Best Regards,
Chaithra E.

[kadirkkkk1]

I checked the message below as to be the solution

[v-echaithra]

Hi @kadirkkkk1 ,

Thank you @AntoineW  for your inputs.

I hope the information provided is helpful. I wanted to check whether you were able to resolve the issue with the provided solutions. Please let us know if you need any further assistance.

Thank you.

[AntoineW]

Hello @kadirkkkk1,

A bit weird but it may have an explication, the refresh duration in Dataflow Gen1 includes not just the query execution time, but also service overhead from the Power BI dataflow engine.

What you can do : 

• If you’re migrating or optimizing, consider Dataflow Gen2 in Fabric, which uses a newer backend with parallel refresh, lower overhead, and shorter startup times. 

• Microsoft is investing a lot in this new tool, you can easily migrate and optimize with Dataflow Gen2.

Source : 

- https://learn.microsoft.com/fr-fr/fabric/data-factory/dataflows-gen2-overview

- https://learn.microsoft.com/fr-fr/fabric/data-factory/move-dataflow-gen1-to-dataflow-gen2

Hope it can help you ! 

Best regards,

Antoine

[kadirkkkk1]

I have only PRO license so i can not use Dataflow Gen2 

[AntoineW]

Yes, you are right, you can't use Dataflow Gen2 'cause it's only available in a Fabric Capacity.

And so, you may have the refresh duration that is more than expected, due to starting server and dataflow engine, I know it's a bit frustrating but does it affect your usage? or have you encounter some issue when you execute your dataflow gen1 ? 

[kadirkkkk1]

No there is no problem in benefit of Gen1. It just confuse me like something is wrong in my queries. And it makes me avoid enabling too much table as each one takes at least 30 seconds

[AntoineW · Accepted solution]

I understand, but rest assured, your queries are fine it’s just Gen1’s design.

It’s not necessarily a best practice to create many small Dataflows Gen1 if each contains only one or two very light tables, because each one will incur that same startup overhead.

Instead, you’ll get better performance by grouping related tables into a single dataflow so they refresh together within the same container session.

Hope it helps you ! 

Best regards,

Antoine