Dataflow Refresh Duration

[KerySolutions]

Hello,

I am looking at the dataflow logging and it seems inconsistent. This refresh took 5 hours (!!) but if you add up the duration column in the log download, the duration is a little over an hour. How can I understand what is holding it up here? Why is the duration so far away from reality? What is happening in the remaining 4 hours?

Thanks

MCK

[v-luwang-msft · Accepted solution]

Hi @KerySolutions ，

The reason for refreshing taking too much time：

Too many datasets to refresh at the same time occupy CPU and memory, Insufficient gateway or underlying data source performance (busy on network / refresh), To refresh the model to be loaded into memory, it takes up 2 times the size of the data set, and you need to wait for memory resources.

 If a refresh is running a long time, you can select the dataflow options (the ellipses next to the dataflow) and then select Cancel refresh. Or scheduled the refresh of dataset at different time.

In addition, I found a same issue ,please check if it could help you a little.

https://community.powerbi.com/t5/Service/Dataflow-vs-Dataset-refresh/td-p/572076

https://medium.datadriveninvestor.com/my-dataset-is-taking-too-long-to-refresh-843fd8fd4a51

https://radacad.com/how-to-use-dataflow-to-make-the-refresh-of-power-bi-solution-faster

Best Regards,

Lucien

[Fowmy]

@KerySolutions 

@Phil_Seamark published an article/technique on this :https://dax.tips/2021/02/15/visualise-your-power-bi-refresh/

This is a good place to start with.

[KerySolutions]

I will provide more information... the dataflow is not accessing an SQL server, it is accessing an ODBC source. The article you posted is specific to dataset refresh against SQL.

I am more curious about getting feedback from Microsoft to better understand how to read the dataflow log from the service. The durations only add up to around 80 minutes but the total refresh time was over 5 hours... if I could know which entity is taking so long, I can take action.