Dataflows refresh duration very inconsistent and slow

[Anonymous]

Hey all!

I wonder if anyone else experiences this, whether it is normal. I have a dataset coming in from Google Bigquery, and I run a dataflow to store the data. I've set a refresh schedule, and I find out 2 things:

1. Dataflows data refresh is really slow. What takes powerBI desktop 11 SECONDS takes dataflows 11 MINUTES. What's going on here?

2. The refresh durations on dataflows are really inconsistent. I'm talking 5 minutes, then 9 minutes, then 16 minutes, then 6 minutes, all for more or less the same data.

Does anyone here experience the same thing? Does anyone know of an explanation for this strange, slow and inconsistent behaviour?

Thanks!

Jaap 

[Anonymous]

@Anonymous, Did you find a solution to this?
I'm also experiencing extremely inconsistent and slow refreshes on my dataflows to an CDS entity.
Sometimes even up to 30 min delay or even worse some days it stops completely until i reactivate it 😰

Best regards

[Anonymous]

Unfortunately not @Anonymous , I still can't explain the difference. I've not had a client that needed to use Dataflows in a while now, so I don't know if anything has changed....

[v-eachen-msft]

Hi @Anonymous ,

You could update your gateway to latest version if your dataflow is running via the on-premise gateway.

[Anonymous]

Hey Eads,

I'm not running a gateway. The data is coming directly from google bigquery over a cloud-to-cloud connection.

Jaap

[v-eachen-msft]

Hi @Anonymous ,

If you have a Premium license, you could follow this blog to set dataflow container size:

https://blog.crossjoin.co.uk/2019/04/21/power-bi-dataflow-container-size/

If you gave a Pro license, you could try the following workarounds:

1. You could broke up a big model into entities in 5-6 different dataflows in the same workspace. Then stagger the refresh time.

2. You could reduce the queries from that biggest table.

[Anonymous]

Although your suggestions seem sound, the problem is that in powerBI desktop, this data refreshes in 11 SECONDS. So splitting up the dataset doesnt make any sense, it's already really really small, like 1000/2000 rows.

Thanks for thinking along, but I think we're looking in the wrong place. 

Jaap

[venal]

@Anonymous 

There was some issues with Data Flow.

Please refer the below link, few of other customers also facing the same issue.

https://community.powerbi.com/t5/Service/Refreshing-In-Power-BI-Service-using-Dataflow-slower-than-Import/m-p/854798#M81497

another post

https://community.powerbi.com/t5/Service/Entities-do-not-appear-in-Dataflow-when-creating-Linked-Entities/m-p/847956#M80952

If this post helps, then please consider Accept it as the solution to help the other members find it more
If this post was helpful may I ask you to mark it as solution and click on thumb symbol?

[Anonymous]

Hey Venal,

I don't think our problems are the same though. Although he is complaining about dataflow speed, his actual dataflow is just as fast as his report, but his connection between the dataflow and his dataflow-report is slow. I don't experience that.

The second link is about missing entities, and while I did run into that issue, I already fixed that by converting my dates into strings and back to dates.

Still looking for an answer!

[venal]

@Anonymous 

I have gone through the support page and found the latest comments from the Product Group Team.

Power BI users in North Europe may experience intermittent scheduled refresh issues resulting in an "established connection failed because connected host has failed to respond" error. As an immediate workaround, customers can perform a manual refresh or retry. Engineers have found the root cause of the issue and are applying mitigations to minimize the impact. The long term fix has been identified, however, due to the end of year deployment schedule, the deployment of the fix will take place in January.

Please refer the support page for known issues.

If you have any concerns, please let us know.

[Anonymous]

Hey Venal,

unfortunately that is not the issue I'm dealing with. I don't get an error message. The refreshes complete just fine. I'm only wondering why the data refresh takes longer on dataflows than in desktop.