Dataflow refresh limit on concurrent refreshes exceeded

[Igor_M]

Hi,

for the past few days I've been having a problem with the refreshing of dataflows on my account (I have Power BI PRO). It seems that whenever a dataflow is refreshed, any refresh of any other dataflow at the same time fails. Sometimes even the refresh of the first dataflow fails if other dataflow refreshes have been triggered. The article on Power Query Online Usage Limits - Power Query | Microsoft Learn says hat the limit of concurrent dataflow rereshes is 20 (I think it was 10 some time ago), however, mine fail when there are only two.

Also, the article says that "limits are enforced at the start of query evaluations. Once an evaluation is underway, only timeout limits are enforced.' However, mine often fail after running for 6 minutes and 6 seconds (most fail after this time regardless of how long they normally take to refresh successfully) and some fail after running for several minutes (e.g. a flow that normally takes 35 minutes to refresh may fail after 30 minutes). It seems to be unrelated to a specific workspace - the issue occurres in all my workspaces.

I don't remember making any changes to dataflows on my account around the time this problem started. I know for a fact that we haven't added any additional dataflows or changed the refresh times of any existing dataflows.

This is the error message that I am getting:

Error: Refresh limit exceeded. The up do date limit on concurrent refreshes can be found here: https://docs.microsoft.com/en-us/power-query/power-query-online-limits. Please change the dataflow refresh schedule or upgrade your workspace to Premium capacity and try again.

Is anyone having this issue? Does anyone know where to check current utilisation of concurrent dataflow runs?

[SaiTejaTalasila]

Hi @Igor_M ,

Do a parallel load on diffrent dataflows by keeping them on different workspaces.At the end you can append/merge the data.Option-reduce your input data volume and whether it will work or not.

Thanks,

Sai Teja 

[Igor_M]

Hi @SaiTejaTalasila, thanks for your input.

I have actually already tried that. The issue is that the limits shown in https://docs.microsoft.com/en-us/power-query/power-query-online-limits are per user, NOT per workspace.

For instance, according to the document a user can only have 20 entities being evaluated at any one one time across all his/her dataflows in all his/her workspaces in Power BI. This basically means that if you have, say,  a dataflow with 12 tables in workspace A and another dataflow with 10 tables in workspace B, and if you try to refresh them at roughly the same time it is very likely one of them (sometimes even both) will fail to refresh throwing the error mentioned in my original post (i.e. the one on concurrent refresh limit being exceeded). 

Therefore simply moving some dataflows to different workspaces within the same user account is not going to work in this particular scenario.

[SaiTejaTalasila]

Hi @Igor_M ,

You can try running the dataflows of any one of the workspace reports under your colleagues account.

Thanks,

Sai Teja 

[apoprio]

We are also having this problem a lot!!!
Any idea of ​​a solution???

[Igor_M]

Hi @apoprio

Unfortunately the issue has not been resolved yet. I'm still in contact with Microsoft support.

However, what I have learnt is that the limit on concurrent query evaluations set in Power Query Online Usage Limits - Power Query | Microsoft Learn does not apply to a dataflow as a whole, but to entities within the dataflows that are being refreshed.

Nonetheless, I'm still getting the same error even when fewer than 20 (dataflow) entities are being refreshed (evaluated) at the same time.

In general, I've noticed that my dataflows now take much longer to refresh than they did a few weeks ago (c. 30-40% longer).

[apo1979prio]

Indeed, we have been experiencing the same constraints in the last 2 days… It’s worth noting that we have a robust pipeline that has been functioning correctly for over 4 years… And this type of issue has occurred sporadically a few months ago… In summary, this is truly an isolated incident within Microsoft, as our pipeline typically runs smoothly without major constraints! 😓😓😫

[SaiTejaTalasila]

Hi,

As per my knowledge you can try refreshing your dataflows on different workspaces at same time.(For each individual workspace refresh only one dataflow at same time).And I hope you will definitely see change in the refresh time.

You can refer this for more details -

https://learn.microsoft.com/en-us/power-query/dataflows/best-practices-reusing-dataflows

[v-tangjie-msft]

Hi @Igor_M ,

This issue has been submitted internally. Please be patient to wait for fixing. If there is any news, I will update it here.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[SaiTejaTalasila]

Hi,

Refresh dataflows one after another using Power Automate. I hope this will help avoid refresh failures.

[Igor_M]

Hi, thank you for your reply. In my scenario, refreshing the dataflows sequentially is not an option, mainly do to timing requirements. Also, refreshes of some of the dataflows are already triggered via Power Automate (they are also affected by the same issue). 

[v-tangjie-msft]

Hi @Igor_M ,

Try below

1.  As it is not taking lot of time so implemnet incremental refresh to reduce the refresh time so you concurrent refresh count will goes down.

2. Scheduling some dataflows to other time.

3.If possible, upgrading your workspace to Premium capacity can increase the limits and improve performance.

4.Utilize the Power BI REST APIs to monitor dataflow transactions. Specifically, the Get Dataflow Transactions API can provide insights into the status of dataflow refreshes. This can help identify if certain dataflows are consistently failing and at what point.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[Igor_M]

Hi, thank you for your reply. My tests show that upgrading to premium would most likely solve the issue. However, this is something that I cannot currently do, as this would require upgrading all the users to premium. What I would like to understand (and fix) is why Power BI is claiming that I'm exceeding the concurrent refreshes limit, where clearly this isn't the case here.