Dataflow refresh limit on concurrent refreshes exceeded

[Igor_M]

Hi,

for the past few days I've been having a problem with the refreshing of dataflows on my account (I have Power BI PRO). It seems that whenever a dataflow is refreshed, any refresh of any other dataflow at the same time fails. Sometimes even the refresh of the first dataflow fails if other dataflow refreshes have been triggered. The article on Power Query Online Usage Limits - Power Query | Microsoft Learn says hat the limit of concurrent dataflow rereshes is 20 (I think it was 10 some time ago), however, mine fail when there are only two.

Also, the article says that "limits are enforced at the start of query evaluations. Once an evaluation is underway, only timeout limits are enforced.' However, mine often fail after running for 6 minutes and 6 seconds (most fail after this time regardless of how long they normally take to refresh successfully) and some fail after running for several minutes (e.g. a flow that normally takes 35 minutes to refresh may fail after 30 minutes). It seems to be unrelated to a specific workspace - the issue occurres in all my workspaces.

I don't remember making any changes to dataflows on my account around the time this problem started. I know for a fact that we haven't added any additional dataflows or changed the refresh times of any existing dataflows.

This is the error message that I am getting:

Error: Refresh limit exceeded. The up do date limit on concurrent refreshes can be found here: https://docs.microsoft.com/en-us/power-query/power-query-online-limits. Please change the dataflow refresh schedule or upgrade your workspace to Premium capacity and try again.

Is anyone having this issue? Does anyone know where to check current utilisation of concurrent dataflow runs?

[apo1979prio]

Indeed, we have been experiencing the same constraints in the last 2 days… It’s worth noting that we have a robust pipeline that has been functioning correctly for over 4 years… And this type of issue has occurred sporadically a few months ago… In summary, this is truly an isolated incident within Microsoft, as our pipeline typically runs smoothly without major constraints! 😓😓😫

[SaiTejaTalasila]

Hi,

As per my knowledge you can try refreshing your dataflows on different workspaces at same time.(For each individual workspace refresh only one dataflow at same time).And I hope you will definitely see change in the refresh time.

You can refer this for more details -

https://learn.microsoft.com/en-us/power-query/dataflows/best-practices-reusing-dataflows

[v-tangjie-msft]

Hi @Igor_M ,

This issue has been submitted internally. Please be patient to wait for fixing. If there is any news, I will update it here.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[SaiTejaTalasila]

Hi,

Refresh dataflows one after another using Power Automate. I hope this will help avoid refresh failures.

[Igor_M]

Hi, thank you for your reply. In my scenario, refreshing the dataflows sequentially is not an option, mainly do to timing requirements. Also, refreshes of some of the dataflows are already triggered via Power Automate (they are also affected by the same issue). 

[v-tangjie-msft]

Hi @Igor_M ,

Try below

1.  As it is not taking lot of time so implemnet incremental refresh to reduce the refresh time so you concurrent refresh count will goes down.

2. Scheduling some dataflows to other time.

3.If possible, upgrading your workspace to Premium capacity can increase the limits and improve performance.

4.Utilize the Power BI REST APIs to monitor dataflow transactions. Specifically, the Get Dataflow Transactions API can provide insights into the status of dataflow refreshes. This can help identify if certain dataflows are consistently failing and at what point.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[Igor_M]

Hi, thank you for your reply. My tests show that upgrading to premium would most likely solve the issue. However, this is something that I cannot currently do, as this would require upgrading all the users to premium. What I would like to understand (and fix) is why Power BI is claiming that I'm exceeding the concurrent refreshes limit, where clearly this isn't the case here.