Power BI dataflow failing due to refresh limit exceeded

[kalaclair]

Since 11/24/2024 a Gen1 dataflow refresh began failing regularly and giving an error message of:

Error: Refresh limit exceeded. The up to date limit on concurrent refreshes can be found here: https://docs.microsoft.com/en-us/power-query/power-query-online-limits. Please change the dataflow refresh schedule or upgrade your workspace to Premium capacity and try again. Request ID: 59ef2e5a-64c9-4216-a96a-9e9a471c9aa4 Activity ID: 246202e9-d990-4807-97aa-b4adbfd15345

We have Power BI Pro and are concurrently refreshing 2 dataflows when we get the refresh failures.  One dataflow consists of 6 tables while the other has 12.  According to the limits on the website provided in the error message, we should be able to have 20 concurrent tables in dataflows refreshing at the same time.

We did not make any changes to the dataflows around the time the refreshes started failing and the refreshes worked fine for months prior to 11/24/2024.  The refresh doesn't fail every time these dataflows run concurrently.  The failures tend to happen 1-2 times a day while the dataflows run concurrently 4 times a day.

I am not able to change the schedule of the dataflows to not run concurrently due to other dataflows taking up the other refresh time slots.

I am currently manually refreshing the dataflow whenever it fails, so any assistance to mitigate this issue would be greatly appreciated!

[Anonymous]

Hi @kalaclair,

Thank you for reaching out in Microsoft Community Forum.

please follow below to acheive the error;

1. Ensure no more than 20 tables are refreshing at the same time. Optimise queries, split large tables if needed, and check refresh history for overlaps.
2. Use Power Automate or Power BI REST APIs to trigger dataflows one after the other, even if their schedules are close.

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.

Regards,
Pavan.

[kalaclair]

Thank you for the feedback and suggestions.  I have verified that less than 20 tables are refreshing at the same time when I get this error message.  Are you saying it is possible to get this error message even if less than 20 tables are refreshing at the same time, possibly due to the size of the tables?

[Anonymous · Accepted solution]

Hi @kalaclair,

Thank you for reaching out in Microsoft Community Forum.

Yes, the error can still occur even with less than 20 tables refreshing, especially if the tables are large. Large tables or complex queries can take more time to refresh, potentially causing delays and hitting the refresh limit.

These are possible approaches to solve your error;

1. Simplify queries to reduce the data being pulled and consider partitioning large tables into smaller segments to improve refresh efficiency.

2. Check which tables are taking longer to refresh and adjust the refresh schedules to avoid overlaps and delays.

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.

Regards,
Pavan.

[GilbertQ]

Hi @kalaclair 

It appears that you said there are only 2 concurrent dataflows running and then you said you have other dataflows running also. What I would recommend doing is to see if you can optimize the steps in your dataflow to use less resources?

Also if you could schedule the dataflows to run one after the other instead?

[kalaclair]

Changing the dataflows to run one after the other will not help because all other time slots to run dataflow refreshes are taken by other dataflows.  It will simply end up scheduled at the same time as another dataflow.  For example, the dataflow that is failing with 6 tables is scheduled to run at 9 a.m., along with another dataflow with 12 tables that also runs at 9 a.m.  The next slot to run a refresh is 9:30 a.m., but that is taken by a different dataflow.  The same is true for 10 a.m., etc.  This is why the 2 dataflows are scheduled concurrently, we have no other open time slots.  All of our dataflows complete in less than 15 minutes, so they are not running into the time frame of the next scheduled dataflow.  Any other suggestions?  Am I incorrect in thinking we can have 20 tables refreshing in dataflows at the same time?