Power BI Gateway Issue

Hi all,

Sorry for not getting back sooner. Reason is that after a week of running agents and providing logs from servers back to MS. They came across the theory that our 8 VPU gateways do not have enough threads to support the number of requests been issued by the service and hence there is a timeout and the error. Just to note, these machines average about 4% utilization but we have followed the MS recommendation of 2 x 8VPU gateways. Anyway they are suggesting that each machine is limited by the number of threads and to remove/reduce the occurrence of the error we need to scale out .... Not easy to setup in a PRD environment. So the next best thing to do was to scale up. So for a period of a week we run 2 x 16VPU machines and noted the number of incidences occuring. Just for context we have 2 main errors cropping up in our PBI Service. 

1) {"errorCode":"Gateway_Offline","errorDescription":"EnterpriseGateway_LongMessage_Gateway_Offline"}

2){"code":"DM_GWPipeline_Gateway_AdoNetProviderOpenConnectionTimeoutError","pbi.error":

The first error is still being looked at by MS... They can't understand that if all the infrastructure is within Azure how can a High Availability setup lose the gateways... more on that one when I get more out of MS.

Here is my analysis for the period. Showing that doubling the machine size improved but did not eliminate the issue for error 2 above. 

Something doesn't seem right here.. 

FOR ::

{"code":"DM_GWPipeline_Gateway_AdoNetProviderOpenConnectionTimeoutError","pbi.error":

Here are MS suggestions to us. 

We would suggest below action plans:

1. smooth the dataset refresh by adding 2-3 seconds interval, I do know that we discussed earlier you expect the refresh every 5 mins for each dataset , but if like dataset A refreshed at 00:00:00 and dataset B refreshed at 00:00:02 then dataset C refreshed at 00:00:04. And after 5 mins, they should start at 00:05:00 and 00:05:02 and 00:05:04, thus for each dataset, they should still refresh every 5 mins. And this should reduce the sudden spike on gateway side.
2. Scale out the cluster with additional 1-2 nodes to handle the spike.
3. Add the retry parameters in your request as we mentioned in earlier call

I hope this helps someone out there. 

Will report back with updates on this.

Nick

Anonymous can you try restating gateway? It should work most of the time.

I had the same error, but in the following auto refresh, the data refreshed successfully, I had updated the data gateway, and restarted the data gateway, but it keep showing the data gateway error occasionally.

Hi  Anonymous ,

Can you provide us with the following information so we can better assist you:

1. Are you using a personal gateway or an on-premises data gateway?

2. What version of Power BI Gateway are you currently using?

3. Have you made any recent changes to your data source or gateway configuration?

You can check the following:

1. Is your gateway up to date, if not, you can download the latest version of the gateway from the link below:

Update an on-premises data gateway | Microsoft Learn

2. Check network connectivity: Ensure that there are no network issues or firewall restrictions preventing the gateway from connecting to the data source.

3. Verify data source credentials: Make sure that the credentials used to connect to the data source are correct and have not expired. You can update the credentials in the gateway settings.

You can also check common gateway related errors and solutions in the link below:

Troubleshoot gateways - Power BI | Microsoft Learn

This is the related document, you can view this content:

Solved: English translation of this error: DM_GWPipeline_G... - Microsoft Fabric Community

Solved: Gateway Timeout Error - Microsoft Fabric Community

Solved: DM_GWPipeline_Gateway_TimeoutError - Microsoft Fabric Community

Best Regards,

Liu Yang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

I had the same error and It happens very frequently, but in the following auto refresh, the data refreshed successfully. I had updated the on-premises data gateway and I restarted the data gateway, but it keeps happening. Please Help us.

Same situation here. We tried restarting the gateway, but still getting this error. 

 we are having the same issue as of this morning 01-Dec-23.  Already tried restarting gateway but with no luck... Anything that has changed on the power BI service side that we should cater for

Hi all,

We are also seeing this same issue appearing intermittently wihtout any root cause. We are running Power BI service with High Availability Gateways all within Azure. We are on premium capacity.

We execute 6 dataset refreshes every 5 minutes via an API call. We get this error many times a day. I'm raising a support ticket to Microsoft to get further investigation as to how this can happen ? We are questioning the reliability of the Power BI service to provide the level of service we need at such a high cadence of refreshes.

DM_GWPipeline_Gateway_AdoNetProviderOpenConnectionTimeoutError

All our gateways are upto date our data sources some of which are Cloud Based are always there. 

No root cause / no reason for these failures.

Please let me know if anyone comes across a root cause for these.

We have been experiencing the same issue, DM_GWPipeline_Gateway_AdoNetProviderOpenConnectionTimeoutError. We are running a cluster gateway with 3 computers running on the on premise gateway. Issue started on 1-16 out of the blue. It appears when multiple datasets are using the same dataflows but this is occurring on excel files and SharePoint. Reached out to Microsoft to create a ticket... they were not helpful. They just said there is a connection problem. Any advice on how to correct this issue would be helpful.

This is also an intermittent issue for refreshes... cannot determine the exact cause of why sometimes it works and sometimes it doesn't. This is effecting multiple dataflow/dataset refreshes not just one. So, can only assume there is not an issue contained within one model.

I have the same issue since 28th January 2024: DM_GWPipeline_Gateway_AdoNetProviderOpenConnectionTimeoutError

We have one cluster with 2 gateways. I upgraded and restarted both of them.

I also tried to make an smaller dataset (with less and simple queries), but the error still appears in most of tests and it seems not to explain the reason of this error.

Has anyone find the cause of this error?

Thanks