I’m running an on-premises data gateway in a cluster setup with two servers. In the past, I occasionally had situations where the servers went offline. Around the same time, new versions of the on-premises data gateway were released, and after installing the updates the servers came back online (although the installation process itself always took quite a long time, since Microsoft’s infrastructure was barely reachable during the update).However, over the past few weeks the problem has become more frequent: sometimes one server is unreachable, sometimes the other, and sometimes even both. For example, yesterday one of the servers was offline for a few hours, and right now the other server is offline. I don’t think this is caused by the local configuration, since the servers randomly become reachable again. Also, the network ports test in the gateway always reports success for all addresses.I have three questions about this:1. Has anyone else seen similar issues?2. If the issue is tied to a specific Service Bus relay, is there any way to force the gateway to use a different relay?3. My paginated Power BI reports fail even if only one of the two servers is offline. Is there a way to configure the reports (or the data source in Power BI) to ignore the offline server?

Hi m_andel Yes, others have seen similar issues : usually after recent gateway updates or when network connectivity to the Azure Service Bus relay is unstable. You can’t manually pick a relay : the gateway automatically negotiates with the Service Bus. Best practice is to make sure outbound ports are fully open (TCP 443, 5671, 5672, 9350-9354). Failover behavior : if one server in the cluster is down, queries may still route to it unless you pause/disable that gateway node in the Power BI Service (Manage Gateways >> Cluster Settings). That way, traffic only goes to the healthy node. If the outages persist, update to the latest gateway version and monitor logs. Microsoft also recommends clustering with at least 2+ nodes for redundancy.https://learn.microsoft.com/en-in/data-integration/gateway/service-gateway-high-availability-clusters

Regard Failover behavior:I only know of these Settings for the gateway:I now disabled "Distribute requests across all active gateways in this cluster" because I understand that now the service will switch to the next server if the other is offline.I don't see any "Cluster Settings".

Hi m_andel I’ve updated my answer below with this reply of yours. There is no separate “Cluster Settings” tab in the Power BI Service UI anymore. What the docs call “cluster settings” is just the single toggle you already found:“Distribute requests across all active gateways in this cluster.” That toggle controls failover. ON = load balancing across nodes (but if one node goes down mid-query, failures can occur). OFF = traffic goes to a single node; if that node is offline, the service automatically switches to the next available node >> this ensures true failover. What you can do now Keep the toggle OFF if you want reliable failover rather than balancing. If a node is unstable, you can only prevent traffic to it by removing it from the cluster (or disabling it via the on-premises gateway app on that machine). There isn’t a separate “disable node” option exposed in the Service UI. Make sure all nodes run the same latest gateway version to avoid mismatches.

If you want to prevent traffic from ever routing to a flaky node, go into Manage Gateways >> [Cluster] >> Gateway cluster settings >> Disable or remove that node until it’s stable.>> There is no option to disable nodes. I only see what you can see on my screenshot i have provided. So no list of the cluster's nodes.

Hi m_andel , Thank you for reaching out to Microsoft Fabric Community. Thank you rohit1991 for the prompt response. There is no “disable node” switch in the Service UI - the only control there is the Distribute requests toggle.If you need to take one node out: On that server, open the On-premises data gateway app > Status > Stop service (temporary). Or use Unregister gateway in the app (permanent, can re-register later). With Distribute requests off, Power BI will just route to the remaining healthy node.

On-premised data gateway unreachable | Microsoft Fabric Community

14 Replies

rohit1991
Super User
1 year ago
Hi m_andel

Yes, others have seen similar issues : usually after recent gateway updates or when network connectivity to the Azure Service Bus relay is unstable.

You can’t manually pick a relay : the gateway automatically negotiates with the Service Bus. Best practice is to make sure outbound ports are fully open (TCP 443, 5671, 5672, 9350-9354).

Failover behavior : if one server in the cluster is down, queries may still route to it unless you pause/disable that gateway node in the Power BI Service (Manage Gateways >> Cluster Settings). That way, traffic only goes to the healthy node.

If the outages persist, update to the latest gateway version and monitor logs. Microsoft also recommends clustering with at least 2+ nodes for redundancy.
https://learn.microsoft.com/en-in/data-integration/gateway/service-gateway-high-availability-clusters
- m_andel
  Helper I
  1 year ago
  Regard Failover behavior:
  I only know of these Settings for the gateway:
  I now disabled "Distribute requests across all active gateways in this cluster" because I understand that now the service will switch to the next server if the other is offline.
  I don't see any "Cluster Settings".
  - rohit1991
    Super User
    1 year ago
    Hi m_andel
    
    I’ve updated my answer below with this reply of yours.
    
    There is no separate “Cluster Settings” tab in the Power BI Service UI anymore. What the docs call “cluster settings” is just the single toggle you already found:
    “Distribute requests across all active gateways in this cluster.”
    
    That toggle controls failover.
    
    ON = load balancing across nodes (but if one node goes down mid-query, failures can occur).
    
    OFF = traffic goes to a single node; if that node is offline, the service automatically switches to the next available node >> this ensures true failover.
    
    What you can do now
    
    Keep the toggle OFF if you want reliable failover rather than balancing.
    
    If a node is unstable, you can only prevent traffic to it by removing it from the cluster (or disabling it via the on-premises gateway app on that machine). There isn’t a separate “disable node” option exposed in the Service UI.
    
    Make sure all nodes run the same latest gateway version to avoid mismatches.
v-venuppu
Community Support
1 year ago
Hi m_andel ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you rohit1991 for the prompt response.

There is no “disable node” switch in the Service UI - the only control there is the Distribute requests toggle.
If you need to take one node out:

On that server, open the On-premises data gateway app > Status > Stop service (temporary).

Or use Unregister gateway in the app (permanent, can re-register later).

With Distribute requests off, Power BI will just route to the remaining healthy node.
v-venuppu
Community Support
1 year ago
Hi m_andel ,

I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

Thank you.
- m_andel
  Helper I
  1 year ago
  Unfortunately I still have the issue, that my gateways come and go. Right now both servers are offline. I still have no idea what I should tell my client what we can do against this.

v-venuppu

Community Support

1 year ago

Hi m_andel ,

If both gateways keep going offline, it’s usually due to network connectivity or version issues rather than Power BI itself.

Key steps to stabilize your cluster:

Update both nodes to the latest gateway version.Check outbound firewall/proxy rules – open TCP ports 443, 5671, 5672, 9350–9354 and whitelist *.servicebus.windows.net.Set failover correctly – turn Distribute requests OFF for true failover. Stop the gateway service on any flaky node until stable.

Check logs (C:\Program Files\On-premises data gateway\Gateway*.log) for disconnect patterns.

Here are few documentations for your Reference:

Update an on-premises data gateway | Microsoft Learn

On-premises data gateway - Power BI | Microsoft Learn

Thank you.

m_andel

Helper I

1 year ago

Hi v-venuppu,

thanks for helping me out here.

- on both servers the same (latest) version is installed

- i asked again my network team, which addresses and ports are available from both servers. For me this looks fine:

- distribute requests is turned off

- when I start the on-premises data gateway client and login it says "<Gateway name> is configured correctly but is unreachable due to local network connectivity problems"

- Result of the network ports test from within the gateway client: Overall Results : 21/21 tests succeeded.

- I found the following exceptions in the log file. i think it is because for go.microsoft.com there is no outbound rule in the firewall, so the exception shouldn't be related to the fact that the gateway isn't reachable:

EnterpriseGatewayConfigurator.exe Error: 0 : Failed to get build version from remote json file in URL https://go.microsoft.com/fwlink/?linkid=839413 with error System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 95.100.186.9:443
   at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.PowerBI.DataMovement.GatewayCommon.ConfiguratorClientExtensions.<GetObjectAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.GatewayCommon.SelfServiceTools.GatewayVersionCheckTool.<GetRemoteVersionAsync>d__13.MoveNext()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 95.100.186.9:443
   at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---<---
.
EnterpriseGatewayConfigurator.exe Information: 0 : Local Version 3000.282.5

- also there are some error message about export files that could not be copied because the files being used by another process (i don't know which process this might be, but again, this shouldn't be releated to my problem

- also there are a lot of these entries:

EnterpriseGatewayConfigurator.exe Information: 0 : False MSAL 4.73.0.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard

(I don't know if False is a problem here?)

- again, not sure, if these messages are problematic:

EnterpriseGatewayConfigurator.exe Information: 0 : URI from service: https://wabi-north-europe-o-primary-redirect.analysis.windows.net/
EnterpriseGatewayConfigurator.exe Information: 0 : Updating telemetry details using ClusterAssignmentRecord value
EnterpriseGatewayConfigurator.exe Error: 0 : Failed to get cluster mapping from CDN and use the local version of cluster mapping
EnterpriseGatewayConfigurator.exe Information: 0 : Service returned 1 gateway clusters for this user.
EnterpriseGatewayConfigurator.exe Information: 0 : Model gateway state: Configured
EnterpriseGatewayConfigurator.exe Information: 0 : Local service is configured, verify gateway is setup in Power BI service
EnterpriseGatewayConfigurator.exe Information: 0 : Looking for GatewayObjectId '<objectId>'
EnterpriseGatewayConfigurator.exe Information: 0 : Gateway Name '<gateway name>', Status 'Installed' (Configured)
EnterpriseGatewayConfigurator.exe Information: 0 : Gateway status check: NotReachable
EnterpriseGatewayConfigurator.exe Information: 0 : Update checks

v-venuppu
Community Support
1 year ago
Hi m_andel ,

If you are still unable to solve the issue after doing Troubleshooting,you can reach out to Microsoft Support by raising a ticket with Microsoft Support.

Please refer below link on how to raise a support ticket:

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thank you.
- m_andel
  Helper I
  1 year ago
  Hi, v-venuppu ,
  miraculously our gateways are online now.
  I will check back with my project manager if we raise a ticket.
  Thank you very much for your help.
v-venuppu
Community Support
1 year ago
Hi m_andel ,

Could you please let us know the ETA by when you will be able to check with project manager? This will help us to know when we can follow up. If you encounter any challenges during implementing the provided solution , please inform us and we’ll be glad to assist.

Thank you.
- m_andel
  Helper I
  1 year ago
  I'v already reached out to one of our global admins for opening a ticket. But we also started to change our data source for at least the most important of our reports, so no on-premises data gateway will be needed for these.
  Unfortunately I can't give you an ETA for when I get in contact with the global admin.
v-venuppu
Community Support
1 year ago
Hi m_andel ,

Thank you for the Information.Please let us know if you need any further assistance.We are happy to help.

Forum Discussion

On-premised data gateway unreachable

14 Replies

Related content

Power BI Service on-premises gateway

On-premises data gateways

On-premise date Gateway

On-premises data gateway older version

Gateway Unreachable