Forum Discussion
New OP data gateway unreachable
Our data gateway has been occasionally failing scheduled refreshes with 'gateway unreachable' errors which I'm fairly confident are because the machine was taxed by other data processing tasks, so to remedy I've added a second data gateway to the same cluster and set the cluster to distribute queries. Unfortunately I'm still seeing the 'gateway unreachable' errors and I am not sure if the secondary gateway is participating in the cluster. However now the pbi service gateway page now reports this error:
Not all gateway instances within this cluster are connected.
Please find more details below about specific errors for each gateway instance.
Activity ID: d5e98a0b-4ddb-42f2-9321-824569cde7a3
Request ID: 7582bc4f-04fa-74f1-fd35-3605dd6e21f6
Cluster URI: https://df-msit-scus-redirect.analysis.windows.net
Status code:
Time: Wed Nov 21 2018 14:27:28 GMT-0800 (Pacific Standard Time)
Version: 13.0.7529.129
ossku-azure-vm2 data gateway: Gateway on endpoint 'sb://df-msit-scus-relay3.servicebus.windows.net/9d5d25de-xxxx-xxxx-xxxx-1012e92e2735/' is unreachable.
There is no indication which gateways are working or failing to connect. Both gateways are running, status says online and ready, version 3000.0.155.1 Nov 2018. Network port tests succeed on the first gateway but fail (Incomplete failed to connect to network) on the new second gateway.
13 Replies
- burny
Microsoft Employee
Oops, clarifying.. I mean there is no indication on the new on prem data gateway software that it's handling queries or not. There is the port test failure and error message indicating its the new cluster member. Both gateways are on the same network and are virtually identical machines.
Any help troubleshooting is appreciated!
- AnonymousNot applicable
Hi burny,
I try to trace your request id and found it seems like a common gateway connection issue:
The endpoint was not found. There are no listeners connected for the endpoint. Please ensure that you can connect to the internet using HTTP port 80 and TCP port 9350.
I'd like to suggest you to Please check power bi gateway service is keep running on your device and firewall white list to confirm it hasn't blocked ports and httprequest.
Regards,
Xiaoxin Sheng
- burny
Microsoft Employee
This machine has no problem reaching the internet. For good measure I've added allow rules for all traffic on port 80 and 9350 and the windows service exe itselft but there is no change in behavior:
1. Data gateway application says online but fails its port test
2. Service reports the same error
3. Another identicaly machine on the same network has no problem
Are there troubleshooting steps or logs that could help? Is there a manual analog to the port test?
- burny
Microsoft Employee
The service appears to be running.
I did notice that my first data gateway machine reports the service starting and stopping to it's own eventlog (On-premises data gateway service.evtx) whereas the new one does not. I dumped logs from the gateway application and found these errors:
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:11:09.9732940Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 806193F3 [DM.EnterpriseGateway] Error getting or creating Windows event log: The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security.
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:12:40.1324321Z DM.EnterpriseGateway 2bf976d1-a83d-410a-9eb8-33318a12dfe1 00000000-0000-0000-0000-000000000000 MFUC 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)Reinstalling and recovering the gateway did not resolve or change the behavior. Is there a clean install process? How to ensure counters and application logs are installed properly and accessible to the application?
I also found these errors in the Configurator log:
=============================
EnterpriseGatewayConfigurator.exe Information: 0 : Initialize EnterpriseGatewayMainViewModel
EnterpriseGatewayConfigurator.exe Information: 0 : GetConfiguratorStateAsync
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SOFTWARE\Microsoft\.NETFramework\v4.0.30319 is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319 is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Exception thrown during GetConfiguration: 'Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClientException: Error retrieving gateway configuration. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://localhost/powerbi/gatewayconfiguration/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.IO.PipeException: The pipe endpoint 'net.pipe://localhost/powerbi/gatewayconfiguration/service' could not be found on your local machine.
--- End of inner exception stack trace ---Server stack trace:
at System.ServiceModel.Channels.PipeConnectionInitiator.GetPipeName(Uri uri, IPipeTransportFactorySettings transportFactorySettings)
at System.ServiceModel.Channels.NamedPipeConnectionPoolRegistry.NamedPipeConnectionPool.GetPoolKey(EndpointAddress address, Uri via)
at System.ServiceModel.Channels.CommunicationPool`2.TakeConnection(EndpointAddress address, Uri via, TimeSpan timeout, TKey& key)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.PowerBI.DataMovement.Pipeline.InternalContracts.IGatewayConfigurationService.GetConfiguration()
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClient.GetConfiguration()
--- End of inner exception stack trace ---
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClient.GetConfiguration()
at Microsoft.PowerBI.DataMovement.GatewayUiCommon.ConfiguratorInitializationContextFactory.<GetConfiguratorStateAsync>d__2.MoveNext()'
EnterpriseGatewayConfigurator.exe Information: 0 : Exception thrown during GetConfiguration: 'Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClientException: Error retrieving gateway configuration. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://localhost/powerbi/gatewayconfiguration/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.IO.PipeException: The pipe endpoint 'net.pipe://localhost/powerbi/gatewayconfiguration/service' could not be found on your local machine.
--- End of inner exception stack trace ---=============================
EnterpriseGatewayConfigurator.exe Information: 0 : Local Version 3000.0.155.1
EnterpriseGatewayConfigurator.exe Information: 0 : Create web request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Information: 0 : Create httpClient request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '1f362f8e-d630-4451-9f51-7431f70e0696', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:12:48 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '85c2fed6-4a0c-47f9-a55f-c633e6dc5301', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:14:07 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()