On-Premise Gateway goes offline and won't restart until credentials are re-entered

[justinhca]

Hi there.  We are encountering an intemittent problem with our On-Premise Data Gateway, where the service fails and will not restart unless the credentials are re-entered.  The service is currently setup to restart once after failure, but then gets stuck in the "Starting" state.

Here's a few stats on our setup:

• Gateway version number: 14.16.6480.1
• Using HTTPS mode on the gateway
• Accessing data from a native SQL database
• Using a non-admin domain account as the Power BI service account
• Gateway installed using an admin account, registered with a work email
• Gateway installed on a dedicated SQL Server (not a domain controller)
• Running Windows Server 2012 R2

Here's the suspect error being thrown first:

[DM.Pipeline.Common] Failed to parse gateway core config file: Object reference not set to an instance of an object.

Here's the full log:

Starting trace on 11/17/2017 6:44:00 PM UTC
Version: 13.0.2682.0


DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:00.4257009Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    EAD7B167 [DM.EnterpriseGateway] Starting On-premises data gateway service
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:02.6078269Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    D1BCCC47 [DM.GatewayCore] Setting thread pool worker threads = 300, min completion port threads = 300
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:02.6268277Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    F1E4EEC3 [DataMovement.PipeLine.MashupCommon] Mashup extension file Impala exists. Start loading.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:04.9359570Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    C2DDEB76 [DataMovement.PipeLine.MashupCommon] Loading mashup extension file Impala succeeded.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:04.9359570Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    0A02C197 [DataMovement.PipeLine.MashupCommon] Mashup extension file Snowflake exists. Start loading.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.0099618Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    3B3113B6 [DataMovement.PipeLine.MashupCommon] Loading mashup extension file Snowflake succeeded.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.0099618Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    E9CC64C1 [DataMovement.PipeLine.MashupCommon] Mashup extension file R exists. Start loading.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.0229583Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    A6098318 [DataMovement.PipeLine.MashupCommon] Loading mashup extension file R succeeded.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.0229583Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    3B1E0DB8 [DataMovement.PipeLine.MashupCommon] Mashup extension file Netezza exists. Start loading.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.0660049Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    0BDBD689 [DataMovement.PipeLine.MashupCommon] Loading mashup extension file Netezza succeeded.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.0670039Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    47D7EC8F [DM.GatewayCore] Starting direct query pool.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:05.1139827Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    B1482368 [DM.EnterpriseGateway] Service OnStart
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:08.9881903Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    06EC9A10 [DM.GatewayCore] New runner was explicitly started on the 'Configuration service' controller
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:08.9951911Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    5FBD3146 [DM.GatewayCore] Attempting to create and start service host for 'Configuration service'
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:10.0652518Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    79BDE466 [DM.GatewayCore] Configuration service started
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:10.0792500Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    8F619B72 [DM.GatewayCore] Successfully created and started service host for 'Configuration service'
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:10.1912586Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    1C9AA209 [DM.EnterpriseGateway] Started the 'On-premises data gateway service'.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:10.2502621Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    CAB8F9C3 [DM.GatewayCore] Attempting to create and start service host for 'Transfer service'
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:11.1753158Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    45ADCDD3 [DM.GatewayCore] Using configured 'Https' for ServiceBus ConnectivityMode
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:11.3573231Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    2E02FDF0 [DM.GatewayCore] Creating GatewayTransferServiceHost against 'sb://wabi-west-us-relay10.servicebus.windows.net/c3400dcc-ccb6-43e4-a2aa-e0b30f2c3a68/' with following binding configuration: {"ReliableSession":{"Enabled":false,"Ordered":true,"InactivityTimeout":"00:10:00"},"HostNameComparisonMode":0,"ConnectionMode":0,"TransferMode":0,"MaxBufferPoolSize":8500000,"MaxBufferSize":8500000,"MaxConnections":1500,"ListenBacklog":10,"IsDynamic":false,"MaxReceivedMessageSize":8500000,"ReaderQuotas":{"MaxStringContentLength":8192,"MaxArrayLength":16384,"MaxBytesPerRead":4096,"MaxDepth":32,"MaxNameTableCharCount":16384,"ModifiedQuotas":0},"Scheme":"sb","EnvelopeVersion":{"NextDestinationActorValue":"http://www.w3.org/2003/05/soap-envelope/role/next"},"Security":{"Mode":1,"RelayClientAuthenticationT..."}
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:13.1914375Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    0756374B [DM.GatewayCore] ServiceEndpoint Connection status: Online, LastError: ''
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:13.2264322Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    500124B1 [DM.GatewayCore] Transfer service started
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:13.2284333Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    30DBA016 [DM.GatewayCore]   <ii>sb://wabi-west-us-relay10.servicebus.windows.net/c3400dcc-ccb6-43e4-a2aa-e0b30f2c3a68/</ii>
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:44:13.3014395Z DM.EnterpriseGateway    00000000-0000-0000-0000-000000000000    00000000-0000-0000-0000-000000000000    0000    00000000-0000-0000-0000-000000000000    0E6D6CC9 [DM.GatewayCore] Successfully created and started service host for 'Transfer service'
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:24.3875085Z DM.EnterpriseGateway    377e7ba2-ee30-4bf7-a95b-a3c17d884a6d    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGPP    6184590f-39fc-4668-8369-238104d08cc5    A08BF3B6 [DM.GatewayCore] Deserialized TestGatewayConnectionRequest, executing
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:26.8566521Z DM.EnterpriseGateway    a2e86903-aba6-44a1-9ab8-055eda4b5a81    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGPP    6184590f-39fc-4668-8369-238104d08cc5    E5E3865A [DM.GatewayCore] Deserialized OpenConnectionRequest, executing
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:26.8566521Z DM.EnterpriseGateway    8c1df6d6-1b8a-40a3-a256-b32933d5a973    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGPP    6184590f-39fc-4668-8369-238104d08cc5    E5E3865A [DM.GatewayCore] Deserialized OpenConnectionRequest, executing
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:26.8566521Z DM.EnterpriseGateway    0e624356-be43-453c-8dd6-7a46296c1799    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGPP    6184590f-39fc-4668-8369-238104d08cc5    E5E3865A [DM.GatewayCore] Deserialized OpenConnectionRequest, executing
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:26.8576499Z DM.EnterpriseGateway    59212c0e-5f2d-439d-be69-17e36badb50e    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGPP    6184590f-39fc-4668-8369-238104d08cc5    E5E3865A [DM.GatewayCore] Deserialized OpenConnectionRequest, executing
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:26.8566521Z DM.EnterpriseGateway    13d10ad4-4148-42cd-8455-c1cc0403fe06    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGPP    6184590f-39fc-4668-8369-238104d08cc5    E5E3865A [DM.GatewayCore] Deserialized OpenConnectionRequest, executing
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:27.9487139Z DM.EnterpriseGateway    8141dc3b-b638-4b26-83d7-08691a8bcb63    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGGC    6184590f-39fc-4668-8369-238104d08cc5    77D2BB92 [DataMovement.PipeLine.MashupCommon] SessionManager: CleanupSessions Starting
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:27.9507142Z DM.EnterpriseGateway    8141dc3b-b638-4b26-83d7-08691a8bcb63    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGGC    6184590f-39fc-4668-8369-238104d08cc5    5C708278 [DataMovement.PipeLine.MashupCommon] SessionManager: CleanupSessions found 0 sessions to cleanup
DM.EnterpriseGateway Warning: 0 : 2017-11-17T18:45:28.4617436Z DM.EnterpriseGateway    8141dc3b-b638-4b26-83d7-08691a8bcb63    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGGC    6184590f-39fc-4668-8369-238104d08cc5    9315B129 [DM.Pipeline.Common] Failed to parse gateway core config file: Object reference not set to an instance of an object.
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:28.5557484Z DM.EnterpriseGateway    0c99237e-f046-45d3-9390-08c82257bccb    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    B9A4804B [DM.GatewayCore] Db pool opening raw database connection to [Microsoft.Mashup.OleDb.1: <pi>Microsoft.Mashup.OleDb.1:TRUNCATED:[Windows] Encrypted Credential information omitted</pi>]
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:28.5557484Z DM.EnterpriseGateway    6cdde815-68ae-498d-85ea-588c81df27b2    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    B9A4804B [DM.GatewayCore] Db pool opening raw database connection to [Microsoft.Mashup.OleDb.1: <pi>Microsoft.Mashup.OleDb.1:TRUNCATED:[Windows] Encrypted Credential information omitted</pi>]
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:28.5557484Z DM.EnterpriseGateway    08170f33-88e4-4202-8ddd-36510f19e32a    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    B9A4804B [DM.GatewayCore] Db pool opening raw database connection to [Microsoft.Mashup.OleDb.1: <pi>Microsoft.Mashup.OleDb.1:TRUNCATED:[Windows] Encrypted Credential information omitted</pi>]
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:28.5637489Z DM.EnterpriseGateway    20aa9f3f-49b4-4aea-9f8a-9079dc21ffe1    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    B9A4804B [DM.GatewayCore] Db pool opening raw database connection to [Microsoft.Mashup.OleDb.1: <pi>Microsoft.Mashup.OleDb.1:TRUNCATED:[Windows] Encrypted Credential information omitted</pi>]
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:28.5677424Z DM.EnterpriseGateway    bbd9f7a2-58f6-4815-ad0d-f6b1ed5bc63e    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    B9A4804B [DM.GatewayCore] Db pool opening raw database connection to [Microsoft.Mashup.OleDb.1: <pi>Microsoft.Mashup.OleDb.1:TRUNCATED:[Windows] Encrypted Credential information omitted</pi>]
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:32.6999794Z DM.EnterpriseGateway    0c99237e-f046-45d3-9390-08c82257bccb    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    1425F0E7 [DataMovement.PipeLine.GatewayDataAccess] EncryptedConnection state = False for datasource {"kind":"SQL","path":"OurServerName;OurDatabase"}
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:32.6999794Z DM.EnterpriseGateway    20aa9f3f-49b4-4aea-9f8a-9079dc21ffe1    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    1425F0E7 [DataMovement.PipeLine.GatewayDataAccess] EncryptedConnection state = False for datasource {"kind":"SQL","path":"OurServerName;OurDatabase"}
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:32.6999794Z DM.EnterpriseGateway    6cdde815-68ae-498d-85ea-588c81df27b2    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    1425F0E7 [DataMovement.PipeLine.GatewayDataAccess] EncryptedConnection state = False for datasource {"kind":"SQL","path":"OurServerName;OurDatabase"}
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:32.7009800Z DM.EnterpriseGateway    08170f33-88e4-4202-8ddd-36510f19e32a    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    1425F0E7 [DataMovement.PipeLine.GatewayDataAccess] EncryptedConnection state = False for datasource {"kind":"SQL","path":"OurServerName;OurDatabase"}
DM.EnterpriseGateway Information: 0 : 2017-11-17T18:45:32.7019800Z DM.EnterpriseGateway    bbd9f7a2-58f6-4815-ad0d-f6b1ed5bc63e    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    1425F0E7 [DataMovement.PipeLine.GatewayDataAccess] EncryptedConnection state = False for datasource {"kind":"SQL","path":"OurServerName;OurDatabase"}
DM.EnterpriseGateway Warning: 0 : 2017-11-17T18:45:32.7349809Z DM.EnterpriseGateway    20aa9f3f-49b4-4aea-9f8a-9079dc21ffe1    823164e6-6ba2-8f2c-7e8b-c93ffd6b5fc0    MGCC    6184590f-39fc-4668-8369-238104d08cc5    A201C83F [DM.MashupExtensionsDiscovery] Mashup extension kind(SQL) is not loaded

[v-jiascu-msft]

Hi @justinhca,

I tried but can't reproduce. What I would suggest:

1. If the Power BI Service user has enough permissions. It seems the user can't access the config file.

2. If there is something wrong with the Server that it can't save credentials. 

3. If you still have problem, please create a support ticket here.

Best Regards!

Dale

[justinhca]

Thanks Dale. I think the service was running as expected when we just used the default service account that was created on install. The problem was when we tried to change the service account to a domain-based account. I double-checked that the account had the Log On As A Service right, but based on the error, it didn't seem sufficient. I elevated the service account to a Local Administrator, and it's working now. It would be nice to know the actual permissions needed for the service account, so you could assign it exactly what it needs. Thanks for the help with this.

[v-jiascu-msft]

I'm glad you solved your issue. Thank you for sharing with the Community.

Best Regards,

Dale