Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Be one of the first to start using Fabric Databases. View on-demand sessions with database experts and the Microsoft product team to learn just how easy it is to get started. Watch now

Reply
aaronhaviland
Advocate II
Advocate II

ServerError_DM_GWPipeline_Gateway_MonitoredDbConnectionError

‎10-02-2024 08:24 AM

We started seeing this error message this morning with our reports in the Service, and we cannot find any information about it. Would anyone be able to shed any light on what this error means?

 

The issue is sporadic, and will usually go away upon a report refresh. However, due to the large number of reports, the gateways are logging the error every few seconds. Somewhere around 1 out of 20 report views/data refreshes will generate this error. It's not tied to a specific report, dataset, workspace, or data source.

 

We are using 2 gateways in a cluster, and the error happens whether we have one, the other, or both active.

We updated to the latest gateway version on 9/20, but the error did not start until this morning (10/2)

Gateway version: 3000.238.11 (September 2024)

 

This is one instance of the error from the GatewayError.log:

 

 

 

DM.EnterpriseGateway Error: 0 : 2024-10-02T14:50:58.3649630Z DM.EnterpriseGateway	ca2c56c1-ef8a-4b24-9eef-035611a03a37	01ebdc9e-147b-44a5-a062-4754b91805a2	MGEQ	3fcd3ed7-9f96-4555-b1cc-9de996f27af9	4c7373c6-2d55-4bba-a8d2-abf7998eee54	4c7373c6-2d55-4bba-a8d2-abf7998eee54	8B21C03B [DM.Pipeline.Diagnostics] Exception object created [IsBenign=False]: Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.GatewayPipelineMonitoredDbConnectionException: Failed to retrieve connection: no connection with ID exists.; ErrorShortName: GatewayPipelineMonitoredDbConnectionException
DM.EnterpriseGateway Error: 0 : 2024-10-02T14:50:58.3649630Z DM.EnterpriseGateway	ca2c56c1-ef8a-4b24-9eef-035611a03a37	01ebdc9e-147b-44a5-a062-4754b91805a2	MGEQ	3fcd3ed7-9f96-4555-b1cc-9de996f27af9	4c7373c6-2d55-4bba-a8d2-abf7998eee54	4c7373c6-2d55-4bba-a8d2-abf7998eee54	952FAEE2 [DM.Pipeline.Diagnostics] StackTrace:    at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.GatewayPipelineMonitoredDbConnectionException.TraceConstructor()
   at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.GatewayPipelineMonitoredDbConnectionException.ConstructorInternal(Boolean deserializing)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_1.<<ExecuteAdoQueryAsync>b__1>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_1.<ExecuteAdoQueryAsync>b__1()
   at Microsoft.PowerBI.DataMovement.Pipeline.Metrics.PipelineMetricUtils.<MeasureQueryOperationAsync>d__8`1.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.Metrics.PipelineMetricUtils.MeasureQueryOperationAsync[T](IMetricsWorker worker, Type type, Func`1 action, String scenario, List`1 dataSourceKinds)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_0.<<ExecuteAdoQueryAsync>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_0.<ExecuteAdoQueryAsync>b__0()
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivityAsync>d__7`1.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.ExecuteInActivityAsync[T](PipelineActivityType pipelineActivityType, Func`1 action)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessorDispatcher.<DispatchImplAsync>d__3.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessorDispatcher.DispatchImplAsync(GatewaySerializationContext responseContext, OperationRequestBase requestObj)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessorDispatcher.<DispatchAsync>d__1.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessorDispatcher.DispatchAsync(GatewaySerializationContext responseContext, OperationRequestBase request)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.Serialization.GatewayDeserializer.<>c__DisplayClass12_0.<<DeserializeAsync>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.Serialization.GatewayDeserializer.<>c__DisplayClass12_0.<DeserializeAsync>b__0()
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivityAsync>d__7`1.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.ExecuteInActivityAsync[T](PipelineActivityType pipelineActivityType, Func`1 action)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.Serialization.GatewayDeserializer.<DeserializeAsync>d__12.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.Serialization.GatewayDeserializer.DeserializeAsync(GatewayRelayContext requestContext)
   at System.Threading.Tasks.Dataflow.ActionBlock`1.ProcessMessageWithTask(Func`2 action, KeyValuePair`2 messageWithId)
   at System.Threading.Tasks.Dataflow.ActionBlock`1.<>c__DisplayClass6_1.<.ctor>b__2(KeyValuePair`2 messageWithId)
   at System.Threading.Tasks.Dataflow.Internal.TargetCore`1.ProcessMessagesLoopCore()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

DM.EnterpriseGateway Error: 0 : 2024-10-02T14:50:58.3649630Z DM.EnterpriseGateway	ca2c56c1-ef8a-4b24-9eef-035611a03a37	01ebdc9e-147b-44a5-a062-4754b91805a2	MGEQ	3fcd3ed7-9f96-4555-b1cc-9de996f27af9	4c7373c6-2d55-4bba-a8d2-abf7998eee54	4c7373c6-2d55-4bba-a8d2-abf7998eee54	590DBA9A [DM.Pipeline.Common.TracingTelemetryService] Event: FireActivityCompletedWithFailureEvent (duration=2, err=GatewayPipelineMonitoredDbConnectionException, rootcauseErrorEventId=0)
DM.EnterpriseGateway Error: 0 : 2024-10-02T14:50:58.3649630Z DM.EnterpriseGateway	9790b7d0-3ea1-46b0-a9c2-0bfccc120368	01ebdc9e-147b-44a5-a062-4754b91805a2	MGPP	3fcd3ed7-9f96-4555-b1cc-9de996f27af9	4c7373c6-2d55-4bba-a8d2-abf7998eee54	4c7373c6-2d55-4bba-a8d2-abf7998eee54	590DBA9A [DM.Pipeline.Common.TracingTelemetryService] Event: FireActivityCompletedWithFailureEvent (duration=2, err=GatewayPipelineMonitoredDbConnectionException, rootcauseErrorEventId=0)
DM.EnterpriseGateway Error: 0 : 2024-10-02T14:50:58.3649630Z DM.EnterpriseGateway	e9442617-5ffa-4a6c-a73c-7f4422131549	01ebdc9e-147b-44a5-a062-4754b91805a2	MDSR	3fcd3ed7-9f96-4555-b1cc-9de996f27af9	4c7373c6-2d55-4bba-a8d2-abf7998eee54	4c7373c6-2d55-4bba-a8d2-abf7998eee54	4C1EF73E [DM.GatewayCore] Error processing request: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.GatewayPipelineMonitoredDbConnectionException: Failed to retrieve connection: no connection with ID exists.
  TemplateMessage: Failed to add or retrieve monitored DbConnection.

GatewayPipelineErrorCode=DM_GWPipeline_Gateway_MonitoredDbConnectionError
GatewayVersion=
  ([0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.GatewayPipelineMonitoredDbConnectionException.StackTrace:)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_1.<<ExecuteAdoQueryAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_1.<<ExecuteAdoQueryAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.Metrics.PipelineMetricUtils.<MeasureQueryOperationAsync>d__8`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessor.<>c__DisplayClass4_0.<<ExecuteAdoQueryAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivityAsync>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessorDispatcher.<DispatchImplAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayProcessorDispatcher.<DispatchAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.Serialization.GatewayDeserializer.<>c__DisplayClass12_0.<<DeserializeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivityAsync>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.Serialization.GatewayDeserializer.<DeserializeAsync>d__12.MoveNext()

 

 

 

 

Labels:
Message 1 of 12
3,673 Views
1 ACCEPTED SOLUTION
v-tianyich-msft
Community Support
Community Support

‎10-17-2024 11:34 PM

Hi @aaronhaviland ,

 

Sorry for late reply.This issue has been resolved, please upgrade the gateway to the October release (3000.242.5).

 

Best regards,
Community Support Team_ Scott Chang

View solution in original post

Message 12 of 12
2,219 Views
11 REPLIES 11
v-tianyich-msft
Community Support
Community Support

‎10-17-2024 11:34 PM

Hi @aaronhaviland ,

 

Sorry for late reply.This issue has been resolved, please upgrade the gateway to the October release (3000.242.5).

 

Best regards,
Community Support Team_ Scott Chang

Message 12 of 12
2,220 Views
veda_ummenthala
Frequent Visitor

‎10-09-2024 11:40 PM

Hi All,

Check the refresh on PBI desktop, if it works repulish to service and make a refresh. Its working.

Message 10 of 12
3,038 Views
0
aaronhaviland
Advocate II
Advocate II
In response to veda_ummenthala

‎10-10-2024 11:55 AM

There is no issue with the reports themselves. 2/3 of the time, they run fine, and don't return any errors.

Still, to put another checkmark in the process of elimination, I've tried as you suggested, and republished a few reports.

It did not resolve the issue.

 

(Even if it had, having to re-upload over 1,000 reports to fix a gateway issue is not realistic solution)

Message 11 of 12
2,890 Views
lbendlin
Super User
Super User

‎10-02-2024 09:57 AM

Check the gateway logs to see which connections are actually being used. Compare QueryStart versus QueryExecution to see if you can find gaps  (ie queries started but didn't complete).  If the logs are nominal look for other connections defined for the gateway but not present in the logs. You could then try to recreate these connections in the Power BI service.

Message 2 of 12
3,655 Views
0
aaronhaviland
Advocate II
Advocate II
In response to lbendlin

‎10-02-2024 12:04 PM

I was able to captured some specific instances of the error, and identified that the issue is happening across all DirectQuery connections (8 Oracle DB connections, and Snowflake). I cannot find any common element between them. Since it's also happening with Snowflake, that rules out the possibility of it being an issue with the in-house DBs.

 

QueryExecutionReport reports "QueryExecution failed with error: Failed to retrieve connection: no connection with ID exists."

 

I did also find this "Object reference" error in GatewayError.log which matches up with the errors in  GatewayInfo.log:

 

[DM.GatewayCore] Error while updating connection status. ConnectionId: c5480ca0-0add-414f-a2e6-cf54f47c9864 - System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Data.Mashup.MashupConnection.UpdateProgress(IEnumerable`1 progresses)
   at Microsoft.Data.Mashup.MashupConnection.get_Progress()
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.AdoNetStatusUtils.GetCurrentStatusFromMashupConnection(MashupConnection connection)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.AdoNetMonitoredDbConnectionPool.GetAndProcessStatus(Guid connectionId, IAdoMonitoredConnection connectionWithStatus, Nullable`1 activeQueryIdToRemove)
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.AdoNetMonitoredDbConnectionPool.UpdateConnectionStatus(Guid connectionId)

 

Message 3 of 12
3,629 Views
0
Stevm8
New Member
In response to aaronhaviland

‎10-09-2024 04:24 PM

Confirming I have this exact issue - PowerBI live service  is failing to refresh seemingly at random due to a "

ServerError_DM_GWPipeline_Gateway_MonitoredDbConnectionError".
Message 8 of 12
3,088 Views
0
lbendlin
Super User
Super User
In response to Stevm8

‎10-09-2024 04:50 PM

If you have a Pro license you can open a Pro ticket at https://admin.powerplatform.microsoft.com/newsupportticket/powerbi

Message 9 of 12
3,079 Views
0
malarkey99
Regular Visitor
In response to aaronhaviland

‎10-08-2024 10:18 PM

Since 10-08-2024 we have exactly the same problem with direct query connections.

 

We get the 'ServerError_DM_GWPipeline_Gateway_MonitoredDbConnectionError'

with Gateway version: 3000.238.11

 

Where you able to identify the reason for this error?

Message 5 of 12
3,275 Views
aaronhaviland
Advocate II
Advocate II
In response to malarkey99

‎10-09-2024 07:12 AM

Per @lbendlin 's suggestion above, I have posted this in the Issues subforum https://community.fabric.microsoft.com/t5/Issues/Direct-Query-refresh-failures-after-upgrading-Gatew...

Message 7 of 12
3,161 Views
0
aaronhaviland
Advocate II
Advocate II
In response to malarkey99

‎10-09-2024 05:08 AM

No, I haven't been ble to find anything. It's still happening consistently. I've been otherwise tied up and haven't had a chance to investigate further.

 

Thanks for commenting, and confirming that we're not the only one experiencing this issue since updating to 3000.238.11.

Message 6 of 12
3,200 Views
0
lbendlin
Super User
Super User
In response to aaronhaviland

‎10-02-2024 12:41 PM

Usually one would say "What if you downgrade the gateways to a well known good version like 226.5".  But that would be cruel, as we all know that you cannot downgrade gateways.

 

If you have a Pro license you can open a Pro ticket at https://admin.powerplatform.microsoft.com/newsupportticket/powerbi
Otherwise you can raise an issue at https://community.fabric.microsoft.com/t5/Issues/idb-p/Issues .

Message 4 of 12
3,620 Views
0

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

Dec Fabric Community Survey

We want your feedback!

Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions.

ArunFabCon

Microsoft Fabric Community Conference 2025

Arun Ulag shares exciting details about the Microsoft Fabric Conference 2025, which will be held in Las Vegas, NV.

December 2024

A Year in Review - December 2024

Find out what content was popular in the Fabric community during 2024.

View All