Refresh schedule failures - DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError

Hi deliah ,

Is your gateway updated as well?  And, as silly as this may sound, rebooting the gateway server seems to resolve this issue for a lot of folks.  Let us know how that goes and we can troubleshoot from there!

Hello Collinq

1. The gateway version is updated (Sep. 2021)

2. reboot isn't helping on it.

3. Gateway diagnostic - Network ports testing - Overall Results : 32/32 tests succeeded.

4. Gateway log- :

DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 45B01A52 [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 completed with an error: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException: Error in dataflow pipeline when sending or receiving a request.

GatewayPipelineErrorCode=DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError
GatewayVersion= --->
Inner exception chain: System.TimeoutException
<pi>System.TimeoutException: The operation has timed out.
at System.Threading.Tasks.Dataflow.Internal.Common.InitializeStackTrace(Exception exception)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass6_0`1.<<GetNextResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()</pi>
--- End of inner exception stack trace ---
([0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException.StackTrace:)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.CheckBlockCompletionStatus(IDataflowBlock block, Exception caughtException)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<GetNextResponseAsync>d__6`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_2.<<RunOperationImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_0.<<RunOperationImplAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationImplAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass7_0.<<RunOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInTopLevelActivity>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.Internal.DiagnosticsContext.DiagnosticsContextExecutor.<ExecuteInTopLevelActivity>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationAsync>d__7.MoveNext()
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd F723F2DE [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 transitioned into Faulted state
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 74770E2E [DM.GatewayCore] Cleaning up 1 outstanding packets for Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 for state transition to Faulted
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 78B5ED3E [DM.GatewayCore] Deleting packet 0
 

1. The gateway version is updated (Sep. 2021)

2. reboot isn't helping on it.

3. Gateway diagnostic - Network ports testing -> Overall Results : 32/32 tests succeeded.

 It will work fine sometimes.

Hi deliah 

I've searched the error "DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError", but didn't find a similar one, may be you could refer to this article, 

https://docs.microsoft.com/en-us/power-bi/connect-data/service-gateway-onprem-tshoot

Btw, you could use fiddler to monitor HTTP traffic.

Best Regards,

Community Support Team _Tang

If this post helps, please consider Accept it as the solution to help the other members find it more quickly.