Refresh schedule failures - DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError

Hello Collinq

1. The gateway version is updated (Sep. 2021)

2. reboot isn't helping on it.

3. Gateway diagnostic - Network ports testing - Overall Results : 32/32 tests succeeded.

4. Gateway log- :

DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 45B01A52 [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 completed with an error: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException: Error in dataflow pipeline when sending or receiving a request.

GatewayPipelineErrorCode=DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError
GatewayVersion= --->
Inner exception chain: System.TimeoutException
<pi>System.TimeoutException: The operation has timed out.
at System.Threading.Tasks.Dataflow.Internal.Common.InitializeStackTrace(Exception exception)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass6_0`1.<<GetNextResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()</pi>
--- End of inner exception stack trace ---
([0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException.StackTrace:)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.CheckBlockCompletionStatus(IDataflowBlock block, Exception caughtException)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<GetNextResponseAsync>d__6`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_2.<<RunOperationImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_0.<<RunOperationImplAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationImplAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass7_0.<<RunOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInTopLevelActivity>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.Internal.DiagnosticsContext.DiagnosticsContextExecutor.<ExecuteInTopLevelActivity>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationAsync>d__7.MoveNext()
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd F723F2DE [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 transitioned into Faulted state
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 74770E2E [DM.GatewayCore] Cleaning up 1 outstanding packets for Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 for state transition to Faulted
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 78B5ED3E [DM.GatewayCore] Deleting packet 0