Refresh schedule failures - DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError

[deliah]

Hello 

Recently we are continuously facing this issue:

Data source error: {"error":{"code":"DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError","pbi.error":{"code":"DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError","parameters":{},"details":[]}}} Table: xls_ExcelData.
Cluster URI: WABI-US-NORTH-CENTRAL-E-PRIMARY-redirect.analysis.windows.net
Activity ID: e49e3825-b0c6-4f5b-8a1a-7e9fe6ff14d2
Request ID: ac517192-6575-4862-9984-58e3a3843de5
Time: 2021-10-04 13:55:50Z

But it will work fine once sometimes.

The Gateway connection is online, and "test all connections" is ok too.

Would you please advise thanks?

[v-xiaotang]

Hi @deliah 

I've searched the error "DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError", but didn't find a similar one, may be you could refer to this article, 

https://docs.microsoft.com/en-us/power-bi/connect-data/service-gateway-onprem-tshoot

Btw, you could use fiddler to monitor HTTP traffic.

Best Regards,

Community Support Team _Tang

If this post helps, please consider Accept it as the solution to help the other members find it more quickly.

[deliah]

 It will work fine sometimes.

[deliah]

1. The gateway version is updated (Sep. 2021)

2. reboot isn't helping on it.

3. Gateway diagnostic - Network ports testing -> Overall Results : 32/32 tests succeeded.

[collinq]

Hi @deliah ,

I think that the key is that the "parameters" are part of the error.  This might be related to your data source.

Here are a couple of threads that give some potential help:

Solved: DM_GWPipeline_Gateway_MashupDataAccessError - Microsoft Power BI Community

Solved: DM_GWPipeline_Gateway_GatewayOutOfDiskSpaceError - Microsoft Power BI Community

[deliah]

Hello Collinq

1. The gateway version is updated (Sep. 2021)

2. reboot isn't helping on it.

3. Gateway diagnostic - Network ports testing - Overall Results : 32/32 tests succeeded.

4. Gateway log- :

DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 45B01A52 [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 completed with an error: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException: Error in dataflow pipeline when sending or receiving a request.

GatewayPipelineErrorCode=DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError
GatewayVersion= --->
Inner exception chain: System.TimeoutException
<pi>System.TimeoutException: The operation has timed out.
at System.Threading.Tasks.Dataflow.Internal.Common.InitializeStackTrace(Exception exception)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass6_0`1.<<GetNextResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()</pi>
--- End of inner exception stack trace ---
([0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException.StackTrace:)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.CheckBlockCompletionStatus(IDataflowBlock block, Exception caughtException)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<GetNextResponseAsync>d__6`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_2.<<RunOperationImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_0.<<RunOperationImplAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationImplAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass7_0.<<RunOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInTopLevelActivity>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.Internal.DiagnosticsContext.DiagnosticsContextExecutor.<ExecuteInTopLevelActivity>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationAsync>d__7.MoveNext()
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd F723F2DE [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 transitioned into Faulted state
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 74770E2E [DM.GatewayCore] Cleaning up 1 outstanding packets for Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 for state transition to Faulted
DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 78B5ED3E [DM.GatewayCore] Deleting packet 0
 

[collinq]

Hi @deliah ,

Is your gateway updated as well?  And, as silly as this may sound, rebooting the gateway server seems to resolve this issue for a lot of folks.  Let us know how that goes and we can troubleshoot from there!

[deliah]

DM.EnterpriseGateway Information: 0 : 2021-10-04T21:04:15.7373696Z DM.EnterpriseGateway 073cf135-5b59-4686-85ff-6179006fb21a 55d691aa-5ec8-4d74-adf4-b5f6f77cfd8f MGPP b974cc62-cc2d-40b0-b5d1-7249b37cf075 f9686faa-2318-49f6-90c3-3b39245acecd f9686faa-2318-49f6-90c3-3b39245acecd 45B01A52 [DM.GatewayCore] Async ID: 31071a30-8adf-4ac0-9583-71e4058e0ea2 completed with an error: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException: Error in dataflow pipeline when sending or receiving a request.

GatewayPipelineErrorCode=DM_GWPipeline_Gateway_DataflowPipelineSendOrReceiveError
GatewayVersion= --->
Inner exception chain: System.TimeoutException
<pi>System.TimeoutException: The operation has timed out.
at System.Threading.Tasks.Dataflow.Internal.Common.InitializeStackTrace(Exception exception)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass6_0`1.<<GetNextResponseAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()</pi>
--- End of inner exception stack trace ---
([0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.DataflowPipelineSendOrReceiveException.StackTrace:)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.CheckBlockCompletionStatus(IDataflowBlock block, Exception caughtException)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass11_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<GetNextResponseAsync>d__6`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_2.<<RunOperationImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass8_0.<<RunOperationImplAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationImplAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<>c__DisplayClass7_0.<<RunOperationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayPipelineTelemetry.PipelineTelemetryService.<ExecuteInTopLevelActivity>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.Internal.DiagnosticsContext.DiagnosticsContextExecutor.<ExecuteInTopLevelActivity>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.<RunOperationAsync>d__7.MoveNext()