Increase timeout of gateway - Scheduled refresh fails due to timeout

[kjohn10]

Is there a configuration option to increase the timeout of the gateway?

For some reason doing a user-initiated refresh of a dashboard works just fine, but our scheduled refresh fails almost every time, we've had a success maybe once in last 2 weeks.

I'm not sure if any of the gateway config files have an entry for this. All I found is this in

Microsoft.PowerBI.DataMovement.GatewayCommon.dll.config:

<setting name="HttpClientDefaultTimeout" serializeAs="String">
   <value>20</value>
</setting>

Not sure what the value is in, assuming minutes? However the service comes back well before this 20 minutes (within 15 from the scheduled refresh) stating "something went wrong".

Logs show these error messages:

DM.EnterpriseGateway Error: 0 : 2017-08-31T16:45:20.1113981Z DM.EnterpriseGateway MDGR 9F524699 [DM.Pipeline.Diagnostics] Exception object created [IsBenign=True]: Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.CancellationException: Task is timeouted with System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.TDFHelpers.<>c__DisplayClass9_0`1.<<GetNextResponse>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.TDFHelpers.<>c__DisplayClass12_0.<<ExecuteBlockOperation>b__0>d.MoveNext(); ErrorShortName: CancellationException/TaskCanceledException[HResult=-2146233029]
DM.EnterpriseGateway Error: 0 : 2017-08-31T16:45:20.1113981Z DM.EnterpriseGateway MDGR FFE3A28E [DM.Pipeline.Diagnostics] StackTrace: at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.CancellationException.TraceConstructor()
at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.CancellationException..ctor(String message, Exception innerException, PowerBIErrorDetail[] errorDetails)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.TDFHelpers.<>c__DisplayClass12_0.<<ExecuteBlockOperation>b__0>d.MoveNext()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.TDFHelpers.<>c__DisplayClass9_0`1.<<GetNextResponse>b__0>d.MoveNext()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Threading.Tasks.TaskCompletionSource`1.TrySetCanceled(CancellationToken cancellationToken)
at System.Threading.Tasks.Dataflow.DataflowBlock.ReceiveTarget`1.<CleanupAndComplete>b__16(Object state)
at System.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

[Anonymous]

If you have a mySQL timeout on app.powerbi.com server and not Power BI Desktop, you may be interested in

http://community.powerbi.com/t5/Issues/Aha-mySQL-timeout-due-to-information-schema-on-app-powerbi-co...

[dpombal]

I am having the same issue  could it be any parameter on this file ?? need help

Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config

[v-jiascu-msft]

Hi @kjohn10, @Drobinson1,

I would suggest upgrading the On-premises Gateway to the latest version (14.16.6423.3).

Best Regards!

Dale

[kjohn10]

Already have that latest version installed.

[v-jiascu-msft]

Hi @kjohn10,

The latest version of Gateway has added more minutes of timeout. power-bi-on-premises-data-gateway-august-update

>>>With this month’s release, we are increasing the timeout from 2 to 5 minutes for requests from the Power BI Service to the gateway. This will help avoid error cases due to long-running requests or high network latency.

There could be two possibilities. 

1) Timeout when establish connection from Service to Gateway.

2) Timeout when transfer data.

1. How long will a one-time refresh take?

2. When will the scheduled refresh happen? How about the network status that time?

Could you please post the snapshot of the error message in the Service?

Best Regards!

Dale

[kjohn10]

1. I don't know how long the refresh will take.

2. I've tried scheduling the refresh at off-peak times, last one was just before midnight (local time).

Something to note, I tried pulling the same data feed using SSIS and found the source table (in SAP) that is causing the problem, and SSIS errors when trying to validate the connection. Therefore we'll likely need to investigate another way to pull that data.

For my initial question - is there a configurable timeout value anywhere in the gateway? or its just hard-coded and we cannot change it?

Here's the error message in the service (note the text cutoff at the end is: "...or from a timeout"):

[Drobinson1]

I am having the same issues!