On-Prem data gateway Timeout / Cancel errors on model processing

[DKHUghes55]

Hello Community,

I am seeing error messages in the On-Prem Data gateway diagnostics.  They indicate the gateway cancelled the operation due to a timeout on a connection.

Below is a snipit of the error message from the Gateway Erros file I exported.

DM.EnterpriseGateway Error: 0 : 2025-11-20T06:20:25.3340787Z DM.EnterpriseGateway 4ea019fb-2a55-45fb-ba0e-11fe00311abd eae3f826-0384-4650-a87a-9c503d8fb6a1 MGON 8886666a-502f-4b52-b71a-d91957aa67b5 7a957211-fcd2-4463-bc11-a6432d3be839 7a957211-fcd2-4463-bc11-a6432d3be839 2D2816FE [DataMovement.PipeLine.GatewayDataAccess] Query ADServers Failed. Fall back to original value.
DM.EnterpriseGateway Error: 0 : 2025-11-20T06:20:25.3965817Z DM.EnterpriseGateway 04927d6d-6253-4016-b80b-6f06905e1c09 f5a64b80-9037-4093-918b-b1e643f32d1d MGAS ddc07aac-f80c-48ce-9283-b6c69c22f50f c0663d61-3aeb-4bc1-9ab7-85ce4efdbb77 d167c80e-7511-47ed-90c2-0ca08754e492 8B21C03B [DM.Pipeline.Diagnostics] Exception object created [IsBenign=False]: Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationStreamingCancelledException: The async operation id 9c1c9483-b534-4e1a-9dda-c78d02ded76d streaming was cancelled.; ErrorShortName: AsyncOperationStreamingCancelledException
DM.EnterpriseGateway Error: 0 : 2025-11-20T06:20:25.1777963Z DM.EnterpriseGateway 5802a228-ae94-49ad-ab59-8c6bdbe7f25b 149a90ba-84b0-4121-bc04-c5e1ac80ba45 MGCC 3ae0f2b7-0835-4ef0-9d1c-fb263a30461e 29c21fe6-c265-4949-a482-8dab9fa79e47 29c21fe6-c265-4949-a482-8dab9fa79e47 A9AE5E6B [DataMovement.PipeLine.GatewayDataAccess] Query result is null on ADServer GC://hou99dcgcpk0001.corp.kbr.com. LookupPropertyName:'Mail'. LookupPropertyValue:'halhouston\SV01545'. TargetPropertyName:'UserPrincipalName'.
DM.EnterpriseGateway Error: 0 : 2025-11-20T06:20:25.4903356Z DM.EnterpriseGateway 04927d6d-6253-4016-b80b-6f06905e1c09 f5a64b80-9037-4093-918b-b1e643f32d1d MGAS ddc07aac-f80c-48ce-9283-b6c69c22f50f c0663d61-3aeb-4bc1-9ab7-85ce4efdbb77 d167c80e-7511-47ed-90c2-0ca08754e492 952FAEE2 [DM.Pipeline.Diagnostics] StackTrace: at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationStreamingCancelledException.TraceConstructor()
at Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationStreamingCancelledException..ctor(Guid asyncOperationId, PowerBIErrorDetail[] errorDetails)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.SpooledOperationController.SpooledOperationPacketManager.EnsureStreamingNotCancelled(CancellationToken packetStreamingCancellationToken)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.SpooledOperationController.SpooledOperationPacketManager.<StreamAsync>d__19.MoveNext()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task.Finish(Boolean bUserDelegateExecuted)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(Task task, Boolean taskWasPreviouslyQueued)
at System.Threading.Tasks.TaskScheduler.TryRunInline(Task task, Boolean taskWasPreviouslyQueued)
at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(Task task, Boolean needsProtection)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at System.Threading.SemaphoreSlim.<WaitUntilCountOrTimeoutAsync>d__31.MoveNext()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(Task completingTask)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Threading.SemaphoreSlim.TaskNode.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

Does anyone have ideas what is causing these to happen?

The Gateway server has 32GB of memory and stats on the performance indicate the CPU reaches around 95% and the Memory reaches around 85% uses.

Gateway software is from June 2025.

Thanks,

DKHughes55

[v-tsaipranay]

Hi @DKHUghes55 ,

We haven’t received an update from you in some time. Could you please let us know if the issue has been resolved? If you still require support, please let us know, we are happy to assist you.

Thank you.

[v-tsaipranay]

Hi @DKHUghes55 ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. 

Thank you.

[v-tsaipranay]

Hi @DKHUghes55 ,

Thank you for reaching out to the Microsoft Fabric Community Forum.

The error AsyncOperationStreamingCancelledException shows that the gateway is cancelling refresh because the AD lookup is timing out. Since your gateway is reaching high load (CPU ~95%, Memory ~85%), these directory queries are more likely to fail during processing.

Appreciate the points from @Zanqueta  on updating the gateway version this is recommended, as newer builds include fixes for AD lookup stability. Also agree with @lbendlin 2 regarding not exposing internal AD details.

Please do the following step helps you resolving your issue:

• Update to the latest On-Premises Data Gateway.

• Reduce heavy or concurrent refresh loads on the gateway.

• Check AD connectivity (network/permissions) and avoid AD lookups during refresh if possible.

If the issue continues after these steps,please can review performance logs further.

Hope this helps. Please reach out for further assistance.

Thank you.

[lbendlin]

You may not want to expose the details of your internal AD servers here on the interwebs.  

What is the query doing - fetching user data from AD?

[Zanqueta · Accepted solution]

Hi @DKHUghes55,

Is not a depper analysis but, looking at your message the error you are encountering in your On-Premises Data Gateway indicates that the operation was cancelled due to a timeout, specifically an AsyncOperationStreamingCancelledException.

This typically occurs when the gateway is unable to maintain a connection to the data source or when the server hosting the gateway is under resource pressure.

• Observation: The logs indicate a failure to query the ADServer (Query result is null on ADServer GC://hou99dcgcpk0001...).
• You are using the June 2025 version of the gateway.
• Recommendation:
  • Verify whether a newer version of the gateway is available and apply updates if so.
  • Microsoft frequently addresses bugs and improves stability in newer releases.

If this response was helpful in any way, I’d gladly accept a 👍much like the joy of seeing a DAX measure work first time without needing another FILTER.

Please mark it as the correct solution. It helps other community members find their way faster (and saves them from another endless loop 🌀.