Looking for comprehensive documentation about the Data Gateway configuration and traces available

[phnslzde]

Hello, I'm dealing with several error for which the error messages are not helpful as such. I would like to understand better whats happening under the hood and trace accordingly. I'm getting errors (in Daatapipeline UI:) A task has been cancelled or  like Microsoft.DataTransfer.Worker.WorkerPool.WorkerPoolException: Not enough resources to start a new worker. Current workers count: 1. (IntegrationRuntimeErrors<xxxx>.log file . 

To give some context: I'm testing a connection, so in my perception this doesn't seem a huge workload that the standard installation can not cover. 

Furthermore, I would like that the log output is written immediatly and not delayed so that I can better see what's going on. I'm looking into the trace files of the gateway in C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Microsoft\On-premises data gateway so that I get an change alert in notebook++ if a file changes . The download method from the GW UI is not a good option when you are investigating. 

Any help is appreciated. 

[v-tejrama]

Hi @phnslzde ,
Thank you for bringing this up. The error message:

“Not enough resources to start a new worker. Current workers count: 1”

usually means the gateway host doesn't have enough memory to start more workers. Here are two main steps you can take:

Modify worker memory settings

Go to: C:\Program Files\On-premises data gateway\Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config

Find the setting: OnPremRuntimeDefaultWorkerMaxWorkingSetInMB

Lower this value (for example, to 200–300 MB). If it’s too high, even one worker might use all the memory, which stops new workers from starting.

Upgrade host resources

If your server or VM has limited RAM or CPU (like 4 GB RAM), consider increasing to at least 8 GB RAM and more vCPUs. Each worker needs memory and CPU, so more resources let more workers run at once.

Other tips:

Make sure your gateway is updated to the latest version. .If you run multiple jobs at once, try lowering pipeline concurrency. Restart the gateway service if you think there are stuck worker processes.

[phnslzde]

Hello, Thank you! I run it with 16 GB and currently 4 GB of RAM free. I changd the parameter to no success. The process "FabricPipelineWorker.exe" Is started when I test a connection. Then it just dies without further details observable for me in the logs. on Fabric I just get the generic error message, sometimes I can find some more. Can I influence trace behaviour of this process? I checked for a .config file related to it but I couldn't find any. That's why I asked for more documentation or comprehensive literature. It is not the only error I have, in relation with another connection type I also observe unclear behaviour. And again, I'm just checking the connection , so no major data transfer , etc is happening. 

The gateway is the latest version, September 2025 release. Cheers.

[v-tejrama · Accepted solution]

Hi @phnslzde ,

Thank you for your detailed follow-up. I recognize the difficulties that arise when the FabricPipelineWorker process fails without clear logs, which can complicate troubleshooting. At present, there is no comprehensive public documentation covering all gateway worker behaviors, but here are several key points:

• Trace configuration: You are monitoring the appropriate gateway logs at C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Microsoft\On-premises data gateway. Traces for FabricPipelineWorker.exe are found in IntegrationRuntimeErrors<xxxx>.log and related DataMovement logs. Please note that logging verbosity is controlled globally via the Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config file; setting this to Verbose will provide more detailed entries, though it will increase log size.
• Real-time log monitoring: Notepad++ with “file change alert” remains the most effective tool for viewing logs in real time. Logs are written asynchronously, so some delay is expected, and immediate line-by-line flushing is not available.
• Worker crash analysis: If FabricPipelineWorker.exe crashes, reviewing the Windows Event Viewer (Application and System logs) can help identify issues such as .NET runtime or memory problems. It is advisable to review these logs alongside gateway logs.
• Resources and connections: Certain drivers or connectors, including Oracle, SAP, and ODBC, may consume more memory than expected, even during basic connection tests, which can result in cancellations without data transfer.

Given that you are using the latest September 2025 build and have updated worker memory settings, I suggest enabling Verbose tracing, reproducing the issue, and collecting both gateway and Event Viewer logs. If the problem continues, submitting a support ticket with these details will help engineering determine whether the cause is related to a connector or resource allocation.

Thank you,

Tejaswi.