Gateway dies during refresh

Hi all,

I have a VM running which hosts an on-premise gateway (8 cores, 16GB RAM). After 1 year of perfect operation, I keep getting this error. The error is fully reproducible.

The good case:

1 Error	2 Initiating recovery	3 Re-takeover of gateway works fine	4 Confirming it's running from the cloud
5 Checking diagnostics	6 Service is up and running	7 Service in VM is running	8 Now hitting refresh
			Reading data from an on-premise excel sheet via the gateway....
9 New Mashup containers starting, all good.	10 the report refreshed successfully

Now, if I run a report that takes a little longer (used to work before), the bad case:

1 After initiation	2 CPU gets a little warmer and more mashup containers get started	3 Running diagnostics while the report update is running
Running a power query against an on-premise server's REST API using the gateway...
4 The report takes unusually long time however GUI tells me it's still running	5 Service tells me various gateway connections don't work anymore (all are using the same gateway and the services behind are active):	6 Now the refresh fails:
7 Using 100% CPU	8 The on-premise gateway tells me it's active	9 diagnostics still good
10 After logging out of the gateway and re- logging back in, I get this (CPU still 100%):	11 After re-taking over the gateway	12 all gateway connections are back active

The automated refresh of the ProductCareProjects report used to work for months and started to fail last night.

Following the instructions above, the error is fully reproducible on multiple other Reports. None of them use custom connectors. 

After exporting the gateway's logfiles and inspecting them for errors, I found these patterns:

pattern 1

ODGLogs\ODGLogs f561cb57 2022-08-16T12_48_51\Mashup20220816.000000061.log (1 hit)
Line 67: {"Start":"2022-08-16T10:45:58.5977034Z","Action":"FileSystemAccessHelper/TryIgnoringAccessExceptions","HostProcessId":"4216","NonFatalError":"Could not find file 'C:\\Windows\\ServiceProfiles\\PBIEgwService\\AppData\\Local\\Temp\\Microsoft\\MashupProvider\\Cache\\02ece5c6-12fe-41b0-9ed2-5e4a76653ea9\\Cache.Version'.","ProductVersion":"2.108.167.0","ActivityId":"83bce659-740f-463a-b6ce-d2f6c681e27a","CorrelationId":"","Process":"Microsoft.Mashup.Container.NetFX45","Pid":7928,"Tid":1,"Duration":"00:00:00.0001178"}

pattern 2

ODGLogs\ODGLogs f561cb57 2022-08-16T12_48_51\GatewayInfo20220816.000000032.log (1 hit)
Line 5315: DM.EnterpriseGateway Information: 0 : 2022-08-16T10:43:04.7201280Z DM.EnterpriseGateway 6d85212c-3ece-4786-b70b-602cd3c4a935 ff2fb0b0-826f-4ba9-92e7-38a866375a5b MGPP dfbfedf8-a42d-4542-bbaf-9431ef680d11 21fd02c8-4997-4b0b-964a-084d28aafd59 21fd02c8-4997-4b0b-964a-084d28aafd59 01FA448C [DM.GatewayCore] Async ID: b7b60c04-4182-4eed-8351-cb1a18463653 completed with an error: System.ObjectDisposedException: The CancellationTokenSource has been disposed.

Is it this error?

pattern 3

ODGLogs\ODGLogs f561cb57 2022-08-16T12_48_51\Mashup20220816.000000053.log (4 hits)
Line 1303: {"Start":"2022-08-16T10:35:07.9965344Z","Action":"FileSystemAccessHelper/TryIgnoringAccessExceptions","HostProcessId":"4216","NonFatalError":"The process cannot access the file '28f199e9-6fd9-45d6-bd09-93c5c4ec36cc.dat' because it is being used by another process.","ProductVersion":"2.108.167.0","ActivityId":"17037ac5-e845-45dd-889a-469c3f49d116","CorrelationId":"","Process":"Microsoft.PowerBI.EnterpriseGateway","Pid":4216,"Tid":74,"Duration":"00:00:00.0000535"}

pattern 4

ODGLogs\ODGLogs f561cb57 2022-08-16T12_48_51\GatewayInfo20220816.000000026.log (2 hits)
Line 1427: DM.EnterpriseGateway Warning: 0 : 2022-08-16T10:25:04.4383643Z DM.EnterpriseGateway f5907b19-09c9-48be-b209-e78b93e178ad ff2fb0b0-826f-4ba9-92e7-38a866375a5b MGQN dfbfedf8-a42d-4542-bbaf-9431ef680d11 8e87f5a1-2038-4075-958a-4188b1b97133 8e87f5a1-2038-4075-958a-4188b1b97133 E10466D9 [DM.GatewayCore] Error while polling connection status. System.NullReferenceException: Object reference not set to an instance of an object.

pattern 5

ODGLogs\ODGLogs f561cb57 2022-08-16T12_48_51\GatewayInfo20220816.000000024.log (2 hits)
Line 2323: DM.EnterpriseGateway Information: 0 : 2022-08-16T10:24:04.5297819Z DM.EnterpriseGateway 56008af4-faa5-4ce7-bfc3-8e3e936c2e01 4bf343a2-5b7c-5582-1dc0-c6b999b4e7bf MGPP 98285227-44cd-4789-9eeb-99a7789f3ccd b34ab238-6660-4ec8-bd56-b1387f92f796 b34ab238-6660-4ec8-bd56-b1387f92f796 ADAE1004 [DM.GatewayCore] Async ID: 742a2574-0b46-41db-a136-7d791bc31347 completed with an error: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationExpiredException: The async operation id 742a2574-0b46-41db-a136-7d791bc31347 was already expired.

pattern 6

ODGLogs\ODGLogs f561cb57 2022-08-16T12_48_51\GatewayInfo20220816.000000022.log (1 hit)
Line 6076: DM.EnterpriseGateway Warning: 0 : 2022-08-16T10:20:46.3555182Z DM.EnterpriseGateway fc012d87-465f-42fb-8733-75ee6dab9cb1 946868c0-afcd-9ca5-17ee-50eed3d85a48 MGQN 99b73981-870a-4f46-bb77-24f75d745e7b a7fee37a-7756-4d23-977b-5cd50d8c325f a7fee37a-7756-4d23-977b-5cd50d8c325f E10466D9 [DM.GatewayCore] Error while polling connection status. System.NullReferenceException: Object reference not set to an instance of an object.

Any idea on what could be the issue? It's certainly no timeout issue as in here. Thanks for helping out!