PySpark Notebook is Killed after Encountering Bad Gateway (localhost:8998)

Here is yet another crazy error in Fabric Spark...

LivyHttpRequestFailure: Submission failed due to error content =[{"error":{"code":"BadGateway","message":"Upstream service is unreachable","innererror":{"errordetail":"System.Net.Http.HttpRequestException: Connection refused (localhost:8998) ...

I have the full details down below.  It is some sort of weird .Net / Livy error that happens in the pyspark cell # 18 of a notebook, and references a port 8998 on the local host.  It happens after 30 mins without issues.

Here is the notebook image ("item snapshot")

I'm guessing this is coming from some sort of a discretionary profiling component and is NOT directly related to anything that I'm trying to accomplish in custom code.

There is nothing in my solution that is remotely related to pub-sub messaging so it is confusing to see a stack that references a stack frame like so:

Microsoft.Analytics.Clusters.Services.PubSubAgent.ClusterGatewayHttpClient.SendRequestAsync

LivyHttpRequestFailure: Submission failed due to error content =[{"error":{"code":"BadGateway","message":"Upstream service is unreachable","innererror":{"errordetail":"System.Net.Http.HttpRequestException: Connection refused (localhost:8998)\n ---> System.Net.Sockets.SocketException (111): Connection refused\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\n   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\n   --- End of inner exception stack trace ---\n   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\n   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\n   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\n   at Microsoft.Analytics.Clusters.Services.NodeServices.Utils.HttpRedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\__w\\1\\s\\Services\\NodeServices\\Common\\Utils\\HttpRedirectHandler.cs:line 46\n   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\n   at Microsoft.Analytics.Clusters.Services.NodeServices.Utils.TypedHttpClient`1.SendAsync(Uri uri, HttpMethod method, HttpContent content, IEnumerable`1 requestHeaders, CancellationToken cancellationToken, String traceId, String clusterMsiToken) in C:\\__w\\1\\s\\Services\\NodeServices\\Common\\Utils\\TypedHttpClient.cs:line 325\n   at Microsoft.Analytics.Clusters.Services.PubSubAgent.ClusterGatewayHttpClient.SendRequestToUpstreamService(PubSubRequest request, Uri upstreamUri, Guid newXmsRequestID, String traceId, CancellationToken cancellationToken) in C:\\__w\\1\\s\\Services\\NodeServices\\PubSubAgent\\Service\\ClusterGatewayHttpClient.cs:line 169\n   at Microsoft.Analytics.Clusters.Services.PubSubAgent.ClusterGatewayHttpClient.FallbackToHttpScheme(PubSubRequest request, Uri upstreamUri, Guid newXmsRequestID, String traceId, CancellationToken cancellationToken) in C:\\__w\\1\\s\\Services\\NodeServices\\PubSubAgent\\Service\\ClusterGatewayHttpClient.cs:line 142\n   at Microsoft.Analytics.Clusters.Services.PubSubAgent.ClusterGatewayHttpClient.SendRequestAsync(PubSubRequest request, CancellationToken cancellationToken, String traceId) in C:\\__w\\1\\s\\Services\\NodeServices\\PubSubAgent\\Service\\ClusterGatewayHttpClient.cs:line 104\n   at Microsoft.Analytics.Clusters.Services.PubSubAgent.PubSubMessageProcessor.ProcessRequestAsync(PubSubRequest request, CancellationToken cancellationToken) in C:\\__w\\1\\s\\Services\\NodeServices\\PubSubAgent\\Service\\PubSubMessageProcessor.cs:line 523"}}}] HTTP status code: 502. Trace ID: d37bfad3-9047-4487-8432-2eaf13bb1b27.

.. Obviously I do NOT get this experience when hosting the same spark workload on OSS Spark or databricks.  Can someone give me a clue about what is causing this failure?  Is there an option to disable whatever is happening here (discretionary cluster logging). 

It is very frustrating that Fabric is generating messages that are virtually meaningless, seem to be coming from discretionary profiling components, and are impossible to independently investigate (especially since this failure has occurred in the subset of this product that is NOT actually open source software).  Moreover Livy isn't even a real component of Spark anymore.  I think Apache abandoned it a long time ago.  Why is Microsoft using a dead component?