Avoiding the dreadful DM_GWPipeline_Gateway_SpooledOperationMissing in 2024

[dbeavon3]

I'm so tired of the persistent bugs in PBI gateways, including this useless message about a missing spooled operation:

Error: Encountered user gateway exception: '[GatewayPipelineErrorCode]=DM_GWPipeline_Gateway_SpooledOperationMissing [ErrorShortName]=GatewayClientErrorResponseException[GatewayId=89123]/UnknownSpooledOperationIdException [ExceptionErrorShortName]=GatewayClientErrorResponseException[GatewayId=89123]'. RootActivityId = 1d89b6a9-bff7-444e-934d---233543ea8f.Param1 = [GatewayPipelineErrorCode]=DM_GWPipeline_Gateway_SpooledOperationMissing [ErrorShortName]=GatewayClientErrorResponseException[GatewayId=89123]/UnknownSpooledOperationIdException [ExceptionErrorShortName]=GatewayClientErrorResponseException[GatewayId=89123]

Is there a reliable way to avoid this bug in 2024? 
Should we be restarting gateway servers once a day?

Should we be transitioning to Gen2 dataflows? 
How do we avoid the "spooling" part of the ETL -  since this seems to be the most fragile step in the process?

It has been a while since I've tried looking into the cause of this error.  If I enabled the enhanced logging and searched my gateway logs, or the event logs, would I be able to discover an underlying trigger that is causing this bug?

Is there any hope that Microsoft will either fix the bug?  Or improve the messaging, when the bug happens?  It is still shocking to me that such a meaningless and non-actionable message is constantly being shown to the users of Power BI.  Even a very technical type of person would NOT have a way to understand this obscure message (let alone the average PBI report developer).

I understand that bugs exist in every software, and I understand that there are unexpected/unpredictable conditions can appear from time to time which may cause software to break.  That is not my the source of my complaint.  The bigger issue is the fact that PBI will surface a totally meaningless message this frequently, in a language that means nothing to the reader.   They might as well just say : "Something bad happened; please try your luck again another day." 

[lbendlin]

Try

      <setting name="StreamBeforeRequestCompletes" serializeAs="String">
        <value>True</value>
      </setting>

in Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config

[dbeavon3]

Hi @lbendlin 

Yes, I think I remember you mentioning that a year ago or so.  I guess I was dragging my feet.  As buggy as the default setting is, it scares me to customize the gateway config.  (It is also a little annoying to document the recipe for every new gateway that we might build in the future.)

There is another factor.  The network team at my company uses this overly sophisticated network path to PBI, that involves a zscaler proxy in Chicago.  That netowrk path is the only other variable in play.  I am waiting to see what happens when they simplify the path.  Although if it was zscaler that is the culprit, I would hope the Gateway's error message would be a little more straightforward.  (Even a "connection reset by peer" message would be more useful to me than DM_GWPipeline_Gateway_SpooledOperationMissing).