Hi all,
I'm looking for help with spooling failures. This type of problem seems to be increasing in frequency. I can start by describing our environment. We have a dedicated Power BI capacity that is basically idle. We also have very beefy gateway servers. Given the amount of hardware we are committing to this, we shouldn't have these severe spooling errors.
In order to find this error, we look for the files named like so:
Report\QueryExecutionReport_200123-DESKTOP_20240523T220000.log
... they can be found in a directory like so:
systemprofile\AppData\Local\Microsoft\On-premises data gateway\
The error looks pretty confusing like you see below. I'm guessing that the most relevant details are hidden from us. They are probably logged on the Microsoft side, and are not found anywhere in our gateway server:
"[""{\""kind\"":\""Web\"",\""path\"":\
""https://power-reporting.ufpi.com/power-reporting-api\""}""]",
Spooling failed with error: The operation failed due to an explicit cancellation. Exception: System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass7_0`1.<<GetNextResponseAsync>b__0>d.MoveNext()--- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.PowerBI.DataMovement.Pipeline.Dataflow.TDFHelpers.<>c__DisplayClass12_0.<<ExecuteBlockOperationAsync>b__0>d.MoveNext()
I've already contacted support, and asked for the logs on the server/kusto side to explain these frequent failures. But as-of now, nothing has been shared yet.
How does this impact us? Oddly enough we do NOT normally observe this error from Power BI in a first-hand way. Instead, we find that there is some type of an "implicit retry" happening as a result of these spooling failures. The failures are observed in a second-hand way. These spooling failures result in retries that are a bigger problem than if Power BI would simply fail on the first attempt. Instead of telling us there was a failure, Power BI appears to suppress these failure messages, and it proceeds to keep initiating one or more retries. The retries will trigger the same mashup container to be launched. The duplicate mashups can run concurrently and they swamp our back-end environment, since they each can take up to an hour to complete.
Furthermore, the cycle of failure and retry is causing our scheduled refresh operations to take longer to complete (users are not getting data when they would expect. )
As mentioned, the mashups themselves are succeeding. But it is the "spooling" opreation that is failing. It is very frustrating because if we look in the normal mashup logs, we never find any problems that we can fix on our own.
There doesn't appear to be any place to configure the retry mechanism that is responsible for these duplicate mashups. These mashups should NOT be repeatedly launched on the same gateway without a custom configuration to that effect.
If anyone recognizes this error message, please let me know how to investigate. I'd like to uncover the reason for the spooling failures, and/or I'd like to be able to configure retries to stop after the first attempt (thereby avoiding the creation of another type of problem on our back-end servers)
