Forum Discussion
On-Prem Data Gateway Silent Error and storage cache
Hello,
I'm writing to try and understand an issue we had in office today. Over the weekend we received a semantic model refresh error that said our data gateway went offline. Our first error came at midnight on Saturday. We have most semantic models scheduled for either midnight or 1am. When I got into the office today and noticed that our reports had not been updated since Friday, I found that within Power BI service under workspace connections when I realized that our on on-prem data gateway was offline.
When I investigated our SQL server which is where our gateway is installed our C: Drive was using 199gb of 200.
The on-premises data gateway service had a status of running.
Our SQL data is stored in the 😧 drive and that was healthy, the gateway is installed on the C: Drive. We expanded the drive and restarted the server. This caused a power automate flow, which uses the SQL trigger "When an Item is created (v2)", to execute everything over everything that was added through the weekend. Our C Drive is now at ~150gb of 250 (since we expanded).
This is leading me to believe that the On-Premises data gateway lost connection sometime Friday, started caching 2.5 days worth of data, and clogged our C Drive.
Is this supposed to happen?
Is there a way for us to know if the gateway lost connection without waiting for a semantic model refresh error?
hello kmeink ,
I will recommend you to follow below steps.
The best proactive options are:
- In the on-premises data gateway app, the Status tab shows whether the gateway cluster is online. In the Power Platform admin center, you can also Check status for the gateway cluster/member. (learn.microsoft.com)
- Microsoft provides cmdlets such as Get-DataGatewayClusterStatus and Get-DataGatewayClusterDatasourceStatus. You can run these on a schedule and send your own email/Teams alert if status changes to offline. (learn.microsoft.com)
- Spooling and persistent cache can hit C:, add Windows monitoring for free disk space, especially on the root/temp path. Microsoft also documents moving the SpoolerDirectory and the gateway temp/cache location to a larger drive. (learn.microsoft.com)
- Microsoft documents gateway log retention/configuration and gateway performance monitoring, which can help you see abnormal behavior before refreshes fail. (learn.microsoft.com)
- Microsoft recommends clustering for high availability; if the primary is unavailable, requests can fail over to another member. (learn.microsoft.com)
Thanks
Harish M
If I resolve your issue. Kindly give kudos to this post and accept it as a solution so other can refer this.
4 Replies
- HarishKM
Super User
hello kmeink ,
I will recommend you to follow below steps.
The best proactive options are:
- In the on-premises data gateway app, the Status tab shows whether the gateway cluster is online. In the Power Platform admin center, you can also Check status for the gateway cluster/member. (learn.microsoft.com)
- Microsoft provides cmdlets such as Get-DataGatewayClusterStatus and Get-DataGatewayClusterDatasourceStatus. You can run these on a schedule and send your own email/Teams alert if status changes to offline. (learn.microsoft.com)
- Spooling and persistent cache can hit C:, add Windows monitoring for free disk space, especially on the root/temp path. Microsoft also documents moving the SpoolerDirectory and the gateway temp/cache location to a larger drive. (learn.microsoft.com)
- Microsoft documents gateway log retention/configuration and gateway performance monitoring, which can help you see abnormal behavior before refreshes fail. (learn.microsoft.com)
- Microsoft recommends clustering for high availability; if the primary is unavailable, requests can fail over to another member. (learn.microsoft.com)
Thanks
Harish M
If I resolve your issue. Kindly give kudos to this post and accept it as a solution so other can refer this. - v-csrikanth
Community Support
Hi kmeink
We would like to inquire whether have you got the chance to check the solutions provided by 98mden HarishKM in community to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community. - v-csrikanth
Hi kmeink
We wanted to follow up and check whether you had a chance to review our previous response and if the issue has been resolved.
If you are still experiencing the issue or have any additional questions, please feel free to share an update. We’ll be happy to assist further.
Thank you,
C Srikanth
Community Support Team 
PauReisResolver II
Hi kmeink,
Your hunch is close, and the thing you're describing has a name. The gateway writes temporary cache files to disk while it shuttles data between your network and the cloud, and Microsoft calls that spooling. There's a whole page on it precisely because filling the disk is a known outcome: "it's possible for an 'out of disk space' error to occur when disk space is full because of the spooler."
By default the spooler lives on C:, under the gateway service account's AppData. Your D: drive is the healthy one, so move it there. In C:\Program Files\On-premises data gateway, take a backup of Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.dll.config, find the SpoolerDirectory setting, point its value at a folder on D:, save, and restart the gateway service. Microsoft suggests putting it on an SSD if you get the choice.
Worth knowing before you assume that's the whole job. If you move the spooler and still run out, the next thing to look at is the mashup engine's persistent cache, which is separate. It uses the root drive and the OS temp path, and you move it by changing the TMP variable for the account the gateway runs under rather than by editing that config file.
On your second question, the spooler folder is the thing to watch. Microsoft's own advice is to monitor that location for adequate free space, so a disk alert pointed at that folder, plus a check on the gateway service state, would have told you on Saturday instead of Monday.
I'd also keep the Power Automate flow in mind separately. A SQL trigger replaying a weekend's rows is its own load, so it may not all be the gateway.
