Intermittent Issues and Monitoring for On-Premises Data Gateway in Azure

[razec18]

Hello Community,

I’m reaching out to see if anyone has experienced similar issues with the On-Premises Data Gateway in Azure. I have a VM running in Azure with the gateway installed, which I use to connect Power Automate Cloud and Power BI with my on-premises database environment.

From time to time, even though the VM is up and running, the gateway service appears to stop functioning or encounters intermittent connectivity issues. This results in errors like "The Gateway is offline / cannot be found." Restarting the service usually resolves the problem temporarily, but I’m looking for more long-term solutions and insights.

Here are my main questions:

1. Alerting: Is there a way to set up an alert for when this specific gateway service goes down? This would allow me to respond more quickly when issues arise.
2. High Availability: Has anyone implemented a load balancing or failover solution with the On-Premises Data Gateway? I’d like to minimize downtime by having a backup server that can take over if the primary gateway fails.
3. Root Cause Analysis: The gateway was working perfectly until Saturday around 2:00 PM Brazil Time, and then intermittent issues began. Does anyone have insights on what might cause this kind of sudden instability?

Here are some of the errors I encountered:

{
"error": {
"code": "BadRequest",
"message": "Http request failed: the server did not respond within the timeout limit."
}
}

{
"error": {
"code": "BadRequest",
"message": "Http request failed: the server did not respond within the timeout limit. Please see logic app limits at https://aka.ms/logic-apps-limits-and-config#http-limits."
}
}

Any advice on monitoring, setting up failover, or understanding what might have caused these issues would be greatly appreciated! Please let me know if additional information would be helpful.

Thanks in advance!

[v-jianpeng-msft · Accepted solution]

Your solution is great rajendraongole1 

Hi, @razec18 

High availability is really possible in the gateway best practices in Power BI, which is implemented with gateway clustering.
You can implement a gateway cluster by installing a child gateway (joining the primary gateway). This usually installs this gateway in another VM. 

Manage on-premises data gateway high-availability clusters and load balancing | Microsoft Learn

On-premises data gateway sizing - Power BI | Microsoft Learn

For the purpose of monitoring gateway status, you can use Azure Logic Apps/Power automate for your purposes, you can refer to the following blog:

Monitor your On-premises data gateway with Azure and Power BI – Part 1 – Gateway up-time – Data – Ma...

Third, according to the error message you provided, it is due to the data source response exceeding the response time set by the gateway, resulting in a timeout.

Third, according to the error message you provided, it is due to the data source response exceeding the response time set by the gateway, resulting in a timeout.
You can refer to the following articles about the timeout setting for Azure logic app:

Edit runtime and environment settings for Standard logic apps - Azure Logic Apps | Microsoft Learn

These two problems occur in the settings in Azure, and you need to check them. You can refer to my previous part for high availability and monitoring.

Best Regards

Jianpeng Li

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[razec18]

Hello my friend, thanks for the reply.
Question,

all of the methods you mentioned above are free?
Or I'll have to pay for some of them?

[rajendraongole1]

Hi @razec18 - let's look at options for alerting, improving high availability, and troubleshooting potential causes of intermittent failures.

Azure Monitor: Azure Monitor can be configured to alert you if there is an issue with the VM running your gateway. Set up a custom metric alert for the On-Premises Data Gateway service status on your VM:
Create a Log Analytics Workspace and install the Log Analytics Agent on your VM.
Configure the workspace to collect Windows Event logs related to gateway service restarts and network connectivity issues (e.g., event IDs related to “service stop” or “unexpected termination”).
In Azure Monitor, create alerts based on these logs and send notifications via email, SMS, or integrate with Azure Logic Apps to trigger automated responses.
Power BI Gateway Monitoring: Use the Power BI Gateway Management Console to track the status of your gateway. For more granular monitoring, you could use PowerShell scripts to periodically check the gateway status and send notifications if it goes offline.

Since your gateway was stable until a specific time, several factors could have triggered the instability. Here are some possible root causes and diagnostic steps:

Network Connectivity: Azure VMs sometimes experience connectivity issues if there are network configuration changes, updates, or maintenance in the Azure region.
Check Network Security Group (NSG) Rules: Ensure that no changes have been made to your NSG rules, which might impact the VM's ability to reach your on-premises environment.
Review Firewall and Proxy Settings: Confirm that the firewall and any proxies in your environment are configured to allow the necessary traffic for the gateway. Some firewalls may intermittently block connections due to configured timeouts.
Gateway Logs: The On-Premises Data Gateway logs may provide insight into the cause of intermittent issues. You can find logs in the Event Viewer under Applications and Services Logs > On-Premises Data Gateway Service.
Look for warnings or errors around the time the issue started. Gateway logs typically include errors like timeouts, network drops, or throttling issues.
Azure VM Resource Health: Monitor the resource health of your Azure VM in the Azure portal. If Azure detects any health issues with the underlying hardware or software, they will be reported here.

Ensure your gateway software is the latest version, as updates often resolve stability and performance issues.
Scheduled Restarts: In cases where connectivity issues are frequent, consider scheduling regular restarts of the gateway service during non-peak hours. This can help mitigate memory or resource leaks.
Backup Configuration: If possible, back up your gateway configuration, which will allow for quicker recovery in case of a failure or reinstallation.

Hope these, strategies should improve your gateway’s resilience, and by combining alerts with high-availability clustering, you'll have greater monitoring and control over gateway uptime.