DM_GWPipeline_Gateway_TimeoutError for all refreshes through a VM

[ghaines]

Since 7 December all refreshes for our organisation have been failing.  They are all administered through an on-premises data gateway, which is showing online in Power BI Service but has at least a 95% failure rate when a refresh is triggered.

We were using the December 2023 which had been functioning without issue since install.  Updating to current latest did not fix.  System restore for the VM to 2 December (and previous data gateway version in the process) did not fix.

Has anyone else had issues with this recently?  I have revewed most of the similar posts on this forum for the same issue and they all have a different context or are unreasonable in their solution:  Setting up a data gateway on a physical machine is not an option.

Using Power BI Pro.

[Poojara_D12]

Hi @V-yubandi-msft 

• Check Connectivity: Ensure the gateway is online in Power BI Service and test data source connections.
• Review Logs: Enable and review gateway logs for errors (found in %localappdata%\On-premises data gateway\).
• Verify Data Sources: Confirm data sources are correctly mapped to the gateway and re-authenticate credentials if needed.
• Check VM Environment: Ensure the VM has sufficient resources, stable network, and required firewall ports (e.g., TCP 443).
• Identify Changes: Investigate any system, network, or service changes since 7 December. Check the Power BI Support page for known issues.
• Reinstall Gateway: If issues persist, uninstall and reinstall the latest gateway version.
• Escalate: Contact Microsoft Support with logs if unresolved.
• Workaround: Use a personal gateway or refresh manually in Power BI Desktop as a temporary solution.

Did I answer your question? Mark my post as a solution, this will help others!

If my response(s) assisted you in any way, don't forget to drop me a "Kudos" 🙂

Kind Regards,
Poojara
Data Analyst | MSBI Developer | Power BI Consultant
Consider Subscribing my YouTube for Beginners/Advance Concepts: https://youtube.com/@biconcepts?si=04iw9SYI2HN80HKS 

[V-yubandi-msft]

Hi @ghaines ,

I hope everything's going great for you!

Since we haven't heard from you, we wanted to kindly follow up to see if the solution we offered resolved your issue. Please let us know if you need additional assistance. Your feedback is important to us, and we look forward to your response.

Thank You.

[V-yubandi-msft]

Hi @ghaines ,

We wanted to follow up on the solution I provided earlier for the [DM_GWPipeline_Gateway_TimeoutError for all refreshes through a VM].

Could you let me know if the solution has been helpful? Additionally, if there’s anything more you need assistance with, I would be happy to help. Your feedback is important to us to ensure that your needs are effectively addressed.

Please inform me if you have had the opportunity to test the solution or share any additional details if further assistance is required.

Thank you for your time, and I look forward to your response.

[V-yubandi-msft]

Hi @ghaines ,

Thanks for reaching out to the Microsoft Fabric Community Forum.

• To tackle timeout errors caused by complex data models or queries, enable incremental refresh in Power BI. This will update only new or changed data instead of refreshing the entire dataset, making the process faster.
• Ensure both Power BI and the gateway are up to date. The gateway has a default timeout of 2 hours, which can be adjusted as needed.
• Check the gateway logs for specific error messages that could provide detailed insights. You might also want to increase the Command Timeout and Connection Timeout settings in Power Query to avoid frequent timeouts.
• Ensure your VM has enough CPU, RAM, and disk I/O resources to handle Power BI refreshes. If the VM is under heavy load, consider adding more resources.
• Make sure firewall or proxy settings are not blocking necessary ports for the gateway to communicate with Power BI. Ensure that ports 443 (HTTPS), 5671, and 5672 are open.
• Review the resource utilization on the VM hosting the gateway (CPU, memory, disk I/O). If resource usage is high, allocate more resources to prevent timeouts. Also, check for network latency between the VM and data source, as high latency can cause delays.
• Testing the gateway on a different VM or network may help determine if the issue is specific to your current setup.

I hope this helps! If this post was useful, please mark it Accept as Solution so others can find it easily.

[Poojara_D12]

Hi @ghaines 

It sounds like you’re facing a widespread issue with your On-Premises Data Gateway that’s impacting refreshes in Power BI. Given that restoring the system to a previous state didn't resolve the problem and upgrading the gateway also didn't help, here are a few things you can check:

Steps to Troubleshoot and Resolve:

1. Check Gateway Logs:

   • Review the gateway logs on the VM to identify specific errors or patterns causing the failures. Logs can be found in:
   • C:\Users\<username>\AppData\Local\Microsoft\OnPremisesDataGateway\GatewayDiagnostic
   • Logs may show timeouts, authentication issues, or other system-level problems.

2. Gateway Service Status:

   • Verify that the On-Premises Data Gateway service is running. Restart the service and check if that resolves the issue.

3. Check Power BI Service:

   • Ensure there are no Power BI Service outages affecting your region. Check the Power BI support page or the Microsoft 365 Admin Center for any ongoing issues with data refreshes.

4. Test Connections:

   • Test the connections from the gateway to the data sources. In Power BI Service, go to Settings > Gateway connection and test the connection for each data source.
   • Ensure the credentials are still valid for all connected data sources.

5. Network and Firewall Issues:

   • If there were any changes to your network configuration, ensure that necessary ports (443, 80) are open for the gateway to communicate with Power BI Service.
   • Check for any proxy/firewall issues that might be blocking the connection.

6. Gateway Version Compatibility:

   • Although you’ve updated to the latest version, sometimes there can be temporary compatibility issues with specific versions of Power BI or the gateway. Try rolling back to a version from before December 2023 (or a stable version) if you haven’t already.

7. Check Resource Usage:

   • Ensure the gateway VM has sufficient resources (CPU, RAM). High resource usage can lead to failures in refreshing data.

8. Try a New Gateway Installation:

   • As a last resort, consider setting up a new On-Premises Data Gateway on a different machine (if possible) to see if the issue persists.

Further Actions:

• Reach out to Microsoft Support with specific error logs if the issue continues. Since it’s widespread across your organization, this might be a known issue or a service-side problem they can resolve.

Did I answer your question? Mark my post as a solution, this will help others!

If my response(s) assisted you in any way, don't forget to drop me a "Kudos" 🙂

Kind Regards,
Poojara
Data Analyst | MSBI Developer | Power BI Consultant
Consider Subscribing my YouTube for Beginners/Advance Concepts: https://youtube.com/@biconcepts?si=04iw9SYI2HN80HKS 

[tharunkumarRTK]

@ghaines 

Have you verified the gateway logs in your gateway machine? if not then please use this report template and try to fine the relevant error messsages 

https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-performance

You should also verify the logs at the datasource end, normally there is a defined timeout period for each datasource, if the source is not responding then power bi will try again for 3 times and post that the refresh request would fail. 

For SQL Server the default timeout is 10 mins. So check the query log at the datasource and check how your refresh request queries are performing at the source end. 

And, there are few scenarios where the network routing can cause the timeout error. Ideally your gateway server should take the nearest network route to avoid timeout errors. Run the telet and tracert commands in your gateway machine and see if there is any need to change the network route.

If none of these steps are helping then you can reach out to the Power BI support team.

Need a Power BI Consultation? Hire me on Upwork

Connect on LinkedIn

Did I answer your question? Mark my post as a solution!
If I helped you, click on the Thumbs Up to give Kudos. Proud to be a Super User!

[ghaines]

I appreciate the reply!  The refreshes are being cancelled after 1-2 hours when a typical refresh time is ~5-15 mins depending on the dataset.  I don't believe the timeout for the individual datasources is the problem.  The other parts were instructive though, thanks for your input.