Refresh failure after upgrading gateway version to 3000.234.5

[mandyly8888]

Since upgrading to the latest gateway version (3000.234.5 and 230.13), we’ve encountered frequent failures with our Power BI dataset refreshes. These refreshes typically fail but succeed one or 2 retry attempts.

This issue began on August 16, 2024, when the newest version was released.

Has anyone else experienced this problem? Is there any information or update regarding this issue?

[mandyly8888 · Accepted solution]

Issue has been mitigated by Microsoft Product Group, here more details about it:

Related to  "code":"DM_GWPipeline_Client_GatewayUnreachable".

Background Azure Relay's end to end connectivity explained in this document: https://learn.microsoft.com/en-us/azure/azure-relay/relay-what-is-it
Relay's Listening client established 2 connections overall to establish sender/listener rendezvous: One connection ((#1 from the document)) over the VIP of Relay cluster and one connection (#7 from the document) over the Instance-Level-IP (ILPIP) of the VM where Sending client connected to (Note: Relay cluster has multiple VMs to serve requests). 

Root Cause The issue started when we recently performed an infra upgrade (compute and networking components) and when the connections from both Listening client and Sending client ended up on the same VM of Relay cluster since Load Balancer (LB) chooses the VM randomly. As the IP packet is routed to the VM; VIP and ILPIP from the incoming packets are replaced with private IP of the VM. At this point both connections are having the same destination IP and port from the VM point of view. If at all the Listening client is functioning behind the load balancer then it may use same ephemeral source port for both the connections. This will cause collision on the Relay VM as both connections have the same source/destination IP/ports (TCP association/tuple). In this case, destination port is 443 for both of the connections with VM's private IP. Eventually, TCP connections are dropped resulting into reconnects. In summary, for this issue to happen, 2 conditions need to be met: 1. Listening and sending clients connect to same VM on the Relay cluster 2. Same source ephemeral port on both listening client connections.

Next Steps We have the fix ready, and it is being validated. We will be rolling it out to production in accordance with Safe Deployment Practices on next 2-3 weeks. Tentative ETA: 9/27

[v-fenling-msft]

Hi, @mandyly8888 

May I ask if you have gotten this issue resolved? If it is solved, please share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.

If it is not resolved, please share us more detailed information and we will do our best to solve the problem you are experiencing.

I hope my suggestions give you good ideas, if you have any more questions, please clarify in a follow-up reply.
Best Regards,
Fen Ling,
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[suparnababu8]

Hi @mandyly8888 

It seems you’re not alone in experiencing issues with the latest Power BI gateway versions (3000.234.5 and 230.13). There have been reports of similar problems since the release on August 16, 20241. Here are a few steps you can take to address this issue:

1. Check for Known Issues and Updates:

• Microsoft Power BI Blog: Regularly check the Power BI Blog for any announcements or updates regardin...1.
• Power BI Community: Visit the Power BI Community forums to see if other users have reported similar issues and if there are any suggested solutions.

2. Roll Back to a Previous Gateway Version:

• If the issues persist, consider rolling back to a previous stable version of the gateway. This can often resolve issues introduced in newer versions.

3. Monitor Gateway Performance:

• Use the gateway performance monitoring tools to identify any specific errors or bottlenecks during the refresh process. This can provide insights into what might be causing the failures.

4. Increase Retry Attempts:

• Temporarily increase the number of retry attempts for dataset refreshes. This can help mitigate the impact of intermittent failures until a permanent solution is found.

If you need more clarity pls follow.

https://powerbi.microsoft.com/en-us/blog/on-premises-data-gateway-july-2024-release/

https://www.microsoft.com/en-us/download/details.aspx?id=53127&lc=1033

https://www.powercommunity.com/on-premises-data-gateway-february-release/

Thanks!

[mandyly8888]

Thanks for your response @suparnababu8. The reason why we upgraded to the new version is that we are trying microsoft fabric which requires to upgrade gateway. We didn't have issue before this upgrading. 

I contacted the support center, they will review the gateway machine ports connections as it seems it is a network intermittent issue. I will keep you posted for any solution that we receive. Thank you again.

[collinq]

Hi @mandyly8888 ,

I just looked in the support and known issues for Powrer BI and there is no indication there of an issue.  I am wondering if your Desktop files are all up to date as well. 

If you ahve O365 logging turned on, you can see if there is an error there.

Otherwise, this feels like you may need to open a support call with Microsoft.

[mandyly8888]

Thanks for your response @collinq. The desktop files are up to date, and the report can be successfully refreshed if I make another try manually. 
I openned a ticket yesterday, and I was advised it could be possibly firewall issue of the gateway. I will keep you posted if I receive any solution. Thank you again.