Updated Gateway: Refresh Failures

[Anonymous]

We keep getting intermittent failures on random datasets within our Power BI workspaces. This has been happening since we updated to the new February 2019 gateway. We were having some refresh failures previously due to the gateway timing out, so we thought that upgrading would resolve the issues. It appears to have exascerbated it. 

The errors look like the below: 

Last refresh failed: Sun Mar 10 2019 17:40:35 GMT-0400 (Eastern Daylight Time) 
Before the data import for Requests finished, its data source timed out. Double-check whether that data source can process import queries, and if it can, try again.

Cluster URI:	WABI-US-EAST2-redirect.analysis.windows.net
Activity ID:	aa4d9bb5-98b5-45a4-8749-4dbd3e36313b
Request ID:	ecc6c044-0306-4ade-b631-5ea50740062c
Time:	2019-03-10 21:40:35Z

[PrahaPalalace]

@StephenMEMC  from https://portal.office.com/AdminPortal/Home#/homepage

[v-qiuyu-msft]

Hi @StephenMEMC, 

Does the issue happen with schedule refresh only? Is manual refresh working? 

I found the error in your gateway log files: 

"The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time.
The duration spent while attempting to connect to this server was - [Pre-Login] initialization=15009; handshake=10331"

Please check if there is any heavy load on the SQL server when the issue occurs. 

Best Regards,
Qiuyun Yu 

[StephenMEMC]

Hello @v-qiuyu-msft,

It's all refreshes (manual or scheduled). When I do manually, at least 2 or 3 of the 30 fail (and I just click it again). Just now when I did it - 8 of them failed!

This server is bored out of its mind 24 hours a day. It's a pretty powerful workhorse:

OS: Windows Server 2012 R2

Processor: 2x Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz (8 cores each)

Memory: 128GB

Hard Disk: PERC H710P RAID6 (6 x 1787GB SSD drives) for 7151GB total usable space. We have 3 partitions:

• C:\ (OS and basic apps only): 150 GB
• D:\ (SWAP file and Installation files): 300GB
• F:\ (SQL and other related applications): 6.54TB

It also has a cachecading SSD hard drive in it which speeds up read speed. It's set up to NIC team (4 x 1Gbps cards) as well.

As a test, I'll go edit the schedule and have them all refresh at 8pm this evening (when the office is closed) and see if the success rate goes up. It is a production system that is in use during refresh times, but I don't believe the server is the culprit here. 

[Anonymous]

My refreshes only fail when they are scheduled. We have less traffic over the weekend on all of ours servers, and that is when we get the most failures. Our reports that are failing hit multiple servers at different times of the day. 

[StephenMEMC]

@v-qiuyu-msft - Refresh at 8pm still had 4 fail, so it's no better. Problem must be elsewhere.

[v-qiuyu-msft]

@Anonymous @StephenMEMC 

I would suggest you create a support ticket to get dedicated support. 

Best Regards,
Qiuyun Yu 

[Anonymous]

I was advised via my support ticket to revert my gateway to the 12/2018 version. An additional update was made to the gateway that was pushed this week, and I can update to the March version now (and the hope is that the refresh failures will not be an issue). I haven't updated to the latest version of March's update. I can confirm that the December 2018 gateway resolved all of my issues. 

[StephenMEMC]

I got the same advice. December 2018 release worked great and I also updated to the March release this morning and can confirm no issues with it. 

[Anonymous]

Thank you for the update! I was a little scared to update to the March release and get another onslaught of failures.