AsyncOperationStreamingRetriesExhausted Refresh Failures

[Xaraja]

I am suddenly getting this error on a number of refresh failures in our tenant. It started around 6 pm CST on 1/10. I have not seen this error before, and I am not finding anything useful when I search it online. There are very few results when you search it in Google (less than a page of unique results). It is not listed in the Microsoft documentation on refresh problems. I am only getting this error on <10% of the scheduled and manual refreshes, and it is not the same datasets or the same tables within the datasets each time. Often the dataset that failed will succeed on the next time it is scheduled or when a manual refresh is kicked off.

We use on premises VMs, 3 of them, for the gateway cluster, and all 3 have been manually rebooted yesterday during the day, and had their normal reboot overnight twice since this started on 1/10. The software is up to date and they show online in the Power BI Service. 

All of the refresh errors reference tables in our on premises SQL server, although as mentioned it is many different tables. The only thing that has changed recently with the SQL Server is that the version of Java running on it had to be updated, and we had a problem with one of the tables last week having corrupted data and we had to follow Microsoft documentation to correct that issue. However, the error on that one was clear in what the problem was and it was limited to that one table in the server.

The Premium Capacity Utilization app does not show any results for any times that the capacity is overloaded. 

We do have a very busy tenant with hundreds of datasets, but that is not new so I don't know why this would start suddenly.

Is anyone else seeing this, or does anyone know what this error means? I will file a support ticket if needed, but I would really like to have some idea what this error means.

[Xaraja · Accepted solution]

Microsoft support ended up telling me that it was an outage in the West region that was affecting some but not all customers. They said that the reason it wasn't shown on the outage page was because it wasn't universal to all customers in the region. It tapered off through Friday and Saturday and seems to be completely gone this morning.

[SantoshKumar]

@Xaraja 
I am seeing the issue which was mentioned as below :
XXXXXXXX  has failed to refresh.
Failure details: The last refresh attempt failed because of an internal service error. This is usually a transient issue. If you try again later and still see this message, contact support.

{"error":{"code":"DM_GWPipeline_Client_AsyncOperationStreamingRetriesExhausted","pbi.error":{"code":"DM_GWPipeline_Client_AsyncOperationStreamingRetriesExhausted","parameters":{},"details":[],"exceptionCulprit":1}}} Table: XXXX

as per solution, it was said that it is resolved but still this issue is shows up constantly to me. 
any advice?

thanks

Santosh kumar p

[Xaraja]

I filed a support ticket with Microsoft and spent some time on a conference call with a representative, but we didn't make any changes to our configuration and I'm not sure we actually accomplished anything to fix the problem. That's all I know to suggest. 

Microsoft support did tell me that the problem wouldn't be totally fixed until the end of February, so it's possible that's why you're seeing it, and reporting it to them might be helpful.

[SantoshKumar]

@Xaraja Thank you. I will do the same.

[JSanders28]

@Xaraja This issue also affected us as well, same dates.   Between the 10th and the 13th we had over 200 dataset refreshes fail in production.  It was a lot!   We have an open ticket too.  We also did see some more on 1/20 though.   We received the same response.  As a side note, for @lbendlin  our gateway configs are not set to stream before completion (which is the default setting). 

[lbendlin]

Did you set the gateway config to stream before completion?

Enable  advanced logging on your gateway cluster members, raise a Pro ticket, and be prepared to provide these advanced logs - it will be the first thing they will ask for.