Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
dbeavon3
Impactful Individual
Impactful Individual

Substantial performance degradation East US (XMLA endpoints and MDX)

This is related to dataset performance issues.


We are having serious issues with performance degradation today (09/09/2024) in East US.


We have premium P1 capacity, and no capacity constraints or throttling issues are appearing in the capacity metrics dashboard:

 

dbeavon3_0-1725927690964.png

 

 

 

 

 

I checked the status page for Fabric, and they say everything is running smoothly:
https://support.fabric.microsoft.com/en-US/support/

 

However, it is clear that if we use SSMS to connect to any of our datasets in any of our workspaces that the service is not behaving well.  We have been experiencing timeouts and disconnections all day long like so:

 

dbeavon3_1-1725927492425.png

 

 

I will also be creating a support ticket, but just wanted to know if others had already done so.  I would love to save some time, and I already have too many active support tickets .

 

Any information would be greatly appreciated.  So far I've been testing the connectivity with these clients (.net):

 


Microsoft.AnalysisServices.AdomdClient 15.17.1
Microsoft.AnalysisServices.AdomdClient 19.84.1

and SSMS 19.2

 

 

 

 

 

1 ACCEPTED SOLUTION
dbeavon3
Impactful Individual
Impactful Individual

Hi @lbendlin 

I saw that too, but it doesn't correlate very well. 
My problems are ongoing.

It might be a problem I need to review with our network team.  They have started playing with lots of annoying components like zscaler security tunnels.  Something like that may be interfering with the retrieval of data (for both my own machine, and for servers in the datacenter as well).  At this point I think it is 50/50 that the problem is on our end or on Microsoft's.


The only reason it seems like a possible network problem is because in SQL profiler, the queries appear to finish in a normal amount of time like so (eg. ~2 seconds):


dbeavon3_0-1725930392255.png

 

 

 

 

... but despite the fact that profiler says the query finished, we find the data is not being delivered.  The TCP connection with Microsoft (20.42.6.81) isn't delivering packets to the SSMS application running locally on my workstation.  It just hangs, and wait cursor in SSMS spins indefinitely.

dbeavon3_1-1725930898123.png

 

Between my network team and my support case with Power BI, I hope to figure this out soon.  Thank goodness for SQL profiler.

 

... Another weird thing I noticed is that if the queries return extremely small payloads then I don't get to the point where the network traffic will hang.  But a moderately sized payload like 5 or 10 MB will cause everything to hang.

 

 

View solution in original post

3 REPLIES 3
dbeavon3
Impactful Individual
Impactful Individual

Going to close this.  It is almost certainly related to some annoying zscaler software that is impacting traffic between us and power bi.

 

That was pretty painful, and I wasted much of my day on it....  I think our network team introduced some fancy new configuration (IP SLA for Zscaler tunnels)

dbeavon3
Impactful Individual
Impactful Individual

Hi @lbendlin 

I saw that too, but it doesn't correlate very well. 
My problems are ongoing.

It might be a problem I need to review with our network team.  They have started playing with lots of annoying components like zscaler security tunnels.  Something like that may be interfering with the retrieval of data (for both my own machine, and for servers in the datacenter as well).  At this point I think it is 50/50 that the problem is on our end or on Microsoft's.


The only reason it seems like a possible network problem is because in SQL profiler, the queries appear to finish in a normal amount of time like so (eg. ~2 seconds):


dbeavon3_0-1725930392255.png

 

 

 

 

... but despite the fact that profiler says the query finished, we find the data is not being delivered.  The TCP connection with Microsoft (20.42.6.81) isn't delivering packets to the SSMS application running locally on my workstation.  It just hangs, and wait cursor in SSMS spins indefinitely.

dbeavon3_1-1725930898123.png

 

Between my network team and my support case with Power BI, I hope to figure this out soon.  Thank goodness for SQL profiler.

 

... Another weird thing I noticed is that if the queries return extremely small payloads then I don't get to the point where the network traffic will hang.  But a moderately sized payload like 5 or 10 MB will cause everything to hang.

 

 

lbendlin
Super User
Super User

this seems to be related (although it also seems to have subsided)

 

Microsoft Azure Down? Current status overview | Downdetector

Helpful resources

Announcements
September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

Top Solution Authors