Scale-out deployment - weird behavior on one server

[anqapl91]

Hello All!

I have really weird situation happening on my scale out deployment. I have two servers that are connecting to MSaaS environment. Both servers are in version Jan 2021. Before them I have loadbalance URL that is distributing users between servers. For some reports, rather bigger than smaller, loading time is sometimes very long, counted in minutes and sometimes report never loads. When I clear cache problem usually disappears but happening pretty quickly again.

I turned on developer tools and I found out that this long loading time is only happening with server which was connected as second in the scale out.  Additionally all is being stuck on ModelsAndExploration, without new data being transferred. Eventually it ends with error message. 

I will paste screenshots tomorrow as I am off work already, but I am kinda running out of ideas what it could be - only on one server, and always the same one... Hardware and setup is identical.

Edit:

Please see error message and response from server:

[d_gosbell]

Your DBA should be able to check pretty easily which instance in the AG each of your PBIRS nodes is connected to, but I would hope that one of them is not connected to a readonly replica.

Have you tried logging onto the problem server via remote desktop and running the report against http://localhost/reports ? This would bypass the load balancer and would tell you if this is possible a load balancer configuration issue or if it is a problem on the one specific server. If it still fails on the server itself you may want to consider raising a support ticket with Microsoft as this is going to need more detailed troubleshooting than is possible on a public forum.

[Icey]

Hi @anqapl91 ,

---

Both servers are in version Jan 2021. Before them I have loadbalance URL that is distributing users between servers.

---

It looks like you are upgrading from a lower version to Jan 2021 (Version: 15.0.1105.230), right?

In addition, have you try to updated the load balancer synchronously?

Best Regards,

Icey

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[anqapl91]

Hi @Icey , thanks for a reply.

I am on Jan 2021 from couple months right now.

By this sentence I ment that my users are using LB link that distributes traffic between nodes. 🙂

Upon further checking I was able to find in the logs that modelsAndExploration method is failing because of the fact that SQL query to ReportServerDB is timing out, Report Server is throwing an exception and unable to build a model. 

Now I am kinda puzzled as I cannot check log on DB side (restriced access) and DBA told me this is typically caused by the fact that connection is not using Multi Subnet Failover. I am pretty sure it must be using, but is there documentation available that is describing this connection details?

[d_gosbell]

---

@anqapl91 wrote:

Now I am kinda puzzled as I cannot check log on DB side (restriced access) and DBA told me this is typically caused by the fact that connection is not using Multi Subnet Failover. I am pretty sure it must be using, but is there documentation available that is describing this connection details?

---

Unfortunately PBIRS does not support multi subnet failover (see https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/reporting-services-... ) but if this was the issue I would expect that both nodes would be failing. This does not really explain why only one node is not working unless you have the two Report Server nodes connected to different database instances in your AG. You could open up the Report Server Configuration Manager on both nodes to double check that they the database connections are configured to use the same database server instance.

[anqapl91]

Ah thanks @d_gosbell , I was looking everywhere to find this info! 🙂

Both Report Servers are connected using same configuration - through listener for Availability Group -  may it be that this second one is connected to a readonly replica?

One more hint - simply removing the report and adding it back in different directory does not help. I am currently observing this issue with two reports, out of over 100. One of them could be really compressed, but is having only Excel as a source and is a size of around 200MB. I am pretty confident developer knows how to model reports, I trust him when it comes to report design. 😃