Performance and memory leak issue - PBIRS Jan 2022

[josef78]

Hello,

After upgrading our production environment to the Power BI Report Server version of January 2022 (1.13.8054.40631), we have significant performance issues.

At real user load (including a slight evening load), we have a significant slowdown in report queries (approximately by 50%), and much higher CPU usage. This happens a few hours after the last restart of services, and worsens over time.

The problem is only on front-end servers (we have scale-out deployment). And the problem is probably somethere in the RSPowerBI.exe process, which higlly utilizes the CPU and also consumes a large amount of RAM (more than the main msmdsrv.exe process). There is nothing unusual in RSPowerBI * .log.

Does anyone else have these problems?

Is there a solution?

[josef78]

Additionall information.
I tested on more servers on another separated environment. There is the same problem, and I think it's a big problem for any large deployment.

The root cause is probably a memory leak bug in RSPowerBI.exe. It seems that the problem is directly proportional to the number of queries going through power bi reports.

In my environment after upgrade to Jan/22, RSPowerBI.exe slowly allocates memory, within 8 hours and about 60 thousand queries, allocates about 50GB of RAM, which also leads to slowdown (about 3x slower response), and unstable state.

[d_gosbell]

@josef78 it does sound like a possible bug, but I've been monitoring our prod servers all day and I'm not seeing anything similar, but it could be due to a difference in workload. I've setup some perfmon counter logs so that I can track this over a longer period.

I would suggest raising a support ticket with Microsoft support. Issues like this will get resolved much faster if Microsoft can get access to performance counter logs and multiple memory dumps and the support engineers would be able to tell you what they need and provide mechanisms for securely uploading these sorts of things.

[josef78]

Hi @d_gosbell,

thank you. I'm still monitoring and investigating. This problem is definitely in RSPowerBI.exe, where is some memory leak, and based on amount of leaked memory is there also additional cpu utilization. Memory utilization of this process gradually increasing and is dependent on server user load (especially on number of DAX queries passed from client side, thru RSPowerBI.exe to internal SSAS engine, and back. I'm not sure all other dependencies and conditions.

In my case (on production), after restart of service, RSPowerBI.exe process slowly allocate about 10GB RAM during one hour! (typically this process takes about 500MB after whole day in prev version), and after hour, only this process load CPU on server about 20% (typically cpu utilization of this process is near to zero), and this affect overall performance of server. End of day (after 8hours), only this one process takes about 50GB of RAM, and utilize about 80% of CPU, which make whole server extreme slow. Only my workaround is restart whole PBIRS, or kill RSPowerBI.exe process, every hour, to keep usability of PBIRS.

Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?

[Anonymous]

Any updates on this? We are having similar issue and consistent on all of our Jan 2022 upgraded Power BI servers.

[d_gosbell]

---

@josef78 wrote:

Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?

---

We are running the Jan 22 release and we have 2 x 8 core servers with about 96Gb of RAM I think (it's Saturday here now and I'm not logged into the work VPN to double check this). When I was checking in task manager I saw each instance using between about 700Mb and 1.5Gb of RAM, but both instances seemed to go up and down in terms of RAM usage. Although the bulk of our reports are against external SSAS tabular models not data models inside the PBIX reports.

I'm not entirely sure what function the RSPowerBI.exe process performs. But I'm assuming by it's name that it is probably the component that manages the loading and unloading of data models into the internal SSAS engine and so it's probably also responsible for marshalling DAX queries to the correct database and it probably does the caching of the datasets for visuals. 

So if I had to guess, I would say that there is probably some issue with the "visual cache" (this is my terminology) It can be a problem with any of sort of caching strategy, that if the cache grows too large (either because of a bug or because of the way it is configured). Then it can take longer to search through the cache than it would have to just bypass the cache and go to the original source. This sort of issue would affect all queries, not just those that are already cached since the service will have to search through the cache index any time a query is run to see if there is an entry in the cache or not. Based on your description of your symptoms this is one possible explanation, but there could be others. 

But this is all speculation on my part, but this is what made me suggest raising a support ticket. I think this issue is probably beyond the depth of something we can fix here although it does not hurt to ask here since others may have experienced this also.

I have setup some perfmon data collector sets to capture some of the key performance counters and the CPU useage and working set size of the RSPowerBI.exe process on both our nodes every 5 minutes. So I will be able to check these next week and let you know if I'm seeing a gradual leak on our servers. 

[josef78]

Hi @d_gosbell ,

Thank you so much. We have a similar configuration (16 virtual CPUs, 196 GB RAM on front-end servers). The function of the RSPowerBI process (not sure) is probably for passing and marshaling DAX queries from the client to the correct SSAS database (integrated or connected live), nothing performance intensive (compared to the msmdsrv (SSAS) process).

After some time of monitoring, I can tell you more where the problem is, but I still don't know exactly what it all depends on, and what I can do.

The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly (you can try monitor Handle count under this process). In the Sep21 version, this process uses an almost constant number of handles, over time. After the upgrade to Jan22, the number of event handles in this process grows rapidly, the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to excessive CPU utilization.
But I don't know if this behavior depends on anything else, such as available resources, OS version, or OS configuration.

In my case, it affects me in both test and production environments, and I also create a clean installation of a new server farm, with the same problem. But the problem is difficult to identify because is showing only under heavy load and after some time.

[josef78]

There is Performance Monitor record, for little over one hour of daily real load. Is only for RSPowerBI process (of Jan22 version), and procesor utilization, memory allocation and handles count. At 9am and 10am (every hour) is process killed and restarted. During one hour, there are gradually allocated almost 400000 handles (on my Sep21 test is  constantly under 1500), memory is gradually allocated to 15GB (on my Sep21 test is near to constantly under 1GB), and CPU utilization only of this process grow to average 50%.

PerfMon

[d_gosbell]

You should definitely log a support ticket for this.

If you license Report Server through a Premium Capacity there is a link here https://powerbi.microsoft.com/en-us/support/pro/ where your tenant admin can log a ticket. If you license Report Server through SQL Server Enterprise with SA then your IT deptment should have a way of logging tickets or you might have a Microsoft Account manager which you can escalate this through.

When you log a ticket a level 1 support engineer will contact you and ask for logs/traces etc. Given what you have found already I don't think it will take them long to escalate to the product team. There is a small chance that maybe there is some undocumented configuration setting that will revert to the Sep 21 behaviour, but otherwise giving the team detailed, real world evidence from a production environment linked to an external customer will give this a much higher priority than if someone internal logs this issue.

[Steffe]

We have the same problem after upgrading from September 2021 to January 2022 version.

Looking at monitoring boards for Memory and CPU we can se big differences.

Memory now increasing during the day, until it "swaps all out" and start over. Before upgrading to January editition it wasn´t beahaving like this. Also much heavier CPU utilization compared to when running on September edition of PBIRS.  

Any news on this?

[josef78]

Hi @d_gosbell,

ok, I will try rise ticket (we have MSSQL SA licence), but I'm afraid it will be a long process.

Thank you