topic Re: Performance and memory leak issue - PBIRS Jan 2022 in Report Server

Performance and memory leak issue - PBIRS Jan 2022

josef78 — Tue, 22 Feb 2022 08:06:46 GMT

Hello,

After upgrading our production environment to the Power BI Report Server version of January 2022 (1.13.8054.40631), we have significant performance issues.

At real user load (including a slight evening load), we have a significant slowdown in report queries (approximately by 50%), and much higher CPU usage. This happens a few hours after the last restart of services, and worsens over time.

The problem is only on front-end servers (we have scale-out deployment). And the problem is probably somethere in the RSPowerBI.exe process, which higlly utilizes the CPU and also consumes a large amount of RAM (more than the main msmdsrv.exe process). There is nothing unusual in RSPowerBI * .log.

Does anyone else have these problems?

Is there a solution?

Re: Power BI RS January 2022 - performance and memory leak issue

josef78 — Tue, 22 Feb 2022 08:04:51 GMT

Additionall information.
I tested on more servers on another separated environment. There is the same problem, and I think it's a big problem for any large deployment.

The root cause is probably a memory leak bug in RSPowerBI.exe. It seems that the problem is directly proportional to the number of queries going through power bi reports.

In my environment after upgrade to Jan/22, RSPowerBI.exe slowly allocates memory, within 8 hours and about 60 thousand queries, allocates about 50GB of RAM, which also leads to slowdown (about 3x slower response), and unstable state.

Re: Performance and memory leak issue - PBIRS Jan 2022

v-henryk-mstf — Thu, 24 Feb 2022 06:08:13 GMT

Hi @josef78 ,

According to your description, for this kind of problem, it may be a problem related to iFrame in IE, try to switch to another browser, such as Edge or Chrome.

Refer to the following similar solutions.

Embedded PowerBI performance and memory leak - Page 2 - Microsoft Power BI Community

If the problem is still not resolved, please point it out. Looking forward to your reply.

Best Regards,
Henry

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Performance and memory leak issue - PBIRS Jan 2022

josef78 — Thu, 24 Feb 2022 06:31:52 GMT

No, it is not client-side related issue, is not related to your link.

It is problem on Power BI Report Server side, exactly in RSPowerBI.exe process, where is critical memory leak bug.

Re: Performance and memory leak issue - PBIRS Jan 2022

v-henryk-mstf — Fri, 25 Feb 2022 06:32:20 GMT

Hi @josef78 ,

Please follow this document to troubleshoot the problem if it helps.

Monitor report performance in Power BI - Power BI | Microsoft Docs

Looking forward to your feedback.

Best Regards,
Henry

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Power BI RS January 2022 - performance and memory leak issue

d_gosbell — Fri, 25 Feb 2022 06:57:00 GMT

@josef78 it does sound like a possible bug, but I've been monitoring our prod servers all day and I'm not seeing anything similar, but it could be due to a difference in workload. I've setup some perfmon counter logs so that I can track this over a longer period.

I would suggest raising a support ticket with Microsoft support. Issues like this will get resolved much faster if Microsoft can get access to performance counter logs and multiple memory dumps and the support engineers would be able to tell you what they need and provide mechanisms for securely uploading these sorts of things.

Re: Performance and memory leak issue - PBIRS Jan 2022

josef78 — Fri, 25 Feb 2022 16:12:30 GMT

Hi @v-henryk-mstf ,

it is not related to Power BI Desktop application. This bug is related Power BI Report Server, server side processing in RSPowerBI.exe process (which provide interface between client and integrated tabular storage (msmdsrv.exe), or external SSAS service).

Re: Power BI RS January 2022 - performance and memory leak issue

josef78 — Fri, 25 Feb 2022 17:07:13 GMT

Hi @d_gosbell,

thank you. I'm still monitoring and investigating. This problem is definitely in RSPowerBI.exe, where is some memory leak, and based on amount of leaked memory is there also additional cpu utilization. Memory utilization of this process gradually increasing and is dependent on server user load (especially on number of DAX queries passed from client side, thru RSPowerBI.exe to internal SSAS engine, and back. I'm not sure all other dependencies and conditions.

In my case (on production), after restart of service, RSPowerBI.exe process slowly allocate about 10GB RAM during one hour! (typically this process takes about 500MB after whole day in prev version), and after hour, only this process load CPU on server about 20% (typically cpu utilization of this process is near to zero), and this affect overall performance of server. End of day (after 8hours), only this one process takes about 50GB of RAM, and utilize about 80% of CPU, which make whole server extreme slow. Only my workaround is restart whole PBIRS, or kill RSPowerBI.exe process, every hour, to keep usability of PBIRS.

Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?

Re: Power BI RS January 2022 - performance and memory leak issue

d_gosbell — Sat, 26 Feb 2022 02:25:56 GMT

@josef78 wrote:

Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?

We are running the Jan 22 release and we have 2 x 8 core servers with about 96Gb of RAM I think (it's Saturday here now and I'm not logged into the work VPN to double check this). When I was checking in task manager I saw each instance using between about 700Mb and 1.5Gb of RAM, but both instances seemed to go up and down in terms of RAM usage. Although the bulk of our reports are against external SSAS tabular models not data models inside the PBIX reports.

I'm not entirely sure what function the RSPowerBI.exe process performs. But I'm assuming by it's name that it is probably the component that manages the loading and unloading of data models into the internal SSAS engine and so it's probably also responsible for marshalling DAX queries to the correct database and it probably does the caching of the datasets for visuals.

So if I had to guess, I would say that there is probably some issue with the "visual cache" (this is my terminology) It can be a problem with any of sort of caching strategy, that if the cache grows too large (either because of a bug or because of the way it is configured). Then it can take longer to search through the cache than it would have to just bypass the cache and go to the original source. This sort of issue would affect all queries, not just those that are already cached since the service will have to search through the cache index any time a query is run to see if there is an entry in the cache or not. Based on your description of your symptoms this is one possible explanation, but there could be others.

But this is all speculation on my part, but this is what made me suggest raising a support ticket. I think this issue is probably beyond the depth of something we can fix here although it does not hurt to ask here since others may have experienced this also.

I have setup some perfmon data collector sets to capture some of the key performance counters and the CPU useage and working set size of the RSPowerBI.exe process on both our nodes every 5 minutes. So I will be able to check these next week and let you know if I'm seeing a gradual leak on our servers.

Re: Performance and memory leak issue - PBIRS Jan 2022

josef78 — Wed, 02 Mar 2022 20:48:48 GMT

There is new build of Jan22 version avaliable (1.13.8086.22725), but is still does not solve this problem.

Re: Power BI RS January 2022 - performance and memory leak issue

josef78 — Wed, 02 Mar 2022 20:54:07 GMT

Hi @d_gosbell ,

Thank you so much. We have a similar configuration (16 virtual CPUs, 196 GB RAM on front-end servers). The function of the RSPowerBI process (not sure) is probably for passing and marshaling DAX queries from the client to the correct SSAS database (integrated or connected live), nothing performance intensive (compared to the msmdsrv (SSAS) process).

After some time of monitoring, I can tell you more where the problem is, but I still don't know exactly what it all depends on, and what I can do.

The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly (you can try monitor Handle count under this process). In the Sep21 version, this process uses an almost constant number of handles, over time. After the upgrade to Jan22, the number of event handles in this process grows rapidly, the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to excessive CPU utilization.
But I don't know if this behavior depends on anything else, such as available resources, OS version, or OS configuration.

In my case, it affects me in both test and production environments, and I also create a clean installation of a new server farm, with the same problem. But the problem is difficult to identify because is showing only under heavy load and after some time.

Re: Power BI RS January 2022 - performance and memory leak issue

josef78 — Wed, 02 Mar 2022 21:09:05 GMT

There is Performance Monitor record, for little over one hour of daily real load. Is only for RSPowerBI process (of Jan22 version), and procesor utilization, memory allocation and handles count. At 9am and 10am (every hour) is process killed and restarted. During one hour, there are gradually allocated almost 400000 handles (on my Sep21 test is constantly under 1500), memory is gradually allocated to 15GB (on my Sep21 test is near to constantly under 1GB), and CPU utilization only of this process grow to average 50%.

PerfMon

Re: Power BI RS January 2022 - performance and memory leak issue

d_gosbell — Wed, 02 Mar 2022 22:39:49 GMT

You should definitely log a support ticket for this.

If you license Report Server through a Premium Capacity there is a link here https://powerbi.microsoft.com/en-us/support/pro/ where your tenant admin can log a ticket. If you license Report Server through SQL Server Enterprise with SA then your IT deptment should have a way of logging tickets or you might have a Microsoft Account manager which you can escalate this through.

When you log a ticket a level 1 support engineer will contact you and ask for logs/traces etc. Given what you have found already I don't think it will take them long to escalate to the product team. There is a small chance that maybe there is some undocumented configuration setting that will revert to the Sep 21 behaviour, but otherwise giving the team detailed, real world evidence from a production environment linked to an external customer will give this a much higher priority than if someone internal logs this issue.

Re: Power BI RS January 2022 - performance and memory leak issue

josef78 — Fri, 04 Mar 2022 11:41:59 GMT

Hi @d_gosbell,

ok, I will try rise ticket (we have MSSQL SA licence), but I'm afraid it will be a long process.

Thank you

Re: Performance and memory leak issue - PBIRS Jan 2022

josef78 — Fri, 04 Mar 2022 13:07:06 GMT

Additionally information, check and repro steps:

I installed clean Power BI Report Server, with default configuration, with September 2021 version. There I saved two small PBI reports (aprox. 50MB) with aprox ten visuals on page. From client computer I opened these reports and setup 5sec refresh interval (using Edge extension) to simulate constant load. On server I setup PerfMon with Procesor utilization, Handles and Private bytes (allocated memory) for Process RSPowerBI. There is result 30minutes of run after restart service:

Sep21

After this test I upgraded this server to latest January 2022 (1.13.8086.22725) version, restarted, and run same test (exactly same reports and same simulated load). And there is result:

Jan22

The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly. In the Sep21 version, this process uses an almost constant number of handles (under 1500), over whole period. After the upgrade to Jan22, the number of event handles in this process grows rapidly (over 60000 during half hour), the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to higher CPU utilization.

Re: Performance and memory leak issue - PBIRS Jan 2022

Anonymous — Sun, 06 Mar 2022 07:51:41 GMT

@josef78 I'm experiencing the same problem, have you escalated it to Microsoft? I've had to reboot the server 3 times in a week as the server has crashed 😔

Re: Performance and memory leak issue - PBIRS Jan 2022

d_gosbell — Mon, 07 Mar 2022 00:23:13 GMT

@Anonymous wrote:

I'm experiencing the same problem, have you escalated it to Microsoft? I've had to reboot the server 3 times in a week as the server has crashed 😔

You should really escalate to Microsoft also. You should not experience crashes. If it is the same issue the more reports then if there are multiple reports from different customers it will increase the priority of this issue. And if it's not the same issue you don't want to wait for this issue to be fixed only to find it does not help your situation.

Re: Performance and memory leak issue - PBIRS Jan 2022

Anonymous — Mon, 07 Mar 2022 10:08:56 GMT

@d_gosbell I've raised it with Microsoft this morning 🙂

Re: Performance and memory leak issue - PBIRS Jan 2022

josef78 — Tue, 08 Mar 2022 14:42:16 GMT

@Anonymous Thank you for ticket. I will rise ticket probably tomorrow (i'm busy, sorry 😞 ), if it increase priority.

@d_gosbell please, if possible and if it increase priority, I will be grateful if you also create a ticket. Thank you so much

Re: Power BI RS January 2022 - performance and memory leak issue

Steffe — Mon, 14 Mar 2022 15:55:57 GMT

We have the same problem after upgrading from September 2021 to January 2022 version.

Looking at monitoring boards for Memory and CPU we can se big differences.

Memory now increasing during the day, until it "swaps all out" and start over. Before upgrading to January editition it wasn´t beahaving like this. Also much heavier CPU utilization compared to when running on September edition of PBIRS.

Any news on this?