Forum Discussion
Performance and memory leak issue - PBIRS Jan 2022
Hello,
After upgrading our production environment to the Power BI Report Server version of January 2022 (1.13.8054.40631), we have significant performance issues.
At real user load (including a slight evening load), we have a significant slowdown in report queries (approximately by 50%), and much higher CPU usage. This happens a few hours after the last restart of services, and worsens over time.
The problem is only on front-end servers (we have scale-out deployment). And the problem is probably somethere in the RSPowerBI.exe process, which higlly utilizes the CPU and also consumes a large amount of RAM (more than the main msmdsrv.exe process). There is nothing unusual in RSPowerBI * .log.
Does anyone else have these problems?
Is there a solution?
I wanted to reply to let you know the May release is now live with a fix for the issue reported on this thread.
51 Replies
Hello, I'm also still waiting for a new fixed version. I'm checking out the status and the new ETA, I will let know. Currently I must reset RSPowerBI every 30min which causes other problems. But without restarts PBIRS is not usable.
Terrible.
I work for an international company, and I am responsible for Power BI in two countries with 10,000 employees, and with 3,000 active Power BI RS consumers. And everyone is upset about these problems.I have been solving this problem since the end of February, and I think it is a critical problem (in the case of large deployments). I entered the ticket in March, and the first reaction was that it was a "this is a design issue", with the ticket being archived and not addressed. After the escalation, they confirmed that it is a "bug", but the fix will be in the next version. I'm waiting, we're all waiting more than three months to fix a critical bug!
And this is not the only problem in PBIRS, but it is one critical issue.
I think MS's approach to enterprises is terrible (in the case of Power BI), it's not just about bugs, but also about missing features. While Cloud Power BI has tons of new features every month. On-prem Power BI RS released once every 4 months does not bring any new essential features. The on-prem Power BI RS lacks many key features for enterprises, compared to the Power BI cloud, but also compared to competing solutions.
I know that 1000 individual Power BI cloud users are more important (and more publicly visible) than one company with 10,000 users. But that's no reason to ignore the enterprise segment.
I understand the "Cloud-first" strategy, but currently it's more like "Cloud only, forget on-premises".
Also, Gartner, in their latest Magic quadrant for BI platforms, criticizes Microsoft for "... Microsoft's on-premises offering lacks significant functional capabilities ..."
What do you think about it?
Yes, that's right. Specifically about unsupported features. I think it's deliberate approach because some not implemented features are even simpler to make working for on-premise than for cloud.
Hello, yesterday I got information from the support team that "due to some other reasons not released yet, and it will release in a couple of days", but in a few hours the PBIRS May 2022 was releassed...
Change log is not updated yet, I hope this problem is solved. I'm going to install and try it.
Hello, josef78 !
I'm also a PBI global admin and I'm facing some slow environment performance.
Could you please share with me your Workloads Tenant configuration?
I'm trying to find a way to improve my PBI environment.
Thanks in advance.
Additionally information, check and repro steps:
I installed clean Power BI Report Server, with default configuration, with September 2021 version. There I saved two small PBI reports (aprox. 50MB) with aprox ten visuals on page. From client computer I opened these reports and setup 5sec refresh interval (using Edge extension) to simulate constant load. On server I setup PerfMon with Procesor utilization, Handles and Private bytes (allocated memory) for Process RSPowerBI. There is result 30minutes of run after restart service:
Sep21
After this test I upgraded this server to latest January 2022 (1.13.8086.22725) version, restarted, and run same test (exactly same reports and same simulated load). And there is result:
Jan22
The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly. In the Sep21 version, this process uses an almost constant number of handles (under 1500), over whole period. After the upgrade to Jan22, the number of event handles in this process grows rapidly (over 60000 during half hour), the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to higher CPU utilization.Any news on this issue? Seems like there are many organization facing the same issue.
Are Microsoft working on it?
Hello, I raised issue ticket too. I have the answer that MS has identified an bug. Now I'm waiting for a bugfix or a new build. I hope for a new build or that it will be resolved in a may release.
It looks like Microsoft doesn't do anything about this issue. Author of this topic posted ticket to Microsoft Support. I'll try to post ticket too if there will be no bug fixes at nearest time.
Guys, how are you? Any update on this problem, the May 2022 version comes out today, I migrated the company's servers yesterday to the January 2022 version and started to have this same problem...
We are also experiencing high memory usage after upgrading to January 2022, even after the latest update as well. The memory increases during the day until the service crashes and then it repeats. We even changed the settings for ModelCleanupCycleMinutes and ModelExpirationMinutes but it does not solve the problem.
We have a mix of live/import reports and SSRS-reports.
Hello,
my current workarround is periodically restart RSPowerBI process (not whole PowerBIReportServer service, because it is time consuming and also need reload all models in to memory).
I'm killing RSPowerBI process every hour (with scheduled powershell script Stop-Process -Name RSPowerBI -Force), PowerBI service immediately start new clean process without any outage.
If there is an active Power BI report data refresh process then with killing RSPowerBI.exe this process will also be killed and stucked in SubscriptionHistory table of PBIRS database. To prevent this I've wrapped this command into the script with checking of existence of active PBI data refresh processes:
$dataSource = "YourSqlServerDBEngineInstanceWherePBIRSDatabaseIsLocated";
$database = "master";
$sql = "
SELECT [SH].[SubscriptionHistoryID]
FROM [PBIRS].[dbo].[SubscriptionHistory] [SH] WITH(NOLOCK)
WHERE [SH].[Status] = 1
";
$auth = "Integrated Security=SSPI;"
$connectionString = "Provider=sqloledb; " + "Data Source=$dataSource; " + "Initial Catalog=$database; " + "$auth; ";
$connection = New-Object System.Data.OleDb.OleDbConnection $connectionString;
$command = New-Object System.Data.OleDb.OleDbCommand $sql, $connection;
$connection.Open();
$adapter = New-Object System.Data.OleDb.OleDbDataAdapter $command;
$dataset = New-Object System.Data.DataSet;
[void] $adapter.Fill($dataSet);
$connection.Close();
$rowsCount=($dataset.Tables | Select-Object -Expand Rows).Count;
if ($rowsCount -eq 0) {
Stop-Process -Name RSPowerBI -Force;
};
Hello Josef, any answer form MS? We have the same problem too. We can make a call (in CZ) if you want 🙂
Hello, sorry for delay (lot of work).
Yes, we can make a call, if you want, please send me private message.
Last Update from MS ticket:
I have information from MS Support, this issue is forwarded to the Product Group team, and they have already identified this issue and confirmed it is a bug in this version and they are working on it to fix this in next version.
The PBI product team has fixed this issue and currently working on porting the fix to the May 2022 test build. Depending on how long that takes, it may be too close to the May 2022 release (current ETA is May 24).
I hope that this issue will be definitively resolved in May 2022 release.
Guys, how are you?
Any update on this problem, the May 2022 version comes out today, I migrated the company's servers yesterday to the January 2022 version and started to have this same problem...
I wanted to reply to let you know the May release is now live with a fix for the issue reported on this thread.
Additionall information.
I tested on more servers on another separated environment. There is the same problem, and I think it's a big problem for any large deployment.The root cause is probably a memory leak bug in RSPowerBI.exe. It seems that the problem is directly proportional to the number of queries going through power bi reports.
In my environment after upgrade to Jan/22, RSPowerBI.exe slowly allocates memory, within 8 hours and about 60 thousand queries, allocates about 50GB of RAM, which also leads to slowdown (about 3x slower response), and unstable state.
josef78 it does sound like a possible bug, but I've been monitoring our prod servers all day and I'm not seeing anything similar, but it could be due to a difference in workload. I've setup some perfmon counter logs so that I can track this over a longer period.
I would suggest raising a support ticket with Microsoft support. Issues like this will get resolved much faster if Microsoft can get access to performance counter logs and multiple memory dumps and the support engineers would be able to tell you what they need and provide mechanisms for securely uploading these sorts of things.
Hi d_gosbell,
thank you. I'm still monitoring and investigating. This problem is definitely in RSPowerBI.exe, where is some memory leak, and based on amount of leaked memory is there also additional cpu utilization. Memory utilization of this process gradually increasing and is dependent on server user load (especially on number of DAX queries passed from client side, thru RSPowerBI.exe to internal SSAS engine, and back. I'm not sure all other dependencies and conditions.
In my case (on production), after restart of service, RSPowerBI.exe process slowly allocate about 10GB RAM during one hour! (typically this process takes about 500MB after whole day in prev version), and after hour, only this process load CPU on server about 20% (typically cpu utilization of this process is near to zero), and this affect overall performance of server. End of day (after 8hours), only this one process takes about 50GB of RAM, and utilize about 80% of CPU, which make whole server extreme slow. Only my workaround is restart whole PBIRS, or kill RSPowerBI.exe process, every hour, to keep usability of PBIRS.
Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?
