Performance and memory leak issue - PBIRS Jan 2022

[josef78]

Hello,

After upgrading our production environment to the Power BI Report Server version of January 2022 (1.13.8054.40631), we have significant performance issues.

At real user load (including a slight evening load), we have a significant slowdown in report queries (approximately by 50%), and much higher CPU usage. This happens a few hours after the last restart of services, and worsens over time.

The problem is only on front-end servers (we have scale-out deployment). And the problem is probably somethere in the RSPowerBI.exe process, which higlly utilizes the CPU and also consumes a large amount of RAM (more than the main msmdsrv.exe process). There is nothing unusual in RSPowerBI * .log.

Does anyone else have these problems?

Is there a solution?

[Steffe]

Any news on this issue? Seems like there are many organization facing the same issue.

Are Microsoft working on it?

[josef78]

Hello, I raised issue ticket too. I have the answer that MS has identified an bug. Now I'm waiting for a bugfix or a new build. I hope for a new build or that it will be resolved in a may release.

[Steffe]

Thanks @josef78 

This sounds promesing 🙂

[Andrino]

It looks like Microsoft doesn't do anything about this issue. Author of this topic posted ticket to Microsoft Support. I'll try to post ticket too if there will be no bug fixes at nearest time.

[file]

To bad, hope they fix this in the May-release then.

[file]

We are also experiencing high memory usage after upgrading to January 2022, even after the latest update as well. The memory increases during the day until the service crashes and then it repeats. We even changed the settings for ModelCleanupCycleMinutes and ModelExpirationMinutes but it does not solve the problem.

We have a mix of live/import reports and SSRS-reports.

[josef78]

Hello,

my current workarround is periodically restart RSPowerBI process (not whole PowerBIReportServer service, because it is time consuming and also need reload all models in to memory).

I'm killing RSPowerBI process every hour (with scheduled powershell script Stop-Process -Name RSPowerBI -Force), PowerBI service immediately start new clean process without any outage.

[Andrino]

If there is an active Power BI report data refresh process then with killing RSPowerBI.exe this process will also be killed and stucked in SubscriptionHistory table of PBIRS database. To prevent this I've wrapped this command into the script with checking of existence of active PBI data refresh processes:

$dataSource = "YourSqlServerDBEngineInstanceWherePBIRSDatabaseIsLocated";
$database = "master";
$sql = "
    SELECT [SH].[SubscriptionHistoryID]
    FROM [PBIRS].[dbo].[SubscriptionHistory] [SH] WITH(NOLOCK)
    WHERE [SH].[Status] = 1
";
$auth = "Integrated Security=SSPI;"
$connectionString = "Provider=sqloledb; " + "Data Source=$dataSource; " + "Initial Catalog=$database; " + "$auth; ";
$connection = New-Object System.Data.OleDb.OleDbConnection $connectionString;
$command = New-Object System.Data.OleDb.OleDbCommand $sql, $connection;
$connection.Open();
$adapter = New-Object System.Data.OleDb.OleDbDataAdapter $command;
$dataset = New-Object System.Data.DataSet;
[void] $adapter.Fill($dataSet);
$connection.Close();
$rowsCount=($dataset.Tables | Select-Object -Expand Rows).Count;
if ($rowsCount -eq 0) {
    Stop-Process -Name RSPowerBI -Force;
};

[josef78]

Yes, thank you, it is great improvement.

If is Power BI RS deployed on single server (or server is shared for frontend and background task), it need this check.

If is Power BI RS in scale-out deployment, and there are dedicated servers for frontend and dedicated servers for background processes, this issue is related only to frontend PBIRS, where is need to killing RSPowerBI process (in this case is this Subscription check not necessary).

[Andrino]

I'm also experiencing this problem. The key point of this behavior is leaping high CPU usage. RAM and disk I/O performance are not relevant to this problem (we've tried to increase memory limits of DB Engine and rsconfig and move database to fast nvme disk with no effects). When PBIRS is recently restarted, the problem is not observed. But after some time we can see high CPU usage by rspowerbi.exe process with leaping peaks of CPU usage (up to 100%) and at the same time noticable degradation of portal's and reports response.

[Anonymous]

@josef78 I'm experiencing the same problem, have you escalated it to Microsoft? I've had to reboot the server 3 times in a week as the server has crashed 😔 

[d_gosbell]

---

@Anonymous wrote:

 I'm experiencing the same problem, have you escalated it to Microsoft? I've had to reboot the server 3 times in a week as the server has crashed 😔 

---

You should really escalate to Microsoft also. You should not experience crashes. If it is the same issue the more reports then if there are multiple reports from different customers it will increase the priority of this issue. And if it's not the same issue you don't want to wait for this issue to be fixed only to find it does not help your situation.

[Anonymous]

Microsoft are archiving my ticket because they can't find anything convincing in the logs are are suggesting repairing the OS and the PBI Report Server.... At the moment we have a 4am restart in place on the server to stop it crashing which is limiting how often the issue occurs. Please log tickets with Microsoft if you haven't already, otherwise they won't know there's a bug.

[Anonymous]

@d_gosbell I've raised it with Microsoft this morning 🙂

[josef78]

@Anonymous Thank you for ticket. I will rise ticket probably tomorrow (i'm busy, sorry 😞 ), if it increase priority. 

@d_gosbell please, if possible and if it increase priority, I will be grateful if you also create a ticket. Thank you so much

[josef78]

Additionally information, check and repro steps:

I installed clean Power BI Report Server, with default configuration, with September 2021 version. There I saved two small PBI reports (aprox. 50MB) with aprox ten visuals on page. From client computer I opened these reports and setup 5sec refresh interval (using Edge extension) to simulate constant load. On server I setup PerfMon with Procesor utilization, Handles and Private bytes (allocated memory) for Process RSPowerBI. There is result 30minutes of run after restart service:

Sep21

After this test I upgraded this server to latest January 2022 (1.13.8086.22725) version, restarted, and run same test (exactly same reports and same simulated load). And there is result:

Jan22

The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly. In the Sep21 version, this process uses an almost constant number of handles (under 1500), over whole period. After the upgrade to Jan22, the number of event handles in this process grows rapidly (over 60000 during half hour), the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to higher CPU utilization.

[josef78]

There is new build of Jan22 version avaliable (1.13.8086.22725), but is still does not solve this problem.

[v-henryk-mstf]

Hi @josef78 ,

Please follow this document to troubleshoot the problem if it helps.

Monitor report performance in Power BI - Power BI | Microsoft Docs

Looking forward to your feedback.

Best Regards,
Henry

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[josef78]

Hi @v-henryk-mstf ,

it is not related to Power BI Desktop application. This bug is related Power BI Report Server, server side processing in RSPowerBI.exe process (which provide interface between client and integrated tabular storage (msmdsrv.exe), or external SSAS service).

[v-henryk-mstf]

Hi @josef78 ,

According to your description, for this kind of problem, it may be a problem related to iFrame in IE, try to switch to another browser, such as Edge or Chrome.

Refer to the following similar solutions.

Embedded PowerBI performance and memory leak - Page 2 - Microsoft Power BI Community

If the problem is still not resolved, please point it out. Looking forward to your reply.

Best Regards,
Henry

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[josef78]

No, it is not client-side related issue, is not related to your link. 

It is problem on Power BI Report Server side, exactly in RSPowerBI.exe process, where is critical memory leak bug.