Forum Discussion
Memorable MemberPerformance and memory leak issue - PBIRS Jan 2022
I wanted to reply to let you know the May release is now live with a fix for the issue reported on this thread.
Super Userjosef78 it does sound like a possible bug, but I've been monitoring our prod servers all day and I'm not seeing anything similar, but it could be due to a difference in workload. I've setup some perfmon counter logs so that I can track this over a longer period.
I would suggest raising a support ticket with Microsoft support. Issues like this will get resolved much faster if Microsoft can get access to performance counter logs and multiple memory dumps and the support engineers would be able to tell you what they need and provide mechanisms for securely uploading these sorts of things.
Memorable MemberHi d_gosbell,
thank you. I'm still monitoring and investigating. This problem is definitely in RSPowerBI.exe, where is some memory leak, and based on amount of leaked memory is there also additional cpu utilization. Memory utilization of this process gradually increasing and is dependent on server user load (especially on number of DAX queries passed from client side, thru RSPowerBI.exe to internal SSAS engine, and back. I'm not sure all other dependencies and conditions.
In my case (on production), after restart of service, RSPowerBI.exe process slowly allocate about 10GB RAM during one hour! (typically this process takes about 500MB after whole day in prev version), and after hour, only this process load CPU on server about 20% (typically cpu utilization of this process is near to zero), and this affect overall performance of server. End of day (after 8hours), only this one process takes about 50GB of RAM, and utilize about 80% of CPU, which make whole server extreme slow. Only my workaround is restart whole PBIRS, or kill RSPowerBI.exe process, every hour, to keep usability of PBIRS.
Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?
- d_gosbell
josef78 wrote:
Please, which version you are currently running? And how many memory and CPU your RSPowerBI.exe utilize?
We are running the Jan 22 release and we have 2 x 8 core servers with about 96Gb of RAM I think (it's Saturday here now and I'm not logged into the work VPN to double check this). When I was checking in task manager I saw each instance using between about 700Mb and 1.5Gb of RAM, but both instances seemed to go up and down in terms of RAM usage. Although the bulk of our reports are against external SSAS tabular models not data models inside the PBIX reports.
I'm not entirely sure what function the RSPowerBI.exe process performs. But I'm assuming by it's name that it is probably the component that manages the loading and unloading of data models into the internal SSAS engine and so it's probably also responsible for marshalling DAX queries to the correct database and it probably does the caching of the datasets for visuals.
So if I had to guess, I would say that there is probably some issue with the "visual cache" (this is my terminology) It can be a problem with any of sort of caching strategy, that if the cache grows too large (either because of a bug or because of the way it is configured). Then it can take longer to search through the cache than it would have to just bypass the cache and go to the original source. This sort of issue would affect all queries, not just those that are already cached since the service will have to search through the cache index any time a query is run to see if there is an entry in the cache or not. Based on your description of your symptoms this is one possible explanation, but there could be others.
But this is all speculation on my part, but this is what made me suggest raising a support ticket. I think this issue is probably beyond the depth of something we can fix here although it does not hurt to ask here since others may have experienced this also.
I have setup some perfmon data collector sets to capture some of the key performance counters and the CPU useage and working set size of the RSPowerBI.exe process on both our nodes every 5 minutes. So I will be able to check these next week and let you know if I'm seeing a gradual leak on our servers.
- josef78
Hi d_gosbell ,
Thank you so much. We have a similar configuration (16 virtual CPUs, 196 GB RAM on front-end servers). The function of the RSPowerBI process (not sure) is probably for passing and marshaling DAX queries from the client to the correct SSAS database (integrated or connected live), nothing performance intensive (compared to the msmdsrv (SSAS) process).
After some time of monitoring, I can tell you more where the problem is, but I still don't know exactly what it all depends on, and what I can do.
The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly (you can try monitor Handle count under this process). In the Sep21 version, this process uses an almost constant number of handles, over time. After the upgrade to Jan22, the number of event handles in this process grows rapidly, the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to excessive CPU utilization.
But I don't know if this behavior depends on anything else, such as available resources, OS version, or OS configuration.In my case, it affects me in both test and production environments, and I also create a clean installation of a new server farm, with the same problem. But the problem is difficult to identify because is showing only under heavy load and after some time.
- josef78
There is Performance Monitor record, for little over one hour of daily real load. Is only for RSPowerBI process (of Jan22 version), and procesor utilization, memory allocation and handles count. At 9am and 10am (every hour) is process killed and restarted. During one hour, there are gradually allocated almost 400000 handles (on my Sep21 test is constantly under 1500), memory is gradually allocated to 15GB (on my Sep21 test is near to constantly under 1GB), and CPU utilization only of this process grow to average 50%.
PerfMon
Any updates on this? We are having similar issue and consistent on all of our Jan 2022 upgraded Power BI servers.