It seems like we have a memory leak in the termsvcs process of our RDS farm which occasionally occurs on a random machine in our farm. We have 10 session hosts on Windows Server 2012 R2 with current patch level who all serve around 20 to 30 users.
In this screenshot you can see the CPU and RAM load. Notice the steady drop in memory within the last 24 hours (http:// removed due to lack of reputation): i.imgur.com/dz11fPz.png
Here you can see the process in question with a very high value in Private Bytes: http://i.imgur.com/Rh9SfzI.png - and in comparison the same process on a healthy session host with the same configuration and roughly the same amount of users connected: http://i.imgur.com/FCMVclD.png
Notice the difference the Private Bytes.
Does anyone else experience something similar?
It seemingly started a week ago. So maybe related to a patch day.
Related
Tried posting on Gitlab's forum and had no luck so I thought I'd try here.
We’ve been using Gitlab10 CE for a few months now. We are a pretty small shop with only 5 developers so our instance of gitlab is busy but not crazy by any stretch of the imagination, yet we are constantly running into memory problems. It is a virtual machine running on Ubuntu 16.04. I initially began with the recommended 1 core, and 4GB of memory, and we were constantly being alerted about memory and CPU issues. I upped the specs to 2 cores, and 8GB of memory. Same issue. I’ve now pushed the box to 8 cores and 32GB of CPU and I am still constantly being alerted about memory issues (although the CPU has died down quite a bit). As of the time of this message, we’ve received 20 memory alerts in the last 5 hours. These things are even coming in through the night hours when we have no one even touching the system.
When I run HTOP, there are 28 processes called sidekiq 5.0.4 gitlab-rails [0 of 25 busy] that claim to be costing 2% of our overall memory each. That is over 16GB worth! Under that there’s a whole host of unicorn workers costing 1.8% of our overall memory each.
We’re pretty new to using gitlab so there could easily be something I’m just missing. Any advice on how to throttle the number of processes for each of these or throttle git’s overall memory consumption would be awesome. Thanks!
I'd bet you are seeing threads, not processes in htop. Press Shift-H to view processes. Those threads are all sharing the same 2% of memory.
Make sure you are keeping up to date with GitLab versions, they fix bugs and optimize their code all the time.
We have the following situation at work: Currently we have problems with obtaining reports, which are generated through an application developed in .NET and is mounted on a SharePoint 2010 site. We have done tests generating reports of short time periods (per month) and it works perfect, the problem arises only when trying to obtain the report per year (due to the large number of records).
There are two servers, production and QA, the problem only occurs in the production environment (obviously it is the one with the highest workload).
We were reviewing the configurations of both servers and found the following difference in IIS:
PROD
Connection Time-out (seconds): 120
QA
Connection Time-out (seconds): 180
We would like to increase this time in the production environment, however we would like to know if there is any risk of impacting some other process / application.
Any recommendation?
Thank you
IIS Connection timeouts help reduce the amount of memory resources that are consumed by idle connections. Time-out settings also allow you to specify how long server resources are allocated to specific tasks or clients.
When increasing the Time Out value, it will be good for large number records process, as it will need more time to connect to the IIS.
We are running cassandra version 2.0.9 in production. It's a 4 node cluster. For the past few days we are experiencing a high spike in CPU Utilisation. You may see in the picture below.
This is the jconsole output.
When we looked into the threads which are eating a lot of CPU we came across Native Transport request these are eating a lot of CPU (Like 12%) which is huge.
Thread stack trace.
Threads info.
Thread CPU%.
What can the problem be how should we go about debugging it?
Why are majority of NTR request stuck on BCrypt.java? Is this the problem?
The cluster was behaving normally a few days back but now out of 4 nodes 3 are always on high CPU Utilisation.
You have authentication enabled which stores bcrypted hash, not the password. So each request needs to to be checked. This will end up being a CPU issue if you are continually creating new connections instead of reusing an authenticated session. Sessions are long lived objects and should be by default (https://github.com/datastax/php-driver/tree/master/features#persistent-sessions) but if using CGI or something constantly creating new processes you will still have issues. Maybe try php-fpm ?
I have an application in the Production environment which is Windows Server 2012/IIS 8 and is load balanced.
Recently out of nowhere, the website app pool suddenly started gettig disabled. The System Windows Logs logged the following error message by the Resource-Exhaustion-Detector ...
Application Pool 'x' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: w3wp.exe (6604) consumed 5080641536 bytes, w3wp.exe (1572) consumed 477335552 bytes, and w3wp.exe (352) consumed 431423488 bytes.
Anyone got any idea how I figure out what is happening? We've never come across this issue before and the application has been running for a good couple of years.
Also, this isn't something that happens regularly but instead seems to happen one every day or so, and even that is at a random time. The Virtual Memory was initially 4GB but because of the issue above, we increased it to 8GB. Recently it spiked at using about 6.8GB out of 8GB, which it has no reason to do so.
Any help would be really appreciated!
The answer is easy here, obviously and certainly you have two issues here
1- You have a serious bug in your process/code that happens intermittently "you need to debug it to find how/when that happens" or at least run a ProcDump
such that you keep it listening on the server on the process W3WP till an exception happens and then analyze this dump to find where the code get stuck and consume that memory/otherwise just debug the code and see what changes were made in last few months "not days"
2- the application get stopped because you have configured/it is configured by default to get disabled break after a certain number of failure repeats, and that's a normal behavior but the main issue as I said is not the application pool itself, its inside the process
please let me know if you need a further explanation or help on this matter
I am researching into the IIS Application Initialization module and from what I can see, when using the AlwaysRunning option for Start Mode setting for the application pool, basically it starts a new worker process that will always run even if there isn't any requests. When applying this option it starts the process automatically.
My concern is memory management and CPU usage, specifically how is this handled since the process always runs.
How can I compare this to setting the Start Mode to OnDemand and increase the Idle Time minutes to couple of days? That way, I guess, the process will run in idle mode for x days before it's terminated, and reinitialized on the next request and keep running for a couple of days. If I set the minutes to let's say 1.5 days, someone is bound to use the application at least once a day, so it will persist the process runtime and it will never be terminated.
Can anyone share experience regarding this topic?
Thanks
I have multisite application that runs few sites under separate app pools. All are set OnDemand for Start Mode and IdleTime for 1740 minutes, also I use Page Output Cache from app with different times for different page types. There is also NHibernate behind scene and DB is MySql.
The most active site have more than 100k visits per day and almost never is idle. When it starts if I recycle, need 30 seconds to 2 minutes to became full operable depending on requests at the moment and CPU usage is going from 40% to 70%. After the site is up CPU usage is very low (0-4%) if there are no new entries in DB and memory usage is around 3GB when all is cached. Sometimes CPU is going to 20% if at that moment are new request (for not cached content) and there is new entry saving.
Also Page Output Cache works on First Come First Served base so maybe this can also cause little problem while caching is done - user must wait, little more CPU to do the caching.
The most biggest problem in my case is using NHibernate and MySql but Page Output Cache resolved the problem for me when I decided to cache the page modules and content. I realize that is better application to starve for memory then for CPU.
3.5k visitors at one moment when everything is cached gave to me same memory usage (3GB) and CPU (server overall) around 40%
Other sites are using around 1-1.5GB memory and CPU never more then 20% at start.
The application with same settings for app pool and using MSSQL with EF I can't even notice that run on server. It is used by 10-60 users in minute there is not much content except embedding codes and it use 1-5% CPU and never more than 8MB memory. On recycle it is up for less then 10 seconds.
With this my experience I can tell you that all depends on what application serves and how it works :) and how much content do you have.
If you use OnDemand with long IdleTime it will be same as AlwaysStart and process is not used at that moment. If you use OnDemand with short IdleTime more often you will need CPU to start the process.