I am facing a CPU overload issue due to a nodejs application that I am running in a remote Ubuntu 16.04 LTS virtual machine. I am using PM2 to schedule my nodejs application as a service.
Initially when the nodejs application is launched, the CPU load remain quite low; about 30% at most. Then slowly I find the CPU load going up till it gets to 100%. This nodejs application is polling a stock website for new information on a stock and then does some calculations and then repeats after 5 minutes. I dont see how its causing this overload on the CPU.
I notice that my 1 nodejs application shows up as 6 different processes in HTOP command. Not sure if this normal or how to fix this. Any help would be highly appreciated.
Thanks
Regards,
Adeel
Thanks, Gerard. Your reply helped solve the problem. Turns out guardian.js was not exiting and just opening up new processes till it overloaded the system.
Related
I have a problem with increasing kernel cpu usage on a web server I am running. On a 6 core cpu the kernel usage increases from 5 to 50% in some 8 hours.
I have noticed it takes less time when there are more active users on the site and I don't have this problem in dev, therefore I don't have any code that can reproduce the problem. I am hoping for some advice how to troubleshoot this though, what should I investigate to figure out what the problem is?
"pm2 restart" will take the cpu usage down so this is what I need to do every 8 hours or so. I have also noticed increasing cpu usage of systemd-resolved up to some 50% in 8 hours but restarting it with "systemctl restart systemd-resolved" will not help.
I am running it on ubuntu 20.04, node v12.19.0, next 9.5.3, express, express-session, express-socket.io.session, mongodb etc. I have had this problem even on older versions of all this though.
I try to host my product to AWS EC2 service and I faced with a problem.
My service is a Node.js app. It do some CPU intensive initialization after start. It stucks after about a 30 sec of work any time I started it. I added detailed logs of the app execution and according them looks like app is totally hung on. However CPU usage is 100%.
I've tried to run the app on many different types of instances. The last try is running on c5n.xlarge instance. OS is Ubuntu 18.04. In all cases my app is stuck. Meanwhile, my app perfectly works on Heroku Free and Hobby dynos (https://www.heroku.com/pricing) and on my local PC with Core i5-3450 CPU.
What the problem might be?
The problem is not linked to AWS EC2. The reason was usage an unstable version of module under the concrete Node.js version.
We are renting a shared hosting server, and we run our nodejs/express app on it. The problem is that we are limited in the number of processes allowed (30), and nodejs spawns some processes from time to time. When we reach 30 processes, the server stops responding and we have to reboot it.
We tried to run our app using pm2, but it created too much processes. So we landed on "forever", which seems to create less processes, but the issue is still occuring. We're not experts in nodejs so we have no idea why this is happening, if anyone knows it would be a life saver !
I'm running an API server using NodeJS 6.10.3 LTS on Ubuntu 14.04 (trusty). I've noticed that my API server tops out at ~600 reqs/min running on a c4.large EC2 instance. By tops out I mean, I see the CPU go uptil 100% Note, I know that I'm not fully utilizing the instance by using the cluster module, but that's ok for now.
I took a .cpuprofile dump of my API server for 10 seconds, and noticed that every second, for ~300ms, the profiler shows my NodeJS code is sitting (idle).
Does anyone know what that (idle) implies? Is it a GC issue? Or is it a internal (to V8) lock that I'm triggering? Any help or pointers to tools to help debug this would be nice. I'm working on anonymizing some of stack traces in the cpuprofile so I can share.
The packages I'm using are ExpressJS 4, Couchbase NodeJS SDK, Socket.IO mainly. The codepaths are mainly reading requests, and pushing to Couchbase. And finally querying couchbase via Views API, and pushing some aggregated data on a Socket.IO channel. So all pretty I/O async friendly stuff. I've made sure that I'm not calling any synchronous functions. There are no patterns of function calls before the (idle) in the cpu profile.
It could also just be I/O wait, meaning none of the sockets have data ready to read yet and so the time is spent idle. If you are using a load testing library you should check that the requests are evenly distributed within a second.
Take a look at https://www.npmjs.com/package/gc-stats to check GC data. There are flags to increase heap space, and to change when GC runs, if the problem turns out to be GC related.
I have an application in the Production environment which is Windows Server 2012/IIS 8 and is load balanced.
Recently out of nowhere, the website app pool suddenly started gettig disabled. The System Windows Logs logged the following error message by the Resource-Exhaustion-Detector ...
Application Pool 'x' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: w3wp.exe (6604) consumed 5080641536 bytes, w3wp.exe (1572) consumed 477335552 bytes, and w3wp.exe (352) consumed 431423488 bytes.
Anyone got any idea how I figure out what is happening? We've never come across this issue before and the application has been running for a good couple of years.
Also, this isn't something that happens regularly but instead seems to happen one every day or so, and even that is at a random time. The Virtual Memory was initially 4GB but because of the issue above, we increased it to 8GB. Recently it spiked at using about 6.8GB out of 8GB, which it has no reason to do so.
Any help would be really appreciated!
The answer is easy here, obviously and certainly you have two issues here
1- You have a serious bug in your process/code that happens intermittently "you need to debug it to find how/when that happens" or at least run a ProcDump
such that you keep it listening on the server on the process W3WP till an exception happens and then analyze this dump to find where the code get stuck and consume that memory/otherwise just debug the code and see what changes were made in last few months "not days"
2- the application get stopped because you have configured/it is configured by default to get disabled break after a certain number of failure repeats, and that's a normal behavior but the main issue as I said is not the application pool itself, its inside the process
please let me know if you need a further explanation or help on this matter