We are renting a shared hosting server, and we run our nodejs/express app on it. The problem is that we are limited in the number of processes allowed (30), and nodejs spawns some processes from time to time. When we reach 30 processes, the server stops responding and we have to reboot it.
We tried to run our app using pm2, but it created too much processes. So we landed on "forever", which seems to create less processes, but the issue is still occuring. We're not experts in nodejs so we have no idea why this is happening, if anyone knows it would be a life saver !
Related
I have developed a chat application using node JS, express JS, mongodb and socket io. Messages sent & received are stored in mongodb. But when I run the chat application the server CPU usage spikes continuously. See the attached screenshot:
It seems that something is stacking up. When I stop the chat application then the usage is resumed to minimum. What can be possible methods to fix this ?
There are a few potential methods you could use to debug this issue:
Use the node.js built-in debugger. You can set breakpoints in your code and then run your chat application under the debugger to see where the CPU usage is spikes are happening.
Use a profiler to take a look at what your chat application is doing when it's running. This can help you to identify which parts of the code are using up the most CPU time.
Use a performance monitoring tool to track the CPU usage of your chat application over time. This can help you to identify whether the issue is getting worse or if there are any patterns to the spikes in usage.
Try to reproduce the issue in a test environment and then use a debugging tool like strace or ltrace to see what system calls are being made when the CPU usage spikes. This can help you to identify what the chat application is doing that is causing the issue.
It's hard to answer your question without more information, but some sanity checks:
Make sure that you are not running too many processes on your server. If you are running multiple node.js applications on the same server, that can lead to high CPU usage. Try running each application on its own server, or limiting the number of processes that are running on each server.
Try using a different server environment. If you are using a shared hosting environment, the CPU usage may be spikes due to other users on the same server. Try using a dedicated server, or a virtual private server, which can help to reduce the CPU usage.
So I have a node (v8.9.4) server that is running on an AWS EC2 instance using the forever package to start it up. The server has worked without any issues for years but now that it's grown and more people are using it, it suddenly starts to time out all requests at seemingly random times, after working for a few hours.
I've found that running forever restart on the server gets all requests working again so I've got a temporary cronjob to restart it every hour but this is not good design and I would much rather have the server running without any issues.
I've gone through my server logs and found this which may be significant:
error: Forever detected script was killed by signal: SIGKILL
error: Script restart attempt #131
Warning: connect.session() MemoryStore is not designed for a production environment, as it will leak memory, and will not scale past a single process.
Another thing that may be important, the server stays running while this issue occurs so any checks on the server status through UptimeRobot (or any server status checker) returns a success.
Considering the server will run fine for a few hours and also start up again with no issues after a restart, I'm thinking it is not an issue with the code but something else that I am not aware of. My current hypothesis is the requests will start timing out if the server runs out of CPU but I would like to explore more options before making the final call on the issue. If anyone had any insight into this issue, I would be super grateful! :)
I have a Parse Server which is a Node.js + express wrapper for a mobile app (about 100 simultaneous users every day), hosted on DigitalOcean. The app server communicates with MongoDB, which is hosted on another droplet of DigitalOcean. I'm using pm2 as a process manager and its monitoring tool, which is web-based. On the same process, we operate LiveQuery, a WebSocket server made by the Parse community as well.
The thing is, I've been having some performance issues with the server. Everything works smoothly, until the Active handles rise up uncontrollably! (see the image below) It's like after one point the server says "I'm done! Now I rest!"
Usually the active handles stay between 30 to 70. The moment I restart the process with pm2 restart everything goes back to normal!
I've been having this issue for quite some time now and I haven’t been able to figure out what’s causing it! Any help will be greatly appreciated!
EDIT: I did a stress test where I created 200 LiveQuery sockets for 1 user, instead of 2 that a user normally has and there was a spike of 300 active handles, for like 5 seconds! The moment the sockets were all created, everything went back to normal!
I usually use restart based on memory usage
pm2 start filename.js --max-memory-restart 160 --exp-backoff-restart-delay=100
pm2 has also built-in cron job or autostart script setup in case the server ever restarts, see https://pm2.keymetrics.io/docs/usage/restart-strategies/
it would be could if pm2 would provide restart options based on active connections or heap memory
I'm trying to figure out why my nodejs app becomes unresponsive after 11h 20min. It happens every time, no matter if I run it on amazon-linux or Red Hat.
My Stack:
nodejs (v. 6.9.4)
mongodb (3.2)
pm2 process manager
AWS EC2 instance T2 medium
Every time I'm running the app it becomes unresponsive with an error returned to the browser:
net::ERR_CONNECTION_RESET
Pm2 doesn't restart the app, so I suspect it has nothing to do with nodejs, I also analysed the app and it doesn't have memory leaks. Db logs also look alright.
The only constant factor is the fact that the app crashes after it runs for 11h 20min.
I'm handling all possible errors from the nodejs app, but no errors in the log files occur so I suspect it has to be something else.
I also checked var/log/messages and /home/centos/messages but nothing related to the crash of the app there either.
/var/log/mongodb/mongo.log doesn't show anything specific either.
What would be the best way to approach the problem ?
Any clues how can I debug it or what could be the reason ?
Thanks
Copied from the comment since it apparently led to the solution:
You're leaking something other than memory is my guess, maybe file descriptors. Try using netstat or lsof to see if there are a lot more open connections or files than you expect.
We use clustering with our express apps on multi cpu boxes. Works well, we get the maximum use out of AWS linux servers.
We inherited an app we are fixing up. It's unusual in that it has two processes. It has an Express API portion, to take incoming requests. But the process that acts on those requests can run for several minutes, so it was build as a seperate background process, node calling python and maya.
Originally the two were tightly coupled, with the python script called by the request to upload the data. But this of course was suboptimal, as it would leave the client waiting for a response for the time it took to run, so it was rewritten as a background process that runs in a loop, checking for new uploads, and processing them sequentially.
So my question is this: if we have this separate node process running in the background, and we run clusters which starts up a process for each CPU, how is that going to work? Are we not going to get two node processes competing for the same CPU. We were getting a bit of weird behaviour and crashing yesterday, without a lot of error messages, (god I love node), so it's bit concerning. I'm assuming Linux will just swap the processes in and out as they are being used. But I wonder if it will be problematic, and I also wonder about someone getting their web session swapped out for several minutes while the longer running process runs.
The smart thing to do would be to rewrite this to run on two different servers, but the files that maya uses/creates are on the server's file system, and we were not given the budget to rebuild the way we should. So, we're stuck with this architecture for now.
Any thoughts now possible problems and how to avoid them would be appreciated.
From an overall architecture prospective, spawning 1 nodejs per core is a great way to go. You have a lot of interdependencies though, the nodejs processes are calling maya which may use mulitple threads (keep that in mind).
The part that is concerning to me is your random crashes and your "process that runs in a loop". If that process is just checking the file system you probably have a race condition where the nodejs processes are competing to work on the same input/output files.
In theory, 1 nodejs process per core will work great and should help to utilize all your CPU usage. Linux always swaps the processes in and out so that is not an issue. You could start multiple nodejs per core and still not have an issue.
One last note, be sure to keep an eye on your memory usage, several linux distributions on EC2 do not have a swap file enabled by default, running out of memory can be another silent app killer, best to add a swap file in case you run into memory issues.