I have a Parse Server which is a Node.js + express wrapper for a mobile app (about 100 simultaneous users every day), hosted on DigitalOcean. The app server communicates with MongoDB, which is hosted on another droplet of DigitalOcean. I'm using pm2 as a process manager and its monitoring tool, which is web-based. On the same process, we operate LiveQuery, a WebSocket server made by the Parse community as well.
The thing is, I've been having some performance issues with the server. Everything works smoothly, until the Active handles rise up uncontrollably! (see the image below) It's like after one point the server says "I'm done! Now I rest!"
Usually the active handles stay between 30 to 70. The moment I restart the process with pm2 restart everything goes back to normal!
I've been having this issue for quite some time now and I haven’t been able to figure out what’s causing it! Any help will be greatly appreciated!
EDIT: I did a stress test where I created 200 LiveQuery sockets for 1 user, instead of 2 that a user normally has and there was a spike of 300 active handles, for like 5 seconds! The moment the sockets were all created, everything went back to normal!
I usually use restart based on memory usage
pm2 start filename.js --max-memory-restart 160 --exp-backoff-restart-delay=100
pm2 has also built-in cron job or autostart script setup in case the server ever restarts, see https://pm2.keymetrics.io/docs/usage/restart-strategies/
it would be could if pm2 would provide restart options based on active connections or heap memory
Related
I have a REST api node app.
Once its running on localhost, it runs until I stop the dev debugging, no errors.
I moved it over to my cPanel hosting, installed a node app.
It starts up the same as localhost.
But after 30 minutes being idle, it shuts down.
The next request after this, restarts the app.
There are no crash or errors in the log, just the restarting messages.
I know this is default behaviour for free hosting, like Heroku but I'm paying for this hosting package.
Does anyone know...
Is this default behaviour for cPanel hosted node apps, or is my app causing this (using too much memory or cpu for example?
Is there any settings that can be edited to change this?
According to the docs, cPanel uses something called Phusion Passenger to run Node.js. In turn, Passenger docs show a default "idle time" of 5 minutes and a default of passenger_min_instances = 1. No idea if cPanel changes the defaults, or if the hosting provider did. I would recommend contacting the hosting provider about the issue in any case, and asking about these options specifically - they may be able to help or tune the service for you.
The startup time for a node app depends on what it's doing. A rest-api could be in the milliseconds, whereas a small Ai app loading a corpus or training a dataset (which mine was) could end up being 30 seconds plus. However the quantity of users did not warrant a dedicated server, so the work-around was to call the endpoint using a CRON, keeping the app alive.
Not perfect, but this type of thing may be useful if you are using aws lambda, which calls a 3rd party service, and which charges based on time taken. Every millisecond counts.
So I have a node (v8.9.4) server that is running on an AWS EC2 instance using the forever package to start it up. The server has worked without any issues for years but now that it's grown and more people are using it, it suddenly starts to time out all requests at seemingly random times, after working for a few hours.
I've found that running forever restart on the server gets all requests working again so I've got a temporary cronjob to restart it every hour but this is not good design and I would much rather have the server running without any issues.
I've gone through my server logs and found this which may be significant:
error: Forever detected script was killed by signal: SIGKILL
error: Script restart attempt #131
Warning: connect.session() MemoryStore is not designed for a production environment, as it will leak memory, and will not scale past a single process.
Another thing that may be important, the server stays running while this issue occurs so any checks on the server status through UptimeRobot (or any server status checker) returns a success.
Considering the server will run fine for a few hours and also start up again with no issues after a restart, I'm thinking it is not an issue with the code but something else that I am not aware of. My current hypothesis is the requests will start timing out if the server runs out of CPU but I would like to explore more options before making the final call on the issue. If anyone had any insight into this issue, I would be super grateful! :)
I'm trying to figure out why my nodejs app becomes unresponsive after 11h 20min. It happens every time, no matter if I run it on amazon-linux or Red Hat.
My Stack:
nodejs (v. 6.9.4)
mongodb (3.2)
pm2 process manager
AWS EC2 instance T2 medium
Every time I'm running the app it becomes unresponsive with an error returned to the browser:
net::ERR_CONNECTION_RESET
Pm2 doesn't restart the app, so I suspect it has nothing to do with nodejs, I also analysed the app and it doesn't have memory leaks. Db logs also look alright.
The only constant factor is the fact that the app crashes after it runs for 11h 20min.
I'm handling all possible errors from the nodejs app, but no errors in the log files occur so I suspect it has to be something else.
I also checked var/log/messages and /home/centos/messages but nothing related to the crash of the app there either.
/var/log/mongodb/mongo.log doesn't show anything specific either.
What would be the best way to approach the problem ?
Any clues how can I debug it or what could be the reason ?
Thanks
Copied from the comment since it apparently led to the solution:
You're leaking something other than memory is my guess, maybe file descriptors. Try using netstat or lsof to see if there are a lot more open connections or files than you expect.
Loving node but I have a question about making my application more fault tolerant. I like to assume the worst so lets assume my application has a fault that leads to application exit every now and then. PM2 or your preferred monitor will reboot it but some requests may fail while it restarts / before load balancers detect its down.
Are there any good containers or tools I can put in front of node to buffer these requests (for a reasonable amount of time) until the app bootstraps and comes back online? Perhaps Nginx can do this?
I'm developing an application using Node.js and Sails.
I'm going to run like: 20 instances of the same app at the same time, and all of them will use a Local MongoDB to store model data.
My problem started like this: Only the first 7 or 8 launched apps was starting, others were failing because they couldn't connect to the database.
Ok, I went though some searching, and saw that I had to increase the number of connections, but what made me thing something was wrong, is that: Each app launched, is creating about 35 connections!
So, when launching 6 or 8 apps, they were taking about 250 connections!!!
That seems to much, since only one connection per app is enough (I think). Is this 'normal', or is it some problem in Sails Waterline core?
Any solution on this issue?
I have the same issue ( load balanced instances connecting to mongo ) without using sails...
Another issue is that due to "zero downtime deploy" i clone the cluster and then change the DNS so temporarily having double amount of connections.
So on my case i'm also listening to SIGINT and SIGQUIT and closing the connections before the app terminates, so then hopefully "keep alive" connections will die together with the app.
There is tons of people with similar problems around, but i failed to find a spot-on solution /=