I installed NodeJS v8.1.2 on an Amazon Linux Distribution on aws.
I have pm2 installed that is in-charge of restarting process in cases it fails.
I catch uncaught exceptions in the process and log them so the process wont restart since I use socket.io and I don't want users to get disconnected on every single exception.
about two month ago after updating nodejs to v7, nodejs would restart randomly with no reason what so ever, so I decided to compile nodejs from sources using nvm, and it resolved the issue.
about a week ago I updated nodejs again to v8.1.2 and today the process restarted again with no reason at all, no exception... nothing on the servers stats where too high.. no reason what so ever.
what do I do?
any information regarding the issue would be greatly appreciated
updates
I checked /var/log/messages and I noticed a segmentation fault error at the time of the restart. do I have to create a core dump to investigate the issue further?
can a segmentation fault of the nodejs process can be caused because of my code ?
what do I do ? :)
I had this situation because of a memory leak.
Try to monitor your RAM.
Related
So I have a node (v8.9.4) server that is running on an AWS EC2 instance using the forever package to start it up. The server has worked without any issues for years but now that it's grown and more people are using it, it suddenly starts to time out all requests at seemingly random times, after working for a few hours.
I've found that running forever restart on the server gets all requests working again so I've got a temporary cronjob to restart it every hour but this is not good design and I would much rather have the server running without any issues.
I've gone through my server logs and found this which may be significant:
error: Forever detected script was killed by signal: SIGKILL
error: Script restart attempt #131
Warning: connect.session() MemoryStore is not designed for a production environment, as it will leak memory, and will not scale past a single process.
Another thing that may be important, the server stays running while this issue occurs so any checks on the server status through UptimeRobot (or any server status checker) returns a success.
Considering the server will run fine for a few hours and also start up again with no issues after a restart, I'm thinking it is not an issue with the code but something else that I am not aware of. My current hypothesis is the requests will start timing out if the server runs out of CPU but I would like to explore more options before making the final call on the issue. If anyone had any insight into this issue, I would be super grateful! :)
I am facing a CPU overload issue due to a nodejs application that I am running in a remote Ubuntu 16.04 LTS virtual machine. I am using PM2 to schedule my nodejs application as a service.
Initially when the nodejs application is launched, the CPU load remain quite low; about 30% at most. Then slowly I find the CPU load going up till it gets to 100%. This nodejs application is polling a stock website for new information on a stock and then does some calculations and then repeats after 5 minutes. I dont see how its causing this overload on the CPU.
I notice that my 1 nodejs application shows up as 6 different processes in HTOP command. Not sure if this normal or how to fix this. Any help would be highly appreciated.
Thanks
Regards,
Adeel
Thanks, Gerard. Your reply helped solve the problem. Turns out guardian.js was not exiting and just opening up new processes till it overloaded the system.
I'm trying to figure out why my nodejs app becomes unresponsive after 11h 20min. It happens every time, no matter if I run it on amazon-linux or Red Hat.
My Stack:
nodejs (v. 6.9.4)
mongodb (3.2)
pm2 process manager
AWS EC2 instance T2 medium
Every time I'm running the app it becomes unresponsive with an error returned to the browser:
net::ERR_CONNECTION_RESET
Pm2 doesn't restart the app, so I suspect it has nothing to do with nodejs, I also analysed the app and it doesn't have memory leaks. Db logs also look alright.
The only constant factor is the fact that the app crashes after it runs for 11h 20min.
I'm handling all possible errors from the nodejs app, but no errors in the log files occur so I suspect it has to be something else.
I also checked var/log/messages and /home/centos/messages but nothing related to the crash of the app there either.
/var/log/mongodb/mongo.log doesn't show anything specific either.
What would be the best way to approach the problem ?
Any clues how can I debug it or what could be the reason ?
Thanks
Copied from the comment since it apparently led to the solution:
You're leaking something other than memory is my guess, maybe file descriptors. Try using netstat or lsof to see if there are a lot more open connections or files than you expect.
Recently I'm having an issue with my server.
My node server stops, and forever does not restart it.
In my forever log I see this line:
error: Forever detected script was killed by signal: SIGKILL
The server itself does not throw an error. In fact the server seems to run without any glitches and then a random SIGKILL is executed.
I don't know if it's AWS shutting down my server, or if it's an issue with forever, or perhaps the node server itself.
Searching Google does not provide much insight.
I thought this might be related to a cpu spike or a memory usage spike, but both seem to be low (but maybe there's a spike for a split second that I don't recognize).
Is this an issue anyone has encountered before?
Any idea how can I fix it?
Well..
Why the problem occurred is still mystery but I was able to resolve it by reducing the queue for my queries on my MongoDB.
While both mongo and node were not using a lot of RAM this seems to be the cause of the issue since by reducing the number of queries, the problem disappeared.
What exactly triggered the SIGKILL is still a mystery, yet I thought this information maybe useful for other users.
For me it had to do with the way mongoose was setup and interacting with the application code.
I was able to fix by creating a connection using the answer from here: Mongoose Connection, creating my schema definitions and only exporting the models to be used.
I hope this is helpful to someone
I have a Node.js app that loads some data from Mysql into Redis when the app starts. It has been working fine up until we modified the data in Mysql.
Now it is just exiting with a Killed message.
I am trying to pinpoint the problem but is is hard to debug using the node-inspector as the problem doesn't appear when running in --debug.
I don't think my problem is in the data itself because it works on my local machine but doesn't work on my production box.
My question is, what causes the Killed message? Is it Node.js or is it in the Mysql driver or elsewhere?
Check your system logs for messages about Node being killed. Your Node application might be using excessive memory and getting killed by the Out-Of-Memory killer.
Not sure if Redis is what causes the Killed message but that was the cause of my problem.
I was sending to much data to multi because I originally thought that was the way to use pipelining (which is automatic).