Recently I'm having an issue with my server.
My node server stops, and forever does not restart it.
In my forever log I see this line:
error: Forever detected script was killed by signal: SIGKILL
The server itself does not throw an error. In fact the server seems to run without any glitches and then a random SIGKILL is executed.
I don't know if it's AWS shutting down my server, or if it's an issue with forever, or perhaps the node server itself.
Searching Google does not provide much insight.
I thought this might be related to a cpu spike or a memory usage spike, but both seem to be low (but maybe there's a spike for a split second that I don't recognize).
Is this an issue anyone has encountered before?
Any idea how can I fix it?
Well..
Why the problem occurred is still mystery but I was able to resolve it by reducing the queue for my queries on my MongoDB.
While both mongo and node were not using a lot of RAM this seems to be the cause of the issue since by reducing the number of queries, the problem disappeared.
What exactly triggered the SIGKILL is still a mystery, yet I thought this information maybe useful for other users.
For me it had to do with the way mongoose was setup and interacting with the application code.
I was able to fix by creating a connection using the answer from here: Mongoose Connection, creating my schema definitions and only exporting the models to be used.
I hope this is helpful to someone
Related
I installed NodeJS v8.1.2 on an Amazon Linux Distribution on aws.
I have pm2 installed that is in-charge of restarting process in cases it fails.
I catch uncaught exceptions in the process and log them so the process wont restart since I use socket.io and I don't want users to get disconnected on every single exception.
about two month ago after updating nodejs to v7, nodejs would restart randomly with no reason what so ever, so I decided to compile nodejs from sources using nvm, and it resolved the issue.
about a week ago I updated nodejs again to v8.1.2 and today the process restarted again with no reason at all, no exception... nothing on the servers stats where too high.. no reason what so ever.
what do I do?
any information regarding the issue would be greatly appreciated
updates
I checked /var/log/messages and I noticed a segmentation fault error at the time of the restart. do I have to create a core dump to investigate the issue further?
can a segmentation fault of the nodejs process can be caused because of my code ?
what do I do ? :)
I had this situation because of a memory leak.
Try to monitor your RAM.
I'm trying to figure out why my nodejs app becomes unresponsive after 11h 20min. It happens every time, no matter if I run it on amazon-linux or Red Hat.
My Stack:
nodejs (v. 6.9.4)
mongodb (3.2)
pm2 process manager
AWS EC2 instance T2 medium
Every time I'm running the app it becomes unresponsive with an error returned to the browser:
net::ERR_CONNECTION_RESET
Pm2 doesn't restart the app, so I suspect it has nothing to do with nodejs, I also analysed the app and it doesn't have memory leaks. Db logs also look alright.
The only constant factor is the fact that the app crashes after it runs for 11h 20min.
I'm handling all possible errors from the nodejs app, but no errors in the log files occur so I suspect it has to be something else.
I also checked var/log/messages and /home/centos/messages but nothing related to the crash of the app there either.
/var/log/mongodb/mongo.log doesn't show anything specific either.
What would be the best way to approach the problem ?
Any clues how can I debug it or what could be the reason ?
Thanks
Copied from the comment since it apparently led to the solution:
You're leaking something other than memory is my guess, maybe file descriptors. Try using netstat or lsof to see if there are a lot more open connections or files than you expect.
I have some serious bunch of asynchronous operations running, but NodeJS process is just not exiting when supposedly all have been done. Can I somehow find out what keeps it running? Can I see heap stack of running process somehow? Or can you give me tips what are the most usual causes of such idlings?
I don't have any kind of server running there, but I am using async.nextTick quite extensively which basically uses setImmediate. I am not sure if this can somehow get stuck. Also there are no connections to any kind of database or remote server. It's just process that does some work on file system.
Maybe there is some recursive loop, but I have tried using node-inspector and paused execution after it was stuck and it didn't showed me any point in code where it would hanging.
Take a look at process._getActiveHandles() and process._getActiveRequests()
error: Forever detected script was killed by signal: SIGKILL
I'm running a node app on production with "forever".
Somewhat randomly, it shows these events in the logs, and this is causing requests with lots of backend processing that access a database to just stop, and you then have to re-request and hope that it finishes before the next SIGKILL.
My question is this: under any circumstances could an application exception cause a SIGKILL like this, in the context of forever?
I can't reproduce this locally in my development environment.
ENVIRONMENT:
ubuntu 14.04
memcached
forever
node by itself (no nginx reverse proxy or anything)
connecting to a postgres database to query data
It's really hard to say for sure if the SIGKILL's are on an set interval, or if they are happening at a certain point in program execution. The logs don't have a timestamp by default. From looking at the output, I'd say it is happening somewhat randomly in program execution since it is at different points in the log file they appear.
Check your system logs to see if the linux kernel's out of memory killer is sending the signal as per this answer
I have a Node.js app that loads some data from Mysql into Redis when the app starts. It has been working fine up until we modified the data in Mysql.
Now it is just exiting with a Killed message.
I am trying to pinpoint the problem but is is hard to debug using the node-inspector as the problem doesn't appear when running in --debug.
I don't think my problem is in the data itself because it works on my local machine but doesn't work on my production box.
My question is, what causes the Killed message? Is it Node.js or is it in the Mysql driver or elsewhere?
Check your system logs for messages about Node being killed. Your Node application might be using excessive memory and getting killed by the Out-Of-Memory killer.
Not sure if Redis is what causes the Killed message but that was the cause of my problem.
I was sending to much data to multi because I originally thought that was the way to use pipelining (which is automatic).