My Node.Js application occasionally fails with a "Segmentation fault" and I am at a loss as to how to diagnose the cause. As mentioned below I have dramatically reduced their frequency by raising maxlisteners but not made the problem go away.
The application runs on a BeagleBone Black under Node v0.8.22 and uses Socket.io to communicate realtime data to browser pages displayed on the BBB's LCD display. It collects data from a sensor connected via I2C using the korevec/node-i2c library. I have, however, isolated that library and still have failures.
The failures generally occur when I have streamed data for a while to the client though they will occasionally/rarely happen at other times as well. This is not surprising as my app uses socket.io to communicate on almost all pages but the streaming page is at much higher volume.
I am getting the below message:
(node) warning: possible EventEmitter memory leak detected. 11
listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
and have been doing so since day 1. I was also seeing the symptoms of a memory leak but since I raised the MaxListeners memory usage stays constant. Failure rate after making this change went down dramatically but has not gone away completely. I am using socket.io on top of http and raised MaxListeners for both socket.it and for http.
How does one go about diagnosing this problem? Is the memory leak error related? I can post code but there is quite a bit of it and I am not sure what parts are most relevant.
Thanks for any help,
Cheers, Will
Related
I am hosting a game website on Heroku that runs on a NodeJS Server. Clients are connected via sockets using the package socket.io.
Once in a while when the garbage collection cycle is triggered, connected clients would experience severe lag and often, disconnections. This is experienced by the clients through delayed incoming chat and delayed inputs to the game.
When I look into the logs, I find error messages relating to the Garbage Collection. Please see the attached logs below. When these GC events happen, sometimes it causes massive memory spikes to the point where the app would exceed it's allowed 0.5GB RAM and would be killed by Heroku. Lately however, the memory spikes don't occur as often, but the severe lag on the client side still happens around once or twice a day.
One aspect of the lag is through the chat. When a user types a message through "All Chat" (and any chat channel), the server currently console.log()'s it to the standard out. I happened to be watching the logs live one time during a spike event and noticed that chat being outputted to the terminal was in real time with no delay, however clients (I was also on the website myself as a client) received these messages in a very delayed fashion.
I have found online a NodeJS bug (that I think was fixed) that would cause severe lag when too much was being console.loged to the screen so I ran a stress test by sending 1000 messages from the client per second, for a minute. I could not reproduce the spike.
I have read many guides on finding memory leaks, inspecting the stack etc. but I'm very unsure how to run these tests on a live Heroku server. I have suspicions that my game objects on closing, are not being immediately cleared out and are all being cleared at once, causing the memory spikes, but I am not confident. I don't know how to best debug this. It is also difficult for me to catch this happening live as it only happens when more than 30+ people are logged in (Doesn't happen often as this is still a fairly small site).
The error messages include references to the circular-json module I use, and I also suspect that this may be causing infinite callbacks on itself somehow and not clearing out correctly, but I am not sure.
For reference, here is a copy of the source code: LINK
Here is a snippet of the memory when a spike happens:
Memory spike
Crash log 1: HERE
Crash log 2: HERE
Is there a way I can simulate sockets or simulate the live server's environment (i.e. connected clients) locally?
Any advice on how to approach or debug this problem would be greatly appreciated. Thank you.
Something to consider is that console.log will increase memory usage. If you are logging verbosely with large amounts of data this can accumulate. Looking quickly at the log, it seems you are running out of memory? This would mean the app starts writing to disk which is slower and will also run garbage collection spiking CPU.
This could mean a memory-leak due to resources not being killed/closed and simply accumulating. Debugging this can be a PITA.
Node uses 1.5GB to keep long-live objects around. Seems like you on a 500mb container so best to configure the web app to start like:
web: node --optimize_for_size --max_old_space_size=460 server.js
While you need to get to the bottom of the leak, you can also increase availability by running more than one worker and also more than one node instance and use socket.io-redis to keep the instance is in sync. I highly recommend this route.
Some helpful content on Nodejs memory on Heroku.
You can also spin up multiple connections via node script to interact with your local dev server using socket.io-client and monitor the memory locally and add logging to ensure connections are being closed correctly etc.
I ended up managing to track my "memory leak" down. It turns out I was saving the games (in JSONified strings) to the database too frequently and the server/database couldn't keep up. I've reduced the frequency of game saves and I haven't had any issues.
The tips provided by Samuel were very helpful too.
We have a quite nasty memory leak going on in a loopback (node.js) app, but it does not seem to happen locally, only on Heroku.
It steadily increases memory usage without any requests, and I fired up 10 000 requests locally without seeing a similar pattern.
I currently have no good ideas for how to debug this further.
It turns out disabling New Relic fixed the issue. We had log level set to debug to figure out another issue, and suddenly all hell broke loose. They do indeed have a notice on their documentation about this.
I believe there is a blog post from Strongloop dealing with memory leak profiling here. It goes over installing heap dump and how to use Chrome dev tools to collect heap snapshots on the client side using the JavaScript console built into the browser. Analysis of the heap can be done within this same console as well.
I have a memory leak on my nodejs application. In order to resume the purpose of the application, it's an api called by an iOS application and a backoffice to administrate some content.
The application is in production and we experience some memory leak due to utilisation.
The memory on the server is always going up and never going down.
I try to analyze the problem using node-heapdump.
First of all, i see a big difference between the heap size of the snapshot given by node-heapdump and the size taken by the app in the memory (heap size ~ 30Mb and RAM size ~ 100Mb), where that difference came from ?
Then i see an increment of the heap size just by refreshing a home page who does not return anything.
Is anyone has an idea of where my problem could be ?
For information i use nodejs version 0.10.x and expressjs 4.0.0
Thanks in advance guys.
EDIT
I install memwatch-next and the leak event is raised.
The error i have is this one :
warning: possible EventEmitter memory leak detected. 11 leak listeners
added. Use emitter.setMaxListeners() to increase limit.
I try to set the defaultMaxListeners but when i stress the application the leak event is raised after sometime.
Does anyone knows what that error means ?
have a look at memwatch-next
I had similar issues with the memwatch package and switched to memwatch-next and it installed without the node-gyp error and produced worked. As for the difference between the RSS and the heapdump , I am in the same boat as you are.
Try to find memory leakage leak and stat from https://www.npmjs.com/package/memwatch.
Hope it would help.
I think you need this tool: easy-monitor
Might I recommend you try running the application with the --inspect argument, this will then allow you to attach Chrome dev tools and take memory snapshots. From here take one at startup one during testing and then one after you have finished testing the application (no more requests to the application but must still be running.) This will allow you to see what exactly is causing the growth in memory.
From here you will be able to see what is causing the growth and hopefully an indication as to where the leak is.
Background
I have a relatively simple node js application (essentially just expressjs + mongoose). It is currently running in production on an Ubuntu Server and serves about 20,000 page views per day.
Initially the application was running on a machine with 512 MB memory. Upon noticing that the server would essentially crash every so often I suspected that the application might be running out of memory, which was the case.
I have since moved the application to a server with 1 GB of memory. I have been monitoring the application and within a few minutes the application tends to reach about 200-250 MB of memory usage. Over longer periods of time (say 10+ hours) it seems that the amount keeps growing very slowly (I'm still investigating that).
I have been since been trying to figure out what is consuming the memory. I have been going through my code and have not found any obvious memory leaks (for example unclosed db connections and such).
Tests
I have implemented a handy heapdump function using node-heapdump and I have now enabled --expore-gc to be able to manually trigger garbage collection. From time to time I try triggering a manual GC to see what happens with the memory usage, but it seems to have no effect whatsoever.
I have also tried analysing heapdumps from time to time - but I'm not sure if what I'm seeing is normal or not. I do find it slightly suspicious that there is one entry with 93% of the retained size - but it just points to "builtins" (not really sure what the signifies).
Upon inspecting the 2nd highest retained size (Buffer) I can see that it links back to the same "builtins" via a setTimeout function in some Native Code. I suspect it is cache or https related (_cache, slabBuffer, tls).
Questions
Does this look normal for a Node JS application?
Is anyone able to draw any sort of conclusion from this?
What exactly is "builtins" (does it refer to builtin js types)?
I've written an app which is handling videos. As we know, video processing takes a huge amount of memory while dealing with HD resolution. My App always seemed to crash. But actually I am 100% sure, that there is no memory leak in my code. Instruments is showing no leak.
At the beginning I am startin up one OpenGLES view and the video engine. For a very short time the memory consumption is high, but falling down to normal level after the initializations are done. I am always getting memory warnings during this period. Normally this is no problem. But if I have a lot of apps in suspended mode running, the App seems to be crashing. Watching into the crash log and using the debugger shows up, that I am only running out of memory.
My customers are flooding my support mail with "app is crashing" mails. But I do know, that they have too much Apps running in the background, so there is no memory left to go. I think it's bad style programing saying the customer that he has to close Background tasks before running the app.
According to this post this is a common problem.
My question is: Is it possible to tell the OS that one needs a lot of memory so the OS should terminate some suspended Apps? This memory stuff makes me crazy, because it's no bug I could fix.
No. It is not possible to affect anything outside of your sandbox without API calls. None exist for affecting other processes in the public API.
Have you tried to minimize your memory usage? In my experience once a memory warning it thrown apps can be more likely to have problems once they are in the background, even when memory usages drops.
If you are using OpenGLES and textures, if you haven't already compress your textures. What is the specific cause of your memory allocation spike?