Need to profile node process. i've some memory leaks in production, after some days of running node process.
i've tried node-inspector + v8, but it doesn't work, in new version of node-inspector there is no Profile tab. and in old version when i start profiling error is fired and debugging stopped.
i've also tried nodetime.com, but it doesn't show what i need, also it takes too much memory, it's not for production.
i've also tried dtrace (http://blog.nodejs.org/2012/04/25/profiling-node-js/) but it doesn't give me necessary information.
so what information i need for profiling memory:
get live instances, instances count, size in memory, instance types
do u know how to get that information?
You can try to use look module. It based on nodetime but works locally.
I've found node-memwatch useful.
The downside is you have to embed it in your application and have a bit of code for it, but it's useful for checking the heap at various places to see how much it changed after you did something.
Related
I am using the node.js 6.11.3 ,cluster module, express 4.14
Seeing memory leaks slowly over a period of 1 weeks.
attached is a screenshot of the heap dumps in chrome dev tools. Cant tell the reason for the leak.click for heap dump
Unfortunately, nobody can answer where your leak comes from without accessing the entire application and environment. The real question is how do you debug a node memory leak.
First its important to try to understand how memory leaks occur in Node. How is it possible if Node has built in garbage collection? Well, variables are marked as garbage when no longer referenced. If you have code (closures etc) that still hold reference to variables, they are not collected. That is just one example. There are also dependencies that can cause memory leaks which can confuse you into thinking you have issues in your own code. And maybe you do, in the way you use the dependency.
Bottom line, its best to get familiar with this debugging process yourself so that you can understand the issues leading to the leak. Best of luck. Here is one article that is helpful.
https://www.alexkras.com/simple-guide-to-finding-a-javascript-memory-leak-in-node-js/
Am facing issue in my application servers. Assume that - there are two nodes in the Load-balancer.
Suddenly one of the node from them becomes unhealthy.
When I logged in that instance. There were no logs coming in pm2.
then I check its CPU it was very high.
So please guide me how can I fix this issue. Or any way to debug it.
Check out flame graphs to see where your Node app is CPU bound.
You can also use the new debugging system in Node 6.3 (--inspect) to debug with the full power of Chrome DevTools.
PM2 has some limited protection for runaway issues like this via the max-memory-restart option. Typically, high CPU will also correlate with high memory usage and this option can be used to restart your app when it begins consuming large amounts of memory (which in your case may or may not be the correct moment but it should help).
--max-memory-restart <memory> specify max memory amount used to autorestart (in octet or use syntax like 100M)
I have a Snap web app which serves some JS files and 1-pixel images (it's main task is to rather work fast then serve huge html/media content). There are several servers behind HAProxy.
I upgraded it from GHC 7.6 to 7.8, also upgrading some libs. After upgrade, app started leaking little by little (on all servers), ending in OOM every 15 minutes on 8GB-RAM machines (and much longer on 16Gb) and restarting afterwards.
The problem is, if I compile app for profiling and run app for some time, I can't see any memory leaks anymore. It just consumes 1 CPU and works in constant small memory.
So I wanted to ask for some general advices on how to find such a bottleneck, if running under profiling doesn't help much.
UPDATE: I noticed after playing with an app that if I remove -A100M runtime option it doen't OOM that fast, but with default value HAProxy's "sessions" gets to it's limit (so, basically it chokes). I'm playing with different RTS options now, hope some will help getting both, performance and long-lived memory consumption.
UPDATE 2: just for the record, I found that with -A30 rts option app, while being memory hungry, lives quite well. 8Gb machines OOM-kill app, but 16Gb one looks like this: http://i.imgur.com/3W9KpFS.png (green line is "RAM available", you can see deploy-procedure which restarted app on a graph). I'm happy with the result, but would be glad to know any techniques to profile memory of multi-threaded app anyway.
UPDATE 3: I'm voting to close this question as "too broad". In general, I see that if such generic set of tools that'll let you profile memory easier would exist, they'd definitely be documented elsewhere on wiki etc.
I have never used it myself but maybe ticky-ticky profiling could help? It's supposed to be immune to the optimization changes caused by ordinary profiling, but at the cost of being harder to interpret.
Basically compile and link the relevant modules with the -ticky and -rtsopts flags, and run with +RTS -rfoo.ticky flag to get heaps of data in foo.ticky.
I've been trying to track down a very slow, but persistent, native memory leak in a node.js app, and I've run out of strategies.
The process has what appears to be a level heap, but as the hours and days roll on, the RSS of the node.js process slowly grows. The process is a job handler that runs the same type of job for different parameters, over and over. The growth of the RSS of the process takes the same shape as the line plotting the cumulative number of jobs run, so each job run is somehow leaking a bit of memory.
Since the heap is more or less constant, the standard heap inspection tools don't seem to be much help.
Here's an example of what the memory consumption looks like:
Currently running on node 0.8.7. Each job does a number of database reads/writes, communicates with a redis instance, and does some web requests using mikael/request.
Have you updated to the newest release?
I know everyone says that :), I just felt like I should join the band wagon of updating my version of node.js on my production servers every two weeks when I think I have an issue. Sounds like a great idea doesn't it?
So I have been wondering the same thing, I have several node.js projects that I have been managing for a few months now (and also that I wrote last year). It seems that very slowly the V8 engine, or my node application, just eats memory and never frees it. (its slow enough that I only have to restart them every now and then)
Which is very stressful, especially considering that it should free up the RSS memory, or eventually peak out.
If you are interested in tracking objects being leaked inside of the runtime (and by that i mean javascript objects, functions, etc), mozilla has a very complete blog post about tracking down memory leaks and a few links to projects that can be used to do this.
For what ever reason they don't have this one on the list though. (it seems simple enough, I'm trying it out now on my own projects to see if it works, I tend to not get any of the V8 based ones to compile correctly)
heapdump and here is a link to a how to guide.
From my own experience the V8 engine seems to allocate memory, and hold onto it just incase it needs that exact same memory chunk later. Also my brother who has been using Node.js heavily for about 3 years has seen the same thing.
Also just for completeness (I know you already have), if any one would like to verify that you are not leaking memory inside of V8, an engineer from joyent has a pretty decent write up of how to track V8 memory leaks down.
Is there any chance to see, how much memory is allocated by all scoped variables?
The best way is to use yourKit java profiler.
You can install the agent on the Domino server and then profile the JVM. This will give you the ability to look and see what's going on during run time, see execution times, and see what the number of classes and instances loaded and how much memory they are consuming.
Not exactly what you asked for, but it may help: type tell http xsp heapdump command on console. It will create heapdump file in binaries directory of Domino. Open that file in Heap Analyzer tool (http://www-01.ibm.com/support/docview.wss?uid=swg21190608), also available in IBM Support Assistant (http://www-01.ibm.com/software/support/isa/).