I'm writing a simple cms in Node.js, Express and MongoDB. I'm planning to run a different Node.js process for every site. The problem is that after startup the process takes about 90m of RAM and for me it's too big (eight site take all server RAM). This memory is taken after the first connection to the site and other connections don't affect the memory.
Is there a guideline or a list of "best practices" to optimize this memory usage? I'm trying to track where the memory is allocated with process.memoryUsage() or a similar function but it's not simple to do this.
Is not a problem of memory leaks or something similar because the memory usage doesn't grow up after the first connection, so probably the optimization could be in loading less modules or do something differently...
The links below may help you to understand and detect memory leaks (if they do exist):
Debugging memory leaks in node.js
Detecting Memory Leaks in Node.js Applications
Tracking Down Memory Leaks in Node.js
These SO questions may also be useful:
How to monitor the memory usage of Node.js?
Node.js Memory Leak Hunting
Here is a quick fix, a node.js lib that will restart the any node process once it reaches a certain size.
https://github.com/DoryZi/memory_limiter
Set the --max_old_space_size CLI flag to control the maximum heap size. There's a post that describes Running a node.js app in a low-memory environment
tl;dr; Try setting this value, in megabytes, to about 80% of the maximum memory footprint you want node to try to remain under. e.g. to run app.js and keep it under 500MB RAM used
node --max_old_space_size=400 app.js
This setting is also described in the Node JS CLI documentation
Related
As title, I have encountered a memory leak problem in loopback framework on Node js.
I can't find any problem from request API call.
So I wonder that is there any way I can dump all the objects and variables in heap memory in NodeJs when the memory usage is constantly arising, so that I can find any clue in my code.
Thanks.
Firstly, if you are running a node process in a memory restricted environment, you have to make sure that you restrict the amount of memory that is allocated to Node and to V8. What may seem like a memory leak could just be a lazy garbage collection process by the V8 engine. To supervise memory usage, I recommend the npm add-on memwatch-next.
You can force V8 to perform a garbage collection by executing your node js program with in the following manner: node --expose-gc test.js
Now, within the code, you are able to call global.gc() at set time intervals when you'd like V8 to perform an old-space cleanup.
Additional information can be found here: https://simonmcmanus.wordpress.com/2013/01/03/forcing-garbage-collection-with-node-js-and-v8/
I'm running Node.js on a server with only 512MB of RAM. The problem is when I run a script, it will be killed due to out of memory.
By default the Node.js memory limit is 512MB. So I think using --max-old-space-size is useless.
Follows the content of /var/log/syslog:
Oct 7 09:24:42 ubuntu-user kernel: [72604.230204] Out of memory: Kill process 6422 (node) score 774 or sacrifice child
Oct 7 09:24:42 ubuntu-user kernel: [72604.230351] Killed process 6422 (node) total-vm:1575132kB, anon-rss:396268kB, file-rss:0kB
Is there a way to get rid of out of memory without upgrading the memory? (like using persistent storage as additional RAM)
Update:
It's a scraper which uses node module request and cheerio. When it runs, it will open hundreds or thousands of webpage (but not in parallel)
If you're giving Node access to every last megabyte of the available 512, and it's still not enough, then there's 2 ways forward:
Reduce the memory requirements of your program. This may or may not be possible. If you want help with this, you should post another question detailing your functionality and memory usage.
Get more memory for your server. 512mb is not much, especially if you're running other services (such as databases or message queues) which require in-memory storage.
There is the 3rd possibility of using swap space (disk storage that acts as a memory backup), but this will have a strong impact on performance. If you still want it, Google how to set this up for your operating system, there's a lot of articles on the topic. This is OS configuration, not Node's.
Old question, but may be this answer will help people. Using --max-old-space-size is not useless.
Before Nodejs 12, versions have an heap memory size that depends on the OS (32 or 64 bits). So, following documentations, on 64-bit machines that (the old generation alone) would be 1400 MB, far away from your 512mb.
From Nodejs12, heap size take care of system RAM; however Nodejs' heap isn't the only thing in memory, especially if your server isn't dedicated to it. So set the --max-old-space-size permit to have a limit regarding the old memory heap, and if your application comes closer, the garbage collector will be triggered and will try to free memory.
I've write a post about how I've observed this: https://loadteststories.com/nodejs-kubernetes-an-oom-serial-killer-story/
I've a static ExpressJS Server like that:
var express = require("express"),
app = express();
app.use(express.static(__dirname));
app.listen(1050);
When i start the server it uses 20MB of v8 heap. If i do a page reload every second the heap used grow continuously. After 4 hours it goes to 40MB of v8 heap used. The total v8 heap goes to 80MB and the RSS (total memory used by the process) goes to 130MB.
Why this simple and static server uses so much ram? It seems a memory leak. If i don't stop the page reload, the used memory keeps growing.
It's impossible to do larger projects if a simple static server like this uses too much ram.
NodeJS version: v0.10.21
ExpressJS version: 3.3.5
EDIT: I noticed that it's a problem with NodeJS, because i tried node-static instead of express, and while the used/total V8 heap stayed constant, the RSS memory used by nodejs continued to grow up.
Screen:
https://www.dropbox.com/s/4j5qs3rv2549dix/Screenshot%202014-03-20%2014.06.57.png
https://www.dropbox.com/s/0c30ou8l3rv2081/Screenshot%202014-03-20%2014.07.54.png
https://www.dropbox.com/s/5be1isk4v70qj8g/Screenshot%202014-03-20%2014.08.10.png
(Starts at 13:48)
Not sure if you are still in need of an answer but ill post for anyone else who might be having the same issues.
I was having this same exact problem and fixed it by using:
--max-old-space-size 5
This limits how much memory is held until it gets deleted by GC.
To help the creators of NodeJS and Express potentially, try taking memory snapshots (done using chrome dev tools, use the url chrome://inspect ) this will allow you to see where your memory is allocated.
I've just fixed a memory leak in my node application which was in the heap of node.
I've profiled that with Google's Profiler and managed to fix the memory leak.
Now my app is running again for some time, and I've seen that the heap size is pretty constant. No memory leak anymore. But when I check my server's free RAM, I see a decrease...
When I restart my node server the RAM is up to it's normal free RAM.
Now I've heard about that Node.js can save objects and stuff outside of the heap. I think that is what's causing the memory leak here.
How can I see what's taking up the memory? Can't really profile anything, or can I?
I'm using:
node.js: v0.8.18 and
socket.io: v0.9.13
Some other node modules that I'm using are: nodetime, heapdump (will delete this, though), jquery, crypto, request and querystring.
Some graphs:
Free OS memory and
Node RSS and Heap used
I am new to pm2 concept,I am facing problem where my cpu usage increases and reaches upto 100% memory and my server goes down resulting to crashing of website,so can anyone please consult me on this.Do I need to change the configuration of my production(live) server such as increasing memory?My code is also neccessary and sufficient.I am ec2 user.
The system requirements will mostly depend on your application which you told nothing about. If CPU reaches 100% then you likely have some tight loop that is actively adding delays by burning cycles synchronously or something like that. The 100% memory usage can mean memory leaks and in that case no RAM will be sufficient because leaking memory will use up all your RAM eventually, no matter how large it is.
You need to profile your application with real usage patterns on a system where that app works and only then you will know how much resources it needs. This is true for every kind of application.
Additionally if you notice that resources usage grown over time then it may be a sign of some resource leaking, like memory leaking, spawning processes that don't exit but use CPU and RAM, etc.
first of all i would like to suggest you to follow these guideline for production envoiremnt.
1) disable morgon if you enable it as a dev envoiremnt.
2) use nginx or pm2 for load balancing.
or you can easily handle load balancing by using this command
pm2 start server.js -i 10
3)handle uncaugh exception. ie:
process.on("uncaughtException".function (err){
//do error handling
})