It is quite surprising to see the counters heapUsed and external showing reduction but still the heapTotal showing a spike.
***Memory Log - Before soak summarization 2
"rss":217214976,"heapTotal":189153280,"heapUsed":163918648,"external":1092977
Spike in rss: 4096
Spike in heapTotal: 0
Spike in heapUsed: 22240
Spike in external: 0
***Memory Log - Before summarizing log summary for type SOAK
"rss":220295168,"heapTotal":192294912,"heapUsed":157634440,"external":318075
Spike in rss: 3080192
Spike in heapTotal: 3141632
Spike in heapUsed: -6284208
Spike in external: -774902
Any ideas why the heapTotal is drastically increasing despite the heapUsed and external going drastically down. I mean I really thought that heapTotal = heapUsed + external.
I am using the following code to track memory
var prevStats;
function logMemory (path,comment) {
if (! fs.existsSync(path)) {
fs.mkdirSync(path, DIR_MODE);
}
path = pathUtils.posix.join(path,"memoryLeak.txt");
if (comment) comment = comment.replace(/(.+)/,' - $1');
var memStats = process.memoryUsage();
fs.appendFileSync(path,`\r\n\r\n***Memory Log ${comment}\r\n`+JSON.stringify(process.memoryUsage()));
if (prevStats) {
_.forEach (
Object.keys(memStats),
key => {
if (memStats.hasOwnProperty(key)) {
fs.appendFileSync(path,`\r\nSpike in ${key}: ${memStats[key]-prevStats[key]}`);
}
}
);
}
prevStats = memStats;
}
Related
I've been learning about memory management in Nodejs and I'm trying to understand why the following two behaviors occurs:
PS: I'm using the following utility functions to help me print memory to console:
function toMb (bytes) {
return (bytes / 1000000).toFixed(2);
}
function printMemoryData(){
const memory = process.memoryUsage();
return {
rss: `${toMb(memory.rss)} MB -> Resident Set Size - total memory allocated for the process execution`,
heapTotal: `${toMb(memory.heapTotal)} MB -> total size of the allocated heap`,
heapUsed: `${toMb(memory.heapUsed)} MB -> actual memory used during the execution`,
external: `${toMb(memory.external)} MB -> V8 external memory`,
};
}
Part 1) fs.readFile with encoding vs buffers
When I do:
let data;
fs.readFile('path/to/500MB', {}, function(err, buffer){
data = buffer
console.log('Memory usage after files read:', printMemoryData());
});
I get the following output:
Memory usage after files read: {
rss: '565.22 MB -> Resident Set Size - total memory allocated for the process execution',
heapTotal: '11.01 MB -> total size of the allocated heap',
heapUsed: '5.66 MB -> actual memory used during the execution',
external: '524.91 MB -> V8 external memory'
}
Even though I'm storing the data in a local data variable/v8object, the heap isn't used.
But when I do add the encoding:
fs.readFile('path/to/500MB', {encoding: 'utf-8'}, function(err, buffer){
console.log('Memory usage after files read:', printMemoryData());
});
I get the following output:
Memory usage after files read: {
rss: '1088.71 MB -> Resident Set Size - total memory allocated for the process execution',
heapTotal: '535.30 MB -> total size of the allocated heap',
heapUsed: '529.95 MB -> actual memory used during the execution',
external: '524.91 MB -> V8 external memory'
}
Why does the heap get used here instead of in the first function call without an encoding? I don't even have to store the buffer in a local variable for the heap to be used. I also understand that after the next event loop tick in the second example the heap will be cleaned up. But this leads me to my next question in Part 2
Part 2) This part is the same as part 1 but with streams.
const readStream = fs.createReadStream('path/to/500MB');
let data;
readStream.on('data', (buffer) => {
data+= buffer;
});
readStream.on('close', () => {
console.log(printMemoryData());
});
I get the output:
{
rss: '574.57 MB -> Resident Set Size - total memory allocated for the process execution',
heapTotal: '692.75 MB -> total size of the allocated heap',
heapUsed: '508.72 MB -> actual memory used during the execution',
external: '7.97 MB -> V8 external memory'
}
Why the difference in behavior with streams in heap used in part 2 vs the first function without encoding in part 1?
They both have an increase in RSS, but only in the streams does the heap get used when I store the buffer in a local variable/v8 object.
Thanks for any feedback.
I develope node js with mongodb application on windows. For memory check, I usually use windows task manager, but it is not an good option, I think.
How to check exact memory usage of mongodb queries? I can aggregate all needed data in one query, but maybe 2 projection will be better option.
You can use simple one line code using Object.entries() and adjust to readable format
Object.entries(process.memoryUsage()).forEach(item => console.log(`${item[0]}: ${(item[1] / 1024 / 1024).toFixed(4)} MB`))
output:
rss: 70.7695 MB
heapTotal: 85.4063 MB
heapUsed: 55.2614 MB
external: 0.0794 MB
you can use the built in memoryUsage method. The usage will be displayed in bytes.
Here is a simple example:
let usage = process.memoryUsage()
for (var i = 0; i < 4; i++) {
console.log(`${Object.keys(usage)[i]}: ${Object.values(usage)[i]} bytes`)
}
example log:
rss: 634034 bytes
heapTotal: 340239 bytes
heapUsed: 129323 bytes
external: 10232 bytes
I'm loading a big js module into memory and I want to release it when is no longer needed to free RAM.
The code I am using to test is like this:
var lex = require('./lex.js'); //Big module (10M array)
setInterval(() => console.log(process.memoryUsage()), 1000);
setTimeout(() => {
lex = null;
delete require.cache[require.resolve('./lex.js')];
}, 5000);
// this script outputs each second
// { rss: 151756800, heapTotal: 131487520, heapUsed: 108843840 }
// { rss: 151760896, heapTotal: 131487520, heapUsed: 108850024 }
// ...
// and after 5 seconds there is no change
After 5 seconds the process is still using the same memory as after the initial module load.
What am I doing wrong? Thank you!
Delete Require cache will help you to load once again the content not from the cache ,as of me it will not delete or free up your memory
Quted from What do the return values of node.js process.memoryUsage() stand for? RSS is the resident set size, the portion of the process's memory held in RAM
(how much memory is held in the RAM by this process in Bytes) file size of 'text.txt' used in example is here is 370KB (378880 Bytes)
var http = require('http');
var fs = require('fs');
var express = require('express');
var app = express();
console.log("On app bootstrap = ", process.memoryUsage());
app.get('/test', function(req, res) {
fs.readFile(__dirname + '/text.txt', function(err, file) {
console.log("When File is available = ", process.memoryUsage());
res.end(file);
});
setTimeout(function() {
console.log("After sending = ", process.memoryUsage());
}, 5000);
});
app.listen(8081);
So on app bootstrap: { rss: 22069248, heapTotal: 15551232, heapUsed: 9169152 }
After i made 10 request for '/test' situation is:
When File is available = { rss: 33087488, heapTotal: 18635008, heapUsed: 6553552 }
After sending = { rss: 33447936, heapTotal: 18635008, heapUsed: 6566856 }
So from app boostrap to 10nth request rss is increased for 11378688 bytes which is roughly 30 times larger than size of text.txt file.
I know that this code will buffers up the entire data.txt file into memory for every request before writing the result back to clients, but i expected that after the requst is finished occupied memory for 'text.txt' will be released? But that is not the case?
Second how to set up maximum size of RAM memory which node process can consume?
In js garbage collector does not run immediately after execution of your code. Thus the memory is not freed immediately after execution. You can run GC independently, after working with large objects, if you care about memory consumption. More information you can find here.
setTimeout(function() {
global.gc();
console.log("After sending = ", process.memoryUsage());
}, 5000);
To look at your memory allocation you can run your server with v8-profiler and get a Heap snapshot. More info here.
Try running your example again and give the process some time to run garbage collection. Keep an eye on your process' memory usage with a system monitor and it should clear after some time. If it doesn't go down the process can't go higher in memory usage than mentioned below.
According to the node documentation the memory limit is 512 mb for 32 bit and 1 gb for 64 bit. They can be increased if necessary.
Running this:
setInterval(function() {
console.log(process.memoryUsage());
}, 1000);
shows memory usage constantly growing:
{ rss: 9076736, heapTotal: 6131200, heapUsed: 2052352 }
... some time later
{ rss: 10960896, heapTotal: 6131200, heapUsed: 2939096 }
... some time later
{ rss: 11177984, heapTotal: 6131200, heapUsed: 3141576 }
Why does this happen?
The program isn't empty, it's running a timer, checking memory usage, and writing to the console once a second. That causes objects to be created in the heap and until garbage collection runs your heap usage will keep increasing.
If you let it go you should eventually see heapUsed go back down each time garbage collection is triggered.