NodeJs - near heap limit Allocation failed - JavaScript heap out of memory - node.js

Question

I have a node js application, running in AWS EC2 as m5xlarge instance and ubuntu 18.04 OS, which has a main.js file and in main file I am using node-cron to schedule multiple cron jobs,once the jobs are scheduled starting the application using another file app.js, in an intermittent way I am facing out of memory error and server stops the logs are shown as follows -
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: node::Abort() [node /home/ubuntu/XXXXXX/main.js]
2: 0x89371c [node /home/ubuntu/XXXXXX/main.js]
3: v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node /home/ubuntu/XXXXXX/main.js]
4: v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node /home/ubuntu/XXXXXX/main.js]
5: 0xe617e2 [node /home/ubuntu/XXXXXX/main.js]
6: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node /home/ubuntu/XXXXXX/main.js]
7: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/ubuntu/XXXXXX/main.js]
8: v8::internal::Heap::AllocateRawWithRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node /home/ubuntu/XXXXXX/main.js]
9: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node /home/ubuntu/XXXXXX/main.js]
10: v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node /home/ubuntu/XXXXXX/main.js]
11: 0x2a583f5041bd
The memory utilization and health check monitors are as follows -
The spikes are due to cron job running on interval the highest ones are hourly cron job.
Now my assumption is it could be due to any of cron job failing with out of memory error in main.js, and hence the application in app.js also stops, or application itself failing in app.js the cron job scheduling looks like below -
const cluster = require('cluster');
const numCPUs = 1 //require('os').cpus().length;
var CronJob = require('cron').CronJob;
const spawn = require('child_process').spawn;
require('dotenv').config()
if (cluster.isMaster) {
if (process.env.kEnvironment == "dev") {
var sampleCron = new CronJob('00 */10 * * * *', function () {
spawn(process.execPath, ['./sampleCron.js'], {
stdio: 'inherit'
})
}, null, true, null);
} else {
var sampleCron = new CronJob('00 15 10 * * 0', function () {
spawn(process.execPath, ['./sampleCron.js'], {
stdio: 'inherit'
})
}, null, true, null);
}
// There are multiple cron like the above
for (let i = 0; i < numCPUs; i++) {
cluster.fork();
}
cluster.on('exit', (worker, code, signal) => {
console.log(`worker ${worker.process.pid} died`);
});
} else {
require('./app.js');
}
Here is a htop preview about this -
couple of questions here -
- Why it shows main.js tree inside main.js, is that kind of nesting normal?
- If it is same why the resource memory utilization differs for both?
I tried to increase the memory for each cron as below -
var sampleCron = new CronJob('00 15 10 * * 0', function () {
spawn(process.execPath, ['./sampleCron.js','--max-old-space-size=4096'], {
stdio: 'inherit'
})
}, null, true, null);
But it still fails. My questions are as follows -
How do I isolate the problem, is it really due to crons or due to application.
How do I solve the problem?

Answer 1

Use commands like "Top" to find out how much memory the node process is using. I think the node script does not use all the available memory. you can also try allocating more memory using the NODE_OPTIONS.
For e.g node SomeScript.js --max-old-space-size=8192