Node.js | JavaScript heap out of memory, but HEAP is only 1GB - node.js

I get this error:
FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
but I'm using --max-old-space-size=12288 and I have 16GB of memory
Using process.memoryUsage() I can see the heap only goes up to ~1GB;
Also, the error always happens at the same line of code, namely types.js:~94, which is part of a library I'm using.
This makes me thing that there's a bug somewhere, maybe in the argument I pass to the function, but I can't figure out how to spot it.
What should I look for?
The full error is:
<--- Last few GCs --->
[28425:0x2739620] 406975 ms: Scavenge 3689.5 (3833.4) -> 3673.7 (3833.9) MB, 4.2 / 0.0 ms (average mu = 0.991, current mu = 0.990) allocation failure
[28425:0x2739620] 407874 ms: Scavenge 3728.8 (3873.4) -> 3713.1 (3873.9) MB, 4.5 / 0.0 ms (average mu = 0.991, current mu = 0.990) allocation failure
[28425:0x2739620] 408785 ms: Scavenge 3768.1 (3913.4) -> 3752.4 (3913.9) MB, 4.5 / 0.0 ms (average mu = 0.991, current mu = 0.990) allocation failure
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x2a42914841bd]
Security context: 0x281401b9e6c9 <JSObject>
1: decode [0x1c8f15ad00a9] [/media/Data/dev/btc-crawler/node_modules/bitcoin-protocol/src/types.js:~94] [pc=0x2a429178f88b](this=0x1c8f15acb299 <Object map = 0x736c200e801>,buffer=0xeeb6cf17c29 <Uint8Array map = 0x3d1fa97d04e9>,offset=4,end=0x25b4eaf822e1 <undefined>)
2: _transform [0x38503cda5d21] [/me...
FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
1: 0x89c2f0 node::Abort() [node]
2: 0x89c33c [node]
3: 0xa8f05e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xa8f278 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xe7cad2 [node]
6: 0xf88847 v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [node]
7: 0xf9a3c8 v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handle<v8::internal::StringTable>, int, v8::internal::PretenureFlag) [node]
8: 0xf9a58a v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::internal::StringTableKey*) [node]
9: 0xfa03f4 v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [node]
10: 0xf678aa v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [node]
11: 0x1112512 in the code, v8::internal::Runtime::GetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*) [node]
12: 0x1112849 [node]
13: 0x1114f09 v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x2a42914841bd
Line 94 of types.js is the entry point of the decode function, defined as follows:
var buffer12 = struct.Buffer(12)
function decode (buffer, offset, end) {
var bvalue = buffer12.decode(buffer, offset, end)
for (var stop = 0; bvalue[stop] !== 0; ++stop);
for (var i = stop; i < bvalue.length; ++i) {
if (bvalue[i] !== 0) throw new Error('Found a non-null byte after the first null byte in a null-padded string')
}
return bvalue.slice(0, stop).toString('ascii')
}

You have a typo!
The correct usage is node --max-old-space-size=4096 somescript.js
4096 is 4gb in this case, adjust it accordingly in your case...
Run node --v8-options for a list of all valid v8 options,
or check this list of v8 flags

Related

Nodejs outof memory when reading big file

I have the following code to stream a file:
const lineReader = readline.createInterface({
input: fs.createReadStream(filePath),
});
lineReader.on('line', (line) => {
console.log(line);
LineEntity.create({line: "some data"});//write some data to db
});
when executing the above code, I'm getting out of memory exception after 30-60 seconds.
when I execute the following, it runs perfectly without stopping...
//just to show that the problem is not with accessing the database
while(true){
await LineEntity.create({line: "some data"});//and without await
}
when I execute the following, it runs perfect also.
lineReader.on('line', (line) => {
console.log(line);
});
The problem is with the combination of the file reading and writing to the db.
Separately, they run perfect. together I'm getting out of memory exception
The error:
<--- Last few GCs --->
[3903:0x47b4180] 42294 ms: Mark-sweep 2032.1 (2072.5) -> 2028.0 (2053.4) MB, 161.9 / 0.0 ms (+ 602.9 ms in 141 steps
since start of marking, biggest step 6.3 ms, walltime since start of marking 838 ms) (average mu = 0.205, current mu =
0.172) finalize[3903:0x47b4180] 43117 ms: Mark-sweep 2034.1 (2053.4) -> 2024.0 (2067.7) MB, 805.3 / 0.0 ms (average
mu = 0.120, current mu = 0.020) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x140dcd9]
1: StubFrame [pc: 0x148be23]
Security context: 0x05a03e5408d1 <JSObject>
2: save [0x1f10ab61b579] [/home/ron/work/hawking/node_modules/sequelize/lib/model.js:3950] [bytecode=0x22c0629957a1
offset=2439](this=0x02d3eccd2f21 <Model map = 0x2e79bc5fda69>,0x02d3eccd2ee9 <Object map = 0x4a68f580439>)
3: /* anonymous */ [0x2b6513fc9bf9](this=0x2ccc7e8c2b21 <JSGlobal Object>,0x1460087004b1 <undefined>)...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa1a640 node::Abort() [node]
2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
3: 0xb9a68e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb9aa09 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd57c85 [node]
6: 0xd58316 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
7: 0xd64bd5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
8: 0xd65a85 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8
::GCCallbackFlags) [node]
9: 0xd6853c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationO
rigin, v8::internal::AllocationAlignment) [node]
What am I doing wrong?
Thanks

Replit console error <--- Last few GCs ---> and <--- JS stacktrace --->

So I was using my bot but since it didn't respond, I decided to check why it was offline. I opened up my code and I found this error in the console. I just wanna know what it means. Here :
<--- Last few GCs --->
[880:0x4b70e20] 26014144 ms: Mark-sweep 234.2 (257.5) -> 234.2 (257.8) MB, 2824.5 / 0.0 ms (average mu = 0.148, current mu = 0.008) allocation failure scavenge might not succeed
[880:0x4b70e20] 26017038 ms: Mark-sweep 234.6 (257.8) -> 234.5 (257.8) MB, 2888.0 / 0.0 ms (average mu = 0.081, current mu = 0.002) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1f02f9e808d1 <JSObject>
0: builtin exit frame: parse(this=0x1f02f9e9ee79 <Object map = 0x1cdd4fa03639>,0x15196d9c0119 <Very long string[8081560]>,0x1f02f9e9ee79 <Object map = 0x1cdd4fa03639>)
1: onMessage [0x118b705220e9] [/home/runner/New-Neptune/node_modules/discord.js/src/client/websocket/WebSocketShard.js:~278] [pc=0x2bb58926f89e](this=0x3f8662a33761 <EventEmitter map = 0x2d550d1ce3a9>,0x1166c1f7f291 <Even...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa1a640 node::Abort() [node]
2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
3: 0xb9a68e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb9aa09 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd57c85 [node]
6: 0xd58316 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
7: 0xd64bd5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
8: 0xd65a85 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0xd6853c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd363dc v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [node]
11: 0xe4561b v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [node]
12: 0xe47e9e v8::internal::JsonParser<unsigned short>::ParseJsonValue() [node]
13: 0xe486af v8::internal::JsonParser<unsigned short>::ParseJson() [node]
14: 0xc4bc05 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
15: 0x140df99 [node]

JavaScript heap out of memory
It ran out of memory.
Without seeing your code, it's impossible to say why in particular, but seeing Builtin_JsonParse in the stack means it was probably parsing a big JSON response from somewhere.
In my case, I was pre autopopulating my document with fields that reference see here its own model...When I tried to hit my API with new data it was working fine like in my case when I was logging in with new signup data it was working fine but with old login data, it was running an infinite loop due to autopopulate which further caused heap out of memory issue..I just realized when I turned on mongoose debugger....After removing those autopopulates my application is working fine
removing autopopulate
Then I removed these autopopulate statements from my code and now it's working fine

node canvas getImageData() out of memory

I have a nodejs application that transforms images.
I run my application in pm2 with 32gb of ram.
When trying to get the image data of an image i am getting this error.
I have tried to increase the node js memory limit using --max-old-space-size
tried to increase it to 8192 as well as 32678 but it I still get an error.
m[9161:0x41069d0] 18179 ms: Mark-sweep 22.3 (29.2) -> 22.0 (25.2) MB, 23.0 / 0.0 ms (+ 1.5 ms in 5 steps since start of marking, biggest step 0.8 ms, walltime since start of marking 55 ms) (average mu = 0.998, current mu = 0.963) finalize incremental m[9161:0x41069d0] 27013 ms: Mark-sweep 23.2 (25.2) -> 22.2 (25.2) MB, 3.8 / 0.0 ms (+ 0.1 ms in 2 steps since start of marking, biggest step 0.1 ms, walltime since start of marking 168 ms) (average mu = 0.999, current mu = 1.000) finalize incremental m
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x21639e1c08d1 <JSObject>
0: builtin exit frame: getImageData(this=0x0529133cc801 <CanvasRenderingContext2D map = 0x381c66f8e359>,32150,22958,0,0,0x0529133cc801 <CanvasRenderingContext2D map = 0x381c66f8e359>)
1: getPoints [0x324589ad9b31] [/var/apps/print-export-worker/getPoints.js:11] [bytecode=0x340f496a0d29 offset=86](this=0x2d04988cd319 <JSGlobal Object>,0x0529133cc801 <CanvasRenderingContext2D map = 0x381c...
FATAL ERROR: v8::ArrayBuffer::New Allocation failed - process out of memory
1: 0xa09830 node::Abort() [node]
2: 0xa09c55 node::OnFatalError(char const*, char const*) [node]
3: 0xb7d71e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb7da99 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xb7db7b [node]
6: 0xb9b93e [node]
7: 0x7f7f87de455f Context2d::GetImageData(Nan::FunctionCallbackInfo<v8::Value> const&) [/var/apps/print-export-worker/node_modules/canvas/build/Release/canvas.node]
8: 0x7f7f87dd91f9 [/var/apps/print-export-worker/node_modules/canvas/build/Release/canvas.node]
9: 0xbe96a9 [node]
10: 0xbeb497 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x13c5c79 [node]
Aborted
It turns out that all I needed to do was to upgrade my node version to 14Lts. There was some limitation in version 12.
The other way of getting the image data from an image without using node-canvas was using sharp like this.
sharp(canvas.toBuffer(), { limitInputPixels: false })
.raw()
.ensureAlpha()
.toBuffer()
.then((img) => {
let imageData = new Uint8ClampedArray(img)
return imageData
})
.catch(err => {
console.log('ERROR IN SHARP', err)
reject(err)
})
It has to use limitInputPixels into false if you are working with large images.

AWS EC2 Nodejs | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory in vuejs project

I am running a nodejs app on AWS ec2 instance(t2.small). After it works around one week. Below out of memory error comes. Searched for same issues, found there is a workaround to increase "--max-old-space-size" option. However, my question is
if i set --max-old-space-size = 4096(4gb). Do i need to update the aws instance to get more memory(as small type only have 2 gb memory)
This max-old-space-size setting,finally will it also hit memory limitation? At the beginning, the app is working fine, but after period of time, app is crushed. If that, how to slove this issue effectively?
The full error message as below:
<--- Last few GCs --->
[30600:0x5007800] 429029005 ms: Mark-sweep 977.8 (1007.7) -> 970.6 (1008.9) MB, 2379.3 / 0.5 ms (average mu = 0.220, current mu = 0.190) allocation failure scavenge might n
ot succeed
[30600:0x5007800] 429031937 ms: Mark-sweep 979.0 (1008.9) -> 971.8 (1010.4) MB, 2396.6 / 0.4 ms (average mu = 0.202, current mu = 0.183) allocation failure scavenge might n
ot succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa02ec0 node::Abort() [node]
2: 0x94e471 node::FatalError(char const*, char const*) [node]
3: 0xb765de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb76957 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd311f5 [node]
6: 0xd31d7f [node]
7: 0xd3fe0b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
8: 0xd439cc v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [no
de]
9: 0xd120ab v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x105802f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x13fdfd9 [node]
./backend.sh: line 6: 30600 Aborted node server.js
Thanks for all your help!

Trying to make a duel feature in Discord bot but while loop is causing FATAL error

So I have a discord bot and I want to make it be able to duel with players. However I kept running into a FATAL ERROR, which I figured was from the while loop. I'm kind of new at this, so is there some sort of rule against using while loop in discord.js? I've been stuck at this for days and I tried googling but I just confused myself even more.
my code:
msg.channel.send(`${taggedUser}, ${msg.guild.members.cache.get(msg.author.id).displayName} wants to duel you. Do you accept? (Reply yes or no.)`);
msg.channel.awaitMessages(m => m.author.id == taggedUser.id, {max: 1, time: 10000})
.then(collected => {
if (collected.first().content.toLowerCase() == 'yes') {
msg.channel.send("Let the battle begin!");
p1points = 100;
p2points = 100;
while (p1points !== 0 || p2points !== 0) {
const randomOption = results[Math.floor(Math.random() * results.length)];
msg.channel.send(`**${player1}** casted ${randomOption.message} (∩•̀ω•́)⊃-⋆`);
p2points = p2points + randomOption.xp;
msg.channel.send(`**${player1}**: ${p1points}, **${player2}**: ${p2points}`);
const randomOption2 = results[Math.floor(Math.random() * results.length)];
msg.channel.send(`**${player2}** casted ${randomOption2.message} ⋆-⊂(•̀ω•́∩)`);
p1points = p1points + randomOption2.xp;
msg.channel.send(`**${player1}**: ${p1points}, **${player2}**: ${p2points}`);
}
if (p1points === 0) {
msg.channel.send(`**${player1}** is DEFEATED. AAAAAAh! (ノ゚Д゚)八(゚Д゚)八(゚Д゚)ノィェーィ!`);
msg.channel.send(`CONGRATULATIONS, ${player2}! (っ◕‿◕)っ :heart:`);
}
if (p2points === 0) {
msg.channel.send(`**${player2}** is DEFEATED. AAAAAAh! (ノ゚Д゚)八(゚Д゚)八(゚Д゚)ノィェーィ!`);
msg.channel.send(`CONGRATULATIONS, ${player1}! (っ◕‿◕)っ :heart:`);
}
}
else if (collected.first().content.toLowerCase() == 'no') {
msg.channel.send("Yikes, are you scared? Maybe next time then!");
}
}).catch(() => {
msg.channel.send('No reply after 10 seconds, duel is cancelled!');
});
Error:
<--- Last few GCs --->
[4:0x3778960] 21250 ms: Mark-sweep 254.7 (258.3) -> 254.2 (258.5) MB, 201.4 / 0.0 ms (average mu = 0.119, current mu = 0.012) allocation failure scavenge might not succeed
[4:0x3778960] 21556 ms: Mark-sweep 255.0 (258.5) -> 254.5 (258.8) MB, 303.1 / 0.0 ms (average mu = 0.059, current mu = 0.012) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x13c5b79]
1: StubFrame [pc: 0x13c6a4d]
Security context: 0x26d9b2a808d1 <JSObject>
2: /* anonymous */ [0x2c9d420f9409] [internal/modules/cjs/loader.js:~605] [pc=0x270b156db629](this=0x2c9d420c0799 <JSFunction Module (sfi = 0x26d9b2ab0d21)>,0x15d111435a01 <String[#7]: ../User>,0x0ac218d43aa1 <JSArray[1]>,0x18752e1c06e9 <false>)
3: /* anonymous */ [0x2c9d420f69f1] [internal/modules/cjs/loader.js:~892] [pc=...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Writing Node.js report to file: report.20200522.160742.4.0.001.json
Node.js report completed
1: 0xa09830 node::Abort() [node]
2: 0xa09c55 node::OnFatalError(char const*, char const*) [node]
3: 0xb7d71e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb7da99 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xd2a1f5 [node]
6: 0xd2a886 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
7: 0xd37105 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
8: 0xd37fb5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0xd3aa6c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd0163b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x104300e v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x13c5b79 [node]
Read the full error, specifically this line
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Looking for JavaScript heap out of memory brings up some answers like this answer and this one
I made a small guess out of what I see going on in the code you provided.
First, you need to make sure you aren't mixing up your operators. In your snippet example, the way you are using the or operator is telling the while loop to execute until both player's points equal exactly 0. And since you are only adding to the player points and not subtracting the points will never equal 0 unless the .xp property is a negative number.
Secondly, for those times where a players points can end up being less than 0 then I'd recommend doing
while (p1points > 0 && p2points > 0) {
instead of
while (p1points !== 0 && p2points !== 0) {
because if you use > 0 then it'll stop executing whenever one of the players points are exactly 0 or less than 0.

Resources