Node js memory heap out on file upload - node.js

I am trying to upload a file sized 200MB using angular5 node js. Every time the app crashed with following error.
==== JS stack trace =========================================
Security context: 0x2c3a92b25ee1 <JSObject>
2: SimpleSlice(aka SimpleSlice) [native array.js:1] [bytecode=0x2c3aa14ed689 offset=41](this=0x2c3ad9b02311 <undefined>,p=0x2c3a51d2e961 <Uint8Array map = 0x2c3ad7ec1d91>,O=0,P=217330529,Q=217330529,R=0x2c3a51d6bce1 <JSArray[217330529]>)
4: ArraySliceFallback [native array.js:1] [bytecode=0x2c3aa14ed021 offset=281](this=0x2c3a51d2e961 <Uint8Array map = 0x2c3ad7ec1d91>,at=0,au=0x2c3ad9b02...
FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
4: v8::internal::HashTable<v8::internal::SeededNumberDictionary, v8::internal::SeededNumberDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::SeededNumberDictionary>, int, v8::internal::PretenureFlag) [/usr/local/bin/node]
5: v8::internal::Dictionary<v8::internal::SeededNumberDictionary, v8::internal::SeededNumberDictionaryShape>::Add(v8::internal::Handle<v8::internal::SeededNumberDictionary>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [/usr/local/bin/node]
6: v8::internal::(anonymous namespace)::DictionaryElementsAccessor::AddImpl(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, unsigned int) [/usr/local/bin/node]
7: v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow) [/usr/local/bin/node]
8: v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::JSObject::AccessorInfoHandling) [/usr/local/bin/node]
9: v8::internal::JSObject::CreateDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::ShouldThrow) [/usr/local/bin/node]
10: v8::internal::Runtime_CreateDataProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0x12f2c16c6838
I have tried different techniques but all in vein.
--max-old-space-size=80896 --trace-gc-verbose
app.use(bodyParser.json({limit: '8096mb'}));
In activity monitoring when node grab more than 1GB the crash happen.
My development environment is mac with 8 GB ram.

Related

I am doing an api call and the client is returning a response so big that node crashes

This is the error I am dealing with
#
# Fatal error in , line 0
# Check failed: i::kMaxInt >= len.
#
#
#
#FailureMessage Object: 0x7ffd4a155b80
1: 0xb6d391 [node]
2: 0x1bf6a04 V8_Fatal(char const*, ...) [node]
3: 0xcfcd77 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [node]
4: 0xbeb719 [node]
5: 0xadb345 [node]
6: 0xd4a82e [node]
7: 0xd4bc4f v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
8: 0x15e7dd9 [node]
Any assitance here would be greatly appreciated.
Ive tried paging size which does fix this problem because this is on an individual record

Node js crashes with native exception

I am getting unexpected exits (exit code 1) - and stalls ie promises never fulfilled - in a node application. Most of the times this happens, there is no error visible, however, I have seen the following stack trace on one occasion:
#
# Fatal error in , line 0
# Check failed: fixed_size_above_fp + (stack_slots * kSystemPointerSize) - CommonFrameConstants::kFixedFrameSizeAboveFp + outgoing_size == result.
#
#
#
#FailureMessage Object: 0x7ffeefbfc9e0
1: 0x100120e62 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/usr/local/bin/node]
2: 0x10103af53 V8_Fatal(char const*, ...) [/usr/local/bin/node]
3: 0x10030a633 v8::internal::Deoptimizer::Deoptimizer(v8::internal::Isolate*, v8::internal::JSFunction, v8::internal::DeoptimizeKind, unsigned int, unsigned long, int) [/usr/local/bin/node]
4: 0x10030839e v8::internal::Deoptimizer::New(unsigned long, v8::internal::DeoptimizeKind, unsigned int, unsigned long, int, v8::internal::Isolate*) [/usr/local/bin/node]
5: 0x34a8fad8216d
6: 0x1006c5e23 v8::internal::NativeRegExpMacroAssembler::Execute(v8::internal::String, int, unsigned char const*, unsigned char const*, int*, int, v8::internal::Isolate*, v8::internal::JSRegExp) [/usr/local/bin/node]
7: 0x1006c5d66 v8::internal::NativeRegExpMacroAssembler::Match(v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int*, int, int, v8::internal::Isolate*) [/usr/local/bin/node]
8: 0x1006d22ad v8::internal::RegExpImpl::IrregexpExecRaw(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, int*, int) [/usr/local/bin/node]
9: 0x1006d2967 v8::internal::RegExpGlobalCache::FetchNext() [/usr/local/bin/node]
10: 0x100725666 v8::internal::Runtime_RegExpExecMultiple(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0x100a81fb9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
12: 0x100ad1c5a Builtins_RegExpReplace [/usr/local/bin/node]
13: 0x100a72e52 Builtins_StringPrototypeReplace [/usr/local/bin/node]
I'm not sure quite what to make of this, as the exits appear to happen at different points. Does the above seem likely to be a memory problem?

Unexpected error during building next.js project

When I try to build my next.js project I am getting this error:
> next build
info - Creating an optimized production build...
info - Compiled successfully
info - Collecting page data...
info - Generating static pages (0/5)
info - Generating static pages (1/5)
info - Generating static pages (2/5)
info - Generating static pages (3/5)
info - Generating static pages (5/5)
info - Finalizing page optimization...
FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope
1: 0xa89e60 node::Abort() [node]
2: 0x9ade29 node::FatalError(char const*, char const*) [node]
3: 0xc7555a v8::Utils::ReportApiFailure(char const*, char const*) [node]
4: 0xdfdde2 v8::internal::HandleScope::Extend(v8::internal::Isolate*) [node]
5: 0xe10b7c v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
6: 0xe22b8e v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector<unsigned char const> const&, v8::internal::AllocationType) [node]
7: 0xc8cf72 v8::String::NewFromOneByte(v8::Isolate*, unsigned char const*, v8::NewStringType, int) [node]
8: 0xc3a317 node::crypto::TLSWrap::InvokeQueued(int, char const*) [node]
9: 0xc3a422 [node]
10: 0xc3a5e4 node::crypto::TLSWrap::~TLSWrap() [node]
11: 0xc3a751 node::crypto::TLSWrap::~TLSWrap() [node]
12: 0xa20c5b node::Environment::RunCleanup() [node]
13: 0x9d670c node::FreeEnvironment(node::Environment*) [node]
14: 0xb52249 node::worker::Worker::Run() [node]
15: 0xb529c8 [node]
16: 0x7ff424e31609 [/lib/x86_64-linux-gnu/libpthread.so.0]
17: 0x7ff424d58293 clone [/lib/x86_64-linux-gnu/libc.so.6]
Aborted
-----> Build failed
Node version: 15.14.0
Npm version: 7.7.6
I do not know why this is failing. It works randomly on local machine an always fails on Heroku. I can't find any solution or related problem.

Why do I keep getting an Out-Of-Memory error with Redux-Persist?

Occasionally when rebooting my app, my emulator throws this error
Accompanied by this stacktrace
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/bin/node]
7: v8::String::WriteUtf8(char*, int, int*, int) const [/usr/local/bin/node]
8: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [/usr/local/bin/node]
9: int node::StreamBase::WriteString<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
10: void node::StreamBase::JSMethod<node::LibuvStreamWrap, &(int node::StreamBase::WriteString<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&))>(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
11: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/local/bin/node]
12: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
13: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x2672e7c842fd
events.js:183
throw er; // Unhandled 'error' event
^
Error: write EPIPE
at WriteWrap.afterWrite [as oncomplete] (net.js:868:14)
/Users/jisata0/Dev/exampleapp/node_modules/react-native/scripts/packager.sh: line 11: 86698 Abort trap: 6 node "./local-cli/cli.js" start "$#"
Process terminated. Press <enter> to close the window
Can anyone provide some clarity on why this might be happening?
A client is required to review the basic application response to performance improvements. One of the most common causes of performance problems is memory leak.
Out of memory is an error that occurs when the limit of available memory is exceeded.
The simplest way to apply it is to have more memory available to the app.
You can use Add to android / app / src / main / AndroidManifest.xml.
<application android:largeHeap="true"
If you still have out of memory, try debugging any code you've written that uses more memory than you intended or creates a memory lock.

Do I just need an EC2 instance with more memory or does this error suggest issues in my code?

The short answer is "of course there are problems in my code," but for the time being, I'm wondering if the error messages I'm receiving can help me fix a specific problem. My app is running on sails, using Node v8.1.2 running on an EC2 instance modified from this Bitnami image. The instance type is t2.small (2GB memory), which I should probably upgrade, in any case.
Is this an error message I should review closely and use to find bugs? Or should I just switch to a larger instance and hope for the best? Thanks in advance.
bitnami#ip-172-31-17-108:~$ sudo forever stopall
^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A
<--- Last few GCs --->
[5547:0x3ec5a50] 111819 ms: Scavenge 6.7 (10.0) -> 6.3 (10.5) MB, 153.7 / 2.2 ms allocation failure
[5547:0x3ec5a50] 130817 ms: Scavenge 7.3 (10.5) -> 6.8 (11.0) MB, 207.3 / 9.4 ms allocation failure
[5547:0x3ec5a50] 164550 ms: Scavenge 7.8 (11.0) -> 7.3 (11.5) MB, 155.3 / 8.0 ms allocation failure
[5547:0x3ec5a50] 203571 ms: Scavenge 8.2 (11.5) -> 7.6 (11.5) MB, 317.2 / 42.5 ms allocation failure
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x9d40a2a9891 <JS Object>
1: patch [/opt/bitnami/nodejs/lib/node_modules/forever/node_modules/graceful-fs/polyfills.js:64] [pc=0x426fa5d7cb1](this=0x38dbb211729 <JS Global Object>,fs=0xfd51435b869 <an Object with deprecated map 0x4376127ec19>)
2: patch [/opt/bitnami/nodejs/lib/node_modules/forever/node_modules/graceful-fs/graceful-fs.js:58] [pc=0x426fa5d6791](this=0x38dbb211729 <JS Global Object>,fs=0xfd51435b869...
FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
1: node::Abort() [/opt/bitnami/nodejs/bin/.node.bin]
2: 0x13647ec [/opt/bitnami/nodejs/bin/.node.bin]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/opt/bitnami/nodejs/bin/.node.bin]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/opt/bitnami/nodejs/bin/.node.bin]
5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/bitnami/nodejs/bin/.node.bin]
6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/opt/bitnami/nodejs/bin/.node.bin]
7: v8::internal::Factory::NewMap(v8::internal::InstanceType, int, v8::internal::ElementsKind) [/opt/bitnami/nodejs/bin/.node.bin]
8: v8::internal::Map::RawCopy(v8::internal::Handle<v8::internal::Map>, int) [/opt/bitnami/nodejs/bin/.node.bin]
9: v8::internal::Map::CopyDropDescriptors(v8::internal::Handle<v8::internal::Map>) [/opt/bitnami/nodejs/bin/.node.bin]
10: v8::internal::Map::AddMissingTransitions(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>) [/opt/bitnami/nodejs/bin/.node.bin]
11: v8::internal::MapUpdater::ConstructNewMap() [/opt/bitnami/nodejs/bin/.node.bin]
12: v8::internal::MapUpdater::ReconfigureToDataField(int, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::Handle<v8::internal::FieldType>) [/opt/bitnami/nodejs/bin/.node.bin]
13: 0xffc3a0 [/opt/bitnami/nodejs/bin/.node.bin]
14: v8::internal::Map::PrepareForDataProperty(v8::internal::Handle<v8::internal::Map>, int, v8::internal::PropertyConstness, v8::internal::Handle<v8::internal::Object>) [/opt/bitnami/nodejs/bin/.node.bin]
15: v8::internal::LookupIterator::PrepareForDataProperty(v8::internal::Handle<v8::internal::Object>) [/opt/bitnami/nodejs/bin/.node.bin]
16: v8::internal::Object::SetDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>) [/opt/bitnami/nodejs/bin/.node.bin]
17: v8::internal::Object::SetPropertyInternal(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed, bool*) [/opt/bitnami/nodejs/bin/.node.bin]
18: v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [/opt/bitnami/nodejs/bin/.node.bin]
19: v8::internal::StoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [/opt/bitnami/nodejs/bin/.node.bin]
20: v8::internal::Runtime_StoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/opt/bitnami/nodejs/bin/.node.bin]
21: 0x426fa38437d
bitnami#ip-172-31-17-108:~$ <--- JS stacktrace --->^C
bitnami#ip-172-31-17-108:~$ cd my-sails-api/
bitnami#ip-172-31-17-108:~/my-sails-api$ sudo sails lift
<--- Last few GCs --->
[5562:0x2739a40] 79195 ms: Scavenge 5.7 (10.1) -> 5.6 (9.6) MB, 304.7 / 8.1 ms allocation failure
[5562:0x2739a40] 98188 ms: Scavenge 6.6 (9.6) -> 5.9 (10.1) MB, 252.5 / 24.5 ms allocation failure
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x25f39c629891 <JS Object>
2: constructor(aka Duplex) [_stream_duplex.js:48] [pc=0x178f7677eb54](this=0x20617eab9791 <a WriteStream with map 0x1d0c9952641>,options=0x20617eab98e9 <an Object with map 0x1d0c9952329>)
3: constructor(aka Socket) [net.js:197] [pc=0x178f7677de42](this=0x20617eab9791 <a WriteStream with map 0x1d0c9952641>,options=0x20617eab98e9 <an Object with map 0x1d0c9952329>)
4: new constructor(...
FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
1: node::Abort() [/opt/bitnami/nodejs/bin/.node.bin]
2: 0x13647ec [/opt/bitnami/nodejs/bin/.node.bin]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/opt/bitnami/nodejs/bin/.node.bin]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/opt/bitnami/nodejs/bin/.node.bin]
5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/bitnami/nodejs/bin/.node.bin]
6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/opt/bitnami/nodejs/bin/.node.bin]
7: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [/opt/bitnami/nodejs/bin/.node.bin]
8: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::MacroAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [/opt/bitnami/nodejs/bin/.node.bin]
9: v8::internal::FullCodeGenerator::MakeCode(v8::internal::CompilationInfo*, unsigned long) [/opt/bitnami/nodejs/bin/.node.bin]
10: v8::internal::FullCodegenCompilationJob::ExecuteJobImpl() [/opt/bitnami/nodejs/bin/.node.bin]
11: v8::internal::CompilationJob::ExecuteJob() [/opt/bitnami/nodejs/bin/.node.bin]
12: 0xd97d90 [/opt/bitnami/nodejs/bin/.node.bin]
13: 0xd98b28 [/opt/bitnami/nodejs/bin/.node.bin]
14: 0xd98d8f [/opt/bitnami/nodejs/bin/.node.bin]
15: 0xd9ce59 [/opt/bitnami/nodejs/bin/.node.bin]
16: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [/opt/bitnami/nodejs/bin/.node.bin]
17: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [/opt/bitnami/nodejs/bin/.node.bin]
18: 0x178f7658437d
bitnami#ip-172-31-17-108:~/my-sails-api$
bitnami#ip-172-31-17-108:~/my-sails-api$ sudo sails lift
-bash: fork: Cannot allocate memory
bitnami#ip-172-31-17-108:~/my-sails-api$
By default node.js will not allow you to allocate more than 1.7GB in one process so no - there's no point in upgrading unless you're sure you can't do anything in your code to consume less memory (for example use streams).
If you have no other choice you can try to increase the size of allowed memory - but keep in mind that it will probably impact the speed of your program - by setting the following option:
--max_old_space_size <size in mb>
Run node --v8-options for more information.

Resources