Angular-cli 'JavaScript heap out of memory' on asset optimization - node.js

I'm creating a leaflet map inside an Angular-cli project. Map tiles are stored locally in the 'assets' folder because it's a raster map. The problem is when I've many zoom levels of the map, I've more than 280 thousands images with approximate size of 1.1 GB and It will get bigger.
when I use ng serve here is the console output (it is stuck # assets optimization):
** NG Live Development Server is running on http://localhost:4200. **
25234ms building modules
69ms sealing
0ms optimizing
0ms basic module optimization
96ms module optimization
1ms advanced module optimization
16ms basic chunk optimization
0ms chunk optimization
1ms advanced chunk optimization
0ms module and chunk tree optimization
179ms module reviving
9ms module order optimization
4ms module id optimization
6ms chunk reviving
23ms chunk order optimization
36ms chunk id optimization
61ms hashing
1ms module assets processing
149ms chunk assets processing
4ms additional chunk assets processing
0ms recording
1ms additional asset processing
2693ms chunk asset optimization
94% asset optimization
<--- Last few GCs --->
119106 ms: Mark-sweep 1333.6 (1409.7) -> 1333.6 (1415.7) MB, 1402.0 / 2.1 ms [allocation failure] [GC in old space requested].
120570 ms: Mark-sweep 1333.6 (1415.7) -> 1333.6 (1427.7) MB, 1464.2 / 2.2 ms [allocation failure] [GC in old space requested].
122079 ms: Mark-sweep 1333.6 (1427.7) -> 1340.0 (1409.7) MB, 1508.1 / 2.5 ms [last resort gc].
123749 ms: Mark-sweep 1340.0 (1409.7) -> 1346.3 (1409.7) MB, 1670.2 / 2.1 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 000001AA792CFB61 <JS Object>
1: inflight [C:\Workspace\node_modules\inflight\inflight.js:~7] [pc=00000202BF9894B7] (this=000001AA792E6661 <JS Global Object>,key=00000045863E5881 <Strin
g[90]\: lstat\x00C:/Workspace/src/assets/leafletmap_files/9/42/176.png>,cb=00000045863E5839 <JS Function lstatcb_ (SharedFunctionInfo 00000253E9EFC219
)>)
2: wrapper [C:\Workspace\...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
I tried using:
node --max_old_space_size=8192 --optimize_for_size --max_executable_size=4096 --stack_size=4096, but It doesn't make any difference.
1- Is there is any other way to avoid optimizing all the assets folder? If it is the source of the problem as I think, because the project were running smoothly before adding more leaflet tile images.
2- Is there are any workarounds to solve this issue?
UPDATE:
There is an open issue at Angular-cli repository related to this question.

Add the below line to package.json file.
"aot": "node --max-old-space-size=8192 ./node_modules/#angular/cli/bin/ng build --prod --aot"
and try running with npm run aot
It worked for me.

You can add NODE_OPTIONS as a system variable. To set memory to 4 gigs,
--max-old-space-size=4096
Lots more info on this thread, they deserve the votes

For those who get the same error, I solved the problem by changing global ng.cmd file.
C:\Users\userName\%AppData%\Roaming\npm
Copy/paste the following code into your ng.cmd.
#IF EXIST "%~dp0\node.exe" (
"%~dp0\node.exe" --max_old_space_size=8048 "%~dp0\node_modules\#angular\cli\bin\ng" %*
) ELSE (
#SETLOCAL
#SET PATHEXT=%PATHEXT:;.JS;=;%
node --max_old_space_size=8048 "%~dp0\node_modules\#angular\cli\bin\ng" %*
)

node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng serve
Open Terminal and add this above text in your terminal and click enter .
that's it .

Add this line to package.json file.
"build": "node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng build --prod"

Error faced while running "ng serve --open"
Source Link
Solution
In package.json "scripts" property update it to following
**"start": "node --max_old_space_size=2192 ./node_modules/#angular/cli/bin/ng serve",**

Run the following command in terminal:
node --max_old_space_size=10048 ./node_modules/#angular/cli/bin/ng serve

For Angular SSR (server side rendering)
If you are trying to run ng run dev:ssr and getting memory errors, you can do this:
node --max_old_space_size=8000 ./node_modules/#angular/cli/bin/ng run ANGULAR:serve-ssr -c development
Change ANGULAR to whatever your project is called in angular.json.
-c is the optional configuration you want to run.

In my case, I had to downgrade the node version to v12.14.1 and it works fine now!!

Related

Getting an out of memory error while using Create React App and Plotly.js

<--- Last few GCs --->
[972:000001913A837320] 40900 ms: Mark-sweep 2049.2 (2066.3) -> 2047.3 (2059.3) MB, 2062.9 / 0.1 ms (+ 179.6 ms in 113 steps since start of marking, biggest step 8.9 ms, walltime since start of marking 2512 ms) (average mu = 0.153, current mu = 0.107) [972:000001913A837320] 42835 ms: Mark-sweep 2048.8 (2059.3) -> 2047.7 (2057.8) MB, 1845.1 / 0.0
ms (+ 81.8 ms in 23 steps since start of marking, biggest step 9.0 ms, walltime since start of marking 1935 ms) (average mu = 0.084, current mu = 0.004) al
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 00007FF68A20B3BD]
Security context: 0x002c4c9808d1 <JSObject>
1: _append [000002A11D9F9439] [D:\ReactProjects\plotlytest\myplotlysite\node_modules\#babel\generator\lib\buffer.js:~99] [pc=0000014BCF9DB744](this=0x03160f6401b1 <Buffer map = 000000A495510339>,0x02ab7a1c78c9 <String[#1]: >,151182,27,0x02ab7a1c01b9 <null>,0x02ab7a1c04b1 <undefined>,0x02ab7a1c06e9 <false>)
2: append [000002A11D9F9391] [D:\ReactPro...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6895F5EBF napi_wrap+114095
2: 00007FF6895A0B46 v8::base::CPU::has_sse+66998
3: 00007FF6895A1946 v8::base::CPU::has_sse+70582
4: 00007FF689DB6E4E v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF689D9EF21 v8::SharedArrayBuffer::Externalize+833
6: 00007FF689C6B18C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436
7: 00007FF689C763C0 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1312
8: 00007FF689C72EE4 v8::internal::Heap::PageFlagsAreConsistent+3204
9: 00007FF689C686E3 v8::internal::Heap::CollectGarbage+1283
10: 00007FF689C66D54 v8::internal::Heap::AddRetainedMap+2452
11: 00007FF689C8809D v8::internal::Factory::NewFillerObject+61
12: 00007FF6899EE06F v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1295
13: 00007FF68A20B3BD v8::internal::SetupIsolateDelegate::SetupHeap+546637
14: 0000014BCF9DB744
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! myplotlysite#0.1.0 start: `react-scripts start`
npm ERR! Exit status 134
npm ERR!
npm ERR! Failed at the myplotlysite#0.1.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Above is the error I am recieving. I followed a pretty standard set-up, identical to the one listed on this site: https://github.com/plotly/react-plotly.js/blob/master/README.md
I created a react app. When I create a react app and "npm start" the local host it works perfectly fine. This error occurs when I do the next step:
"npm install react-plotly.js plotly.js" after I do this, the local host no longer works and I get this error.
How can I fix this?
By the way, I'm curious, why does this error still exist for so many years? Is plotly not well for react or something?
There are a couple issues reported regarding this exact problem on the repo issues tracking, like:
JavaScript heap out of memory error with the quickstart Plot example.
Javascript runs out of memory when making production build
In these issues, we clearly see this is a problem with NodeJS <= 10 memory allocation issue and there are several solutions to this:
Try to upgrade your NodeJS version to latest 12.x version
Node >= 12 increases the heap sizes automatically and it will work. I suggesst you install Node Version Manager (nvm) for Windows which will allow you quickly and easily install and switch through multiple node version.
Manually increase max_old_space_size to React scripts
Increase heap size using max_old_space_size by adding this option with a memory size into the package.json script:
"scripts": {
...
"start": "react-scripts --max_old_space_size=4096 start",
"build": "react-scripts --max_old_space_size=4096 build",
...
}
Use Partial bundles
Partial bundles
Starting in v1.15.0, plotly.js also ships with several partial bundles:
basic
cartesian
geo
...
Starting in v1.39.0, each plotly.js partial bundle has a corresponding npm package with no dependencies.
Starting in v1.50.0, the minified version of each partial bundle is also published to npm in a separate "dist min" package.
plotly.js basic
The basic partial bundle contains trace modules scatter, bar and pie.
You can load the library from plotly.js-basic-dist which it's compiled and will consume less memory when bundling:
import React from "react";
// import Plot from "react-plotly.js";
import Plotly from "plotly.js-basic-dist";
import createPlotlyComponent from "react-plotly.js/factory";
const Plot = createPlotlyComponent(Plotly);

Serverless Framework - JavaScript heap out of memory

I'm facing with this issue of "JavaScript heap out of memory" when I deploy or ru service with 'serverless offiline' command.
I'm using nestjs - a node framework - and building the project for node 10x.
On my terminal I got this below.
I found some fixes like
type " node --max-old-space-size=1024 index.js" on terminal
use this package https://www.npmjs.com/package/increase-memory-limit
append on script tag in package.json some like ""scripts": {
"webpacker": "node --max_old_space_size=4096"
not of theses works.
any clue?
PS D:\m1_workspace\dw-api> serverless offline
Serverless: Compiling with Typescript...
Serverless: Using local tsconfig.json
<--- Last few GCs --->
al[21864:000001EF81231660] 20688 ms: Mark-sweep 1394.2 (1429.4) -> 1392.3 (1429.9) MB, 977.1 / 0.0 ms (+ 0.0 ms in 62 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 987 ms) (average mu = 0.074, current mu = 0.010) all[21864:000001EF81231660] 21557 ms: Mark-sweep 1392.3 (1429.9) -> 1392.2 (1427.9) MB, 868.1 / 0.0 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 868 ms) (average mu = 0.037, current mu = 0.001) allo
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x002e2c61e6e9 <JSObject>
0: builtin exit frame: splice(this=0x03a8c4a97e89 <JSArray[8]>,0x0237e40868f9 <TypeObject map = 000001453BA516C9>,0,8,0x03a8c4a97e89 <JSArray[8]>)
1: getUnionType(aka getUnionType) [00000057B5C33821] [D:\m1_workspace\dw-api\node_modules\#hewmen\serverless-plugin-typescript\node_modules\typescript\lib\typescript.js:~34245] [pc=000003F28C0363E9](this=0x007f886026f1 <undefined>,types=0x010...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
A quick workaround is to try to run below command first:
export NODE_OPTIONS=--max_old_space_size=8192
I have a large serverless project which ran into similar issue when I tried to deploy with "sls deply". And this workaround works for me.
Hope it can help.
This was happening to me too -
I realized I had defined my serverless configuration to package each lambda individually.
Which looks like this:
package:
individually: true
Changing that to:
package:
individually: false
worked for me.
(Of course if packaging your lambda functions individually is crucial for you, then you'll lose that, but for me it wasn't).

Building Angular project lead to heap out of memory

When trying to build an Angular project on a Linux server with -aot flag, I get an error. I tried upgrading my instance to a higher CPU core, more RAM and turn on a SWAP partition but still the error just occur faster.
Here's while trying to build from terminal along with the command with the flags I use.
ng build --target=production --environment=prod --aot
Your global Angular CLI version (1.7.2) is greater than your local
version (1.4.7). The local Angular CLI version is used.
To disable this warning use "ng set --global warnings.versionMismatch=false".
10% building modules 5/5 modules 0 activeTemplate parse warnings:
The <template> element is deprecated. Use <ng-template> instead ("
[WARNING ->]<template [ngIf]="!isClosed">
<div [class]="'alert alert-' + type" role="alert" [ngClass]="classes""): ng:///****************node_modules/ngx-bootstrap/alert/alert.component.d.ts.AlertComponent.html#1:0
92% chunk asset optimization
<--- Last few GCs --->
133611 ms: Mark-sweep 1302.6 (1435.2) -> 1297.8 (1435.2) MB, 1112.6 / 0.0 ms [allocation failure] [GC in old space requested].
134724 ms: Mark-sweep 1297.8 (1435.2) -> 1297.7 (1435.2) MB, 1112.7 / 0.0 ms [allocation failure] [GC in old space requested].
135854 ms: Mark-sweep 1297.7 (1435.2) -> 1302.7 (1406.2) MB, 1129.8 / 0.0 ms [last resort gc].
136983 ms: Mark-sweep 1302.7 (1406.2) -> 1307.9 (1406.2) MB, 1128.8 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0xb8bacacfb39 <JS Object>
1: find_variable [0xb8baca04381 <undefined>:~3539] [pc=0xbdb4151ee32] (this=0xa9144042161 <an AST_Function with map 0xd6ccb950f99>,name=0x1f852792bb29 <String[12]: DOMException>)
2: visit [0xb8baca04381 <undefined>:3449] [pc=0xbdb4151d969] (this=0x3da6d780b621 <a TreeWalker with map 0xd6ccb9394c9>,node=0x2f9919fb8e91 <an AST_SymbolRef with map 0xd6ccb9628c9>,descend=0x2e68ca83edd1 <JS Fu...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [ng]
2: 0x109bf8c [ng]
3: v8::Utils::ReportApiFailure(char const*, char const*) [ng]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [ng]
5: v8::internal::Factory::NewInternalizedStringImpl(v8::internal::Handle<v8::internal::String>, int, unsigned int) [ng]
6: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [ng]
7: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [ng]
8: v8::internal::Runtime::GetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [ng]
9: 0xed3501 [ng]
10: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [ng]
11: 0xbdb262092a7
Aborted (core dumped)
I tried the solution mentioned here, https://www.npmjs.com/package/increase-memory-limit
But still with no luck
For prod build, run following command:
node --max_old_space_size=4096 node_modules/#angular/cli/bin/ng build
--target production
For more optimization flags, below command:
It has inbuilt cache busting and other goodies:
node --max_old_space_size=4096 node_modules/#angular/cli/bin/ng build
--target production --build-optimizer --vendor-chunk
try running
node --max-old-space-size=8192
where 8192 is size in MB
As Filipe Silva recommends in the angular-cli repository, add a new entry in the scripts section of your package.json file:
"ng-high-memory": "node --max_old_space_size=8000 ./node_modules/#angular/cli/bin/ng",
and use it this way:
npm run ng-high-memory --
You have to remember to use the double dash before the arguments, otherwise they might not be parsed correctly:
npm run ng-high-memory -- serve
This does not look like a hardware limitation problem as such which can be resolved by upgrading the memory or processor.
I once had a build running out of memory, but it turned out that there was a circular reference where a source file A required a source file B and then B also required A. It was a code design issue, of course.
I would suggest you to try a simple build first and check to see if it goes well.
ng build
Also, the Angular doc mentions that when using ng build --prod, (not --target=production), it performs an AOT compilation, among other things:
https://angular.io/guide/deployment
The --prod meta-flag engages the following optimization features.
Ahead-of-Time (AOT) Compilation: pre-compiles Angular component templates.
Production mode: deploys the production environment which enables production mode.
You might want to install the latest version of the Angular CLI and see if that helps.
I faced same issue on Angular 4 project. The problem is my Node Js version is not compatible Angular 4, I installed Node Js v6.17.1 and problem resolved for me.
For more details Angular and Node Js compatibility you can visit this link : Is there a compatibility list for Angular / Angular-CLI and Node.js?
I faced with the same problem. as a solution adding Swap file worked for me.

Meteor build running our of memory

I'm trying to build my meteor app and am constantly running into the below error. This is not the first time I'm building the app and everything worked fine until yesterday's build. I already tried: as suggested in one of the answeres in [this SO post][1] but it did not help.
#!/usr/bin/env node --max_old_space_size=4096 --optimize_for_size --max_executable_size=4096 --stack_size=4096
Console output:
meteor build .
WARNING: The output directory is under your source tree.
Your generated files may get interpreted as source code!
Consider building into a different directory instead
meteor build ../output
Minifying app code \
<--- Last few GCs --->
103230 ms: Mark-sweep 1385.5 (1455.5) -> 1387.9 (1455.5) MB, 898.4 / 0 ms [allocation failure] [GC in old space requested].
104206 ms: Mark-sweep 1387.9 (1455.5) -> 1387.9 (1455.5) MB, 975.8 / 0 ms [allocation failure] [GC in old space requested].
105196 ms: Mark-sweep 1387.9 (1455.5) -> 1384.1 (1455.5) MB, 990.2 / 0 ms [last resort gc].
106101 ms: Mark-sweep 1384.1 (1455.5) -> 1385.1 (1455.5) MB, 905.3 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x249f6fdb4629 <JS Object>
1: /* anonymous */(aka /* anonymous */) [0x249f6fd041b9 <undefined>:~4943] [pc=0xcd10dd2f48c] (this=0x249f6fd041b9 <undefined>,self=0x1400b413881 <an AST_ObjectKeyVal with map 0xc3d3a4651b9>,output=0x17417c4edd79 <an Object with map 0x16588927e021>)
2: doit(aka doit) [0x249f6fd041b9 <undefined>:4190] [pc=0xcd10d7a3298] (this=0x249f6fd041b9 <undefined>)
3: print [0x249f6fd041b9 <unde...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Aborted (core dumped)
This same issue was driving me nuts, but I finally managed to resolve it under meteor 1.4.3.1.
Background:
The issue is that meteor calls node to build. When it runs, node allocates a certain amount of memory for the V8 engine it runs on. In bigger projects, the default memory allocated for V8 is not sufficient to keep track of everything - it tried to garbage-collect as it gets closer to the limit, but eventually runs out of space and crashes with the error shown.
If we were just running node directly, we could run it with the --max-old-space-size option, which would allow us to set the maximum memory for the V8 engine. The issue is that meteor calls node in its own context and with its own options, so we can't just add the flag directly to our meteor call.
Solution:
It appears that meteor 1.4.3.1 (and maybe others) will pass along flags and options specified in the TOOL_NODE_FLAGS environment variable when it calls node (others have mentioned NODE_OPTIONS, but it isn't working for my version of meteor - the flags just get dropped)
So if you want to increase the max memory of the node engine to 4 GB, add an environmental variable
TOOL_NODE_FLAGS="--max-old-space-size=4096"
to the context you are running meteor in - the option should be passed through to the node call.
(If you don't know where to set environment variables - it is usually going to be in your IDE build configuration or build script. If you want to sanity check if the --max-old... option is actually being read, try changing it to gibberish - it should cause meteor to throw an error)
You need to take notice of that initial warning:
WARNING: The output directory is under your source tree.
Your generated files may get interpreted as source code!
Consider building into a different directory instead
meteor build ../output
Read what it says - basically it will be producing files, and then compiling them in as well. No wonder it gets into trouble and runs out of memory. Put the build in a different directory (not within the Meteor project) and it should be a lot happier :)

On Windows, wepback build is blocked during 'emit' phase

I created a simple webpack project for a Typescript library : https://github.com/lbar/test-webpackts-issue
On linux, npm run build finishes successfully:
> rimraf dist && webpack --progress --profile
10% 0/1 build modulests-loader: Using typescript#1.8.10 and /tmp/testwebpack/tsconfig.json 8030ms1217ms optimi5ms emit
Hash: 8eac1502f3b7a9a19be2
Version: webpack 1.13.1
Time: 2596ms
Asset Size Chunks Chunk Names
index.js 816 bytes 0 [emitted] index
index.js.map 6.07 kB 0 [emitted] index
/tmp/testwebpack/dist/index.d.ts 31 bytes [emitted]
/tmp/testwebpack/dist/src/info.d.ts 49 bytes [emitted]
+ 2 hidden modules
On Windows, npm run build is blocked during 'emit' phase. The output is :
> rimraf dist && webpack --progress --profile
10% 0/1 build modulests-loader: Using typescript#1.8.10 and C:\tmp\testwebpack\tsconfig.jso
1836ms build modules
3ms seal
7ms optimize
3ms hashing
7ms create chunk assets
79ms additional chunk assets
19ms optimize chunk assets
837ms optimize assets
95% emit
The version of node differs: 6.2.2 on Windows and 4.4.5 on Linux. I don't know if it may be the cause.
Has anyone an idea about the problem? Issue in webpack, ts-loader or typescript?
One more thing, call to tsc is working on both systems.
Thanks for your help
This problem is caused by an issue in ts-loader: https://github.com/TypeStrong/ts-loader/issues/185
A Pull Request is waiting fixes for project core team.
Webpack can use "awesome-typescript-loader" to avoid this issue.

Resources