Webpack runs out of memory - node.js

Scenario
I have a 500MB RAM build machine and I want to use it to build a JavaScript bundle using webpack.
NODE_ENV=production webpack --config webpack.prod.js
This worked fine before, but as the project grew bigger, I started running out of memory. I tried setting an upper limit for node, hoping that webpack will do something smart but nope.
NODE_OPTIONS="--max-old-space-size=2048" NODE_ENV=production webpack --config webpack.prod.js
Constraints
Time it takes to build is not important to me. If we have to use the SSD for the build, that is fine.
This is a Linux box, but I have no swap.
I am happy for alternative solutions in my architecture, currently I output one single 1.6MB JS blob for my SPA.
Errors
Errors when running out of memory
FATAL ERROR: NewSpace::Rebalance Allocation failed - process out of memory
Error when I add restriction for node
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Any help or recommendations are greatly appreciated. Thanks!
Edit: additional info
"webpack": "4.6.0",
"webpack-cli": "2.0.15",

Related

mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory, after upgrading an application to Angular 10, it started giving an error of JavaScript heap out of memory.
Can someone please advise. Tried multiple ways to increase the memory but didn't get resolve. Thanks!
First things first: which node version are you using? You should use v10.x or v12.x, according to compatibility informations. I've seen some weird errors happening from using node v8.x with new Angular versions.
With this said, in the project I work on (massive project with 500+ components and 190+ modules, with a lot of problems in module hierarchy) we've had to change our build scripts to increase node memory. It can be done in your package.json file:
// package.json file
"scripts": {
"build": "node --max_old_space_size=6144 ./node_modules/#angular/cli/bin/ng build",
"postinstall": "ngcc --properties es2015 es5 browser module main --first-only --create-ivy-entry-points"
},
You'll notice the postinstall script. It's a good idea to use it, as since Angular v9, the new Ivy compiler is the default for projects, but not for libraries. This implies the use of the compatibility compiler (ngcc), which will process all Angular libraries. The postscript will run ngcc after npm install, instead of running it at build time.
Cheers!

How can I prevent React and Plotly crashing the development server due to JavaScript heap out of memory?

I am trying to build a create-react-app with react-plotly.js but when the plot components are included the compiler hangs and eventually fails displaying the error:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
I checked the node heap limit using this command:
node -e 'console.log(`node heap limit = ${require("v8").getHeapStatistics().heap_size_limit / (1024 * 1024)} Mb`)'
which returned:
node heap limit = 1456.1747760772705 Mb
I am using the WSL 2 Linux subsystem for Windows.
I have tried editing the package.json and replacing react-scripts start
with node --max_old_space_size=4096 node_modules/.bin/react-scripts start as suggested in this answer but the error was the same. I have also tried reducing it and using 512 as I have read that is sometimes advised if there is not enough space on the machine.
Any ideas how I can get this to work?
I had success resolving this issue by disabling the GENERATE_SOURCEMAP flag in my .env file, ie
GENERATE_SOURCEMAP=false
as suggested by the (slightly unrelated) answer here.

npm build gives "Ineffective mark-compacts near heap limit Allocation failed"

I have a reactjs/typescript project, running on windows 10. Im trying to build by ts-scripts with
"rimraf ../wwwroot/* && react-scripts-ts build && copyfiles
./build/**/* ../wwwroot/ -u 1
This has worked ok before, but when I removed the node_modules-folder, re-run the npm install-command and then the command above I get the error message
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation
failed - JavaScript heap out of memory
I have no idea why. After googling Ive seen
NODE_OPTIONS="–max-old-space-size=2048"
but I dont know where to put this
Full error message
Last few GCs
[11376:0000024682F49880] 60039 ms: Mark-sweep 2034.1 (2085.2) -> 2033.7 (2085.2) MB, 1029.4 / 0.0 ms (average mu = 0.073, current mu = 0.006) allocation failure scavenge might not succeed
[11376:0000024682F49880] 61094 ms: Mark-sweep 2034.4 (2085.2) -> 2034.1 (2085.7) MB, 1047.7 / 0.0 ms (average mu = 0.039, current mu = 0.007) allocation failure scavenge might not succeed
JS stacktrace
==== JS stack trace =========================================
0: ExitFrame [pc: 000001CDF84DC5C1]
Security context: 0x007044b9e6e1 <JSObject>
1: bindChildrenWorker(aka bindChildrenWorker) [000001AD1C0ACD59] [C:\Users\robcar\source\repos\Boost\Boost.Web\ClientApp\node_modules\typescript\lib\typescript.js:~27657] [pc=000001CDF8FA47CA](this=0x03da79d826f1 <undefined>,node=0x034b8724ab71 <NodeObject map = 000000660468F931>)
2: bind(aka bind) [000001AD1C0AE2D9] [C:\Users\robcar\source\repos\B...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF6B5EA121A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4810
2: 00007FF6B5E7A5B6 node::MakeCallback+4518
3: 00007FF6B5E7AFA0 node_module_register+2160
4: 00007FF6B610B3EE v8::internal::FatalProcessOutOfMemory+846
5: 00007FF6B610B31F v8::internal::FatalProcessOutOfMemory+639
6: 00007FF6B6649304 v8::internal::Heap::MaxHeapGrowingFactor+11476
7: 00007FF6B663FA67 v8::internal::ScavengeJob::operator=+25543
8: 00007FF6B663DFDC v8::internal::ScavengeJob::operator=+18748
9: 00007FF6B6646F57 v8::internal::Heap::MaxHeapGrowingFactor+2343
10: 00007FF6B6646FD6 v8::internal::Heap::MaxHeapGrowingFactor+2470
11: 00007FF6B61E9DD7 v8::internal::Factory::NewFillerObject+55
12: 00007FF6B6281ABA v8::internal::WasmJs::Install+29530
13: 000001CDF84DC5C1
I had a similar issue, although on a Mac computer.
Setting an environment variable did the trick for me. That way, your node process will take the value and use it independently of which script you're running.
export NODE_OPTIONS=--max_old_space_size=4096
The run your command again. Good Luck!
The solution which I used was to upgrade the node js version from v10 to v12.
The command export NODE_OPTIONS=--max_old_space_size=4096 works but has limitations, e.g. you should have ample memory on your machine to run the production build.
I had the same issue on Windows. The solution was to set the NODE_OPTIONS variable before running npm run build. To set this variable, simply run set NODE_OPTIONS=--max_old_space_size=8192
I got a fix after increasing the node max_old_space. Here is the fix.
Instead of running npm start or ng start. Run the below command
node --max_old_space_size=8048 ./node_modules/#angular/cli/bin/ng serve
OR
You can add the command in package.json file as well
"start": "node --max_old_space_size=8048 ./node_modules/#angular/cli/bin/ng serve"
The root cause of the problem is the large source file sizes.
You can resolve the issue in 2 steps.
First try to increase space size and check the application. If the problem is cause by your source code, then this can be resolved by increasing the Node Heap Size as below,
"start": "react-scripts --max_old_space_size=4096 start",
"build": "react-scripts --max_old_space_size=4096 build"
Second step is setting the Environment Variable.
If the problem lies within your dependencies, you may need to increase the Node Heap allocation globally to resolve the issue.
For Windows: set NODE_OPTIONS=--max_old_space_size=4096
For Linux: export NODE_OPTIONS=--max_old_space_size=4096
Also note that, if you are using very low memory like 512MB, then you will have to upgrade it to at least 2GB to get the better result.
Here I used 4096MB = 4GB size, but you can change this according to your machine configuration specifically RAM. You can set it to 2GB or 1GB and check the application.
I don't think that the solution is to increase memory limit. This way the issue is always postponed.
I've encounter this issue 2 times. I've solved, for each time:
using a specific version of node (v12.11.1, react v16.13.1 and react-scripts v3.2.0);
switching from npm to yarn (node v12.11.1, react v16.13.1 and react-scripts v3.4.1).
In addition, check your package.json, delete package-lock.json, remove node-modules folder and try to install your dependencies again.
Next try to build your project.
This is what I wrote in my package.json run command
"scripts": {
"build": "node --max_old_space_size=4096 build/build.js"
}
basically, you have to run --max_old_space_size=4096 before your actual build command
I have been struggling with this issue in my Nuxt.js environment as well.
As everyone says here, appllying the --max_old_space_size={size} with robust amount provided seemed to work for me. However, since I made some additional remediation on my Nuxt.js app stack, the same symptom has reappeared...
Furthermore, I also tried the Node.js option --heapsnapshot-near-heap-limit={number} and it SEEMED to work, not persistently.
Anyway, along with searching for another fix via here or somewhere else, I intened to do the cord review so that I can expect if there is something redundant within our application stack.
I have been facing this problem for some time. The problem is most likely caused by a dependancy or component which is a large .js file. I am currently using this package to successfully fix the issue.
You basically install it globally:
npm install -g increase-memory-limit
And then from within your project root you run:
increase-memory-limit
Which will append --max-old-space-size=4096 in all node calls inside your node_modules/.bin/* files.

How to allocate more memory to my Virtual Machine running on Fedora to avoid Heap out of Memory Error

I'm running Jenkins on a Fedora Virtual Machine and have an app created by create-react-app .
When I try to build for production on my local machine, after ~8 minutes, it does get compiled successfully (although with the message: 'the bundle size is significantly larger than recommended...'
However, when I run the same script during my Jenkins build process, I get the following error: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory.
The build script is as follows: npm run build-css && node --max_old_space_size=8192 node_modules/.bin/react-scripts-ts build && npm run copy-to-build.
My question is, how can I allocate more memory for my Virtual Machine running on Fedora so the script can run successfully before throwing the FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory .
The solution for me was to set GENERATE_SOURCEMAP=false in the .env.production file as described here.
A better solution (although more time consuming) is code split the huge files ( >1MB)

Nodejs Appveyor build running out of memory

We're running builds on AppVeyor to build an AOT angular 2 application.
The build has started failing, giving the following:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
From what I can tell, you can use --max_old_space_size=xxxx where xxxx is the size you want to increase memory to. The default is 512.
The problem is, I'm not sure where to add this flag. I've tried both dashes (what I found originall) and underscores. I've placed the flag as part of the ps build command definition, in the build scripts build configuration, and in the package.json scripts definitions to no avail.
I'm sure this just has to be put in the right spot, but I'm unclear as to where to add it. Any help would be appreciated.
Thanks
In appveyor.yml:
init:
- ps: Install-Product node 8
- cmd: set NODE_OPTIONS=--max-old-space-size=1028
This sets a node env var during init. Set it as high as you like.
This reduced build times for us by about 80% as well. Incredible.

Resources