What are the difference between "task" and "allocation failure" in these lines (end of line)?
5:13:36 PM worker.1 | [33809:0x7ff6fa04d000] 1810211 ms: Scavenge 167.9 (175.1) -> 167.2 (175.1) MB, 0.9 / 0.1 ms (average mu = 0.981, current mu = 0.926) task;
5:13:48 PM worker.1 | [33809:0x7ff6fa499000] 1814585 ms: Scavenge 77.2 (80.6) -> 76.6 (80.6) MB, 1.1 / 0.0 ms (average mu = 0.984, current mu = 0.945) allocation failure;
Are "allocation failure" really means a failure for my app, is it normal to have this?
Related
I have an angular project that I recently upgraded to Angular 13. This project is building in Azure DevOps.
About 1 in 3 builds fail due to memory issues. I get lots of different errors each run, but they all are about memory.
I have scoured the internet, and I have been using the max_old_space_size but that doesn't seem to solve my problem. Is there a better way to make the angular build actually use less memory during build?
Here are the 3 errors I often see. Note each run is different, and sometimes it builds fine.
##[error]Error(0,0): Error [main.5aa4446f2024043f.js: ;]DataCloneError: Data cannot be cloned, out of memory.
Error : Optimization error [main.5aa4446f2024043f.js]: DataCloneError: Data cannot be cloned, out of memory. [D:\a\1\s\MyApp\MyApp.Web\MyApp.Web.csproj]
at WorkerInfo.postTask (D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:305:23)
at ThreadPool._onWorkerAvailable (D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:518:24)
at D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:381:46
at AsynchronouslyCreatedResourcePool.maybeAvailable (D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:237:17)
at WorkerInfo.onMessage (D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:424:26)
at WorkerInfo._handleResponse (D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:289:14)
at MessagePort.<anonymous> (D:\a\1\s\MyApp\MyApp.Web\ClientApp\node_modules\piscina\dist\src\index.js:258:51)
at MessagePort.[nodejs.internal.kHybridDispatch] (node:internal/event_target:643:20)
at MessagePort.exports.emitMessage (node:internal/per_context/messageport:23:28)
Another one here:
#FailureMessage Object: 0000001D9D3F9EB0
#
# Fatal error in , line 0
# Fatal process out of memory: Zone
#
#
#
#FailureMessage Object: 0000001D9D2FA470
1: 00007FF783AF79CF public: __cdecl v8::internal::CodeObjectRegistry::~CodeObjectRegistry(void) __ptr64+114207
2: 00007FF783A13E9F public: class std::basic_ostream<char,struct std::char_traits<char> > & __ptr64 __cdecl std::basic_ostream<char,struct std::char_traits<char> >::operator<<(__int64) __ptr64+65103
3: 00007FF7846F26C2 void __cdecl V8_Fatal(char const * __ptr64,...)+162
4: 00007FF7843A55DE public: void __cdecl v8::SharedArrayBuffer::Externalize(class std::shared_ptr<class v8::BackingStore> const & __ptr64) __ptr64+286
5: 00007FF783F3FA57 private: unsigned __int64 __cdecl v8::internal::Zone::NewExpand(unsigned __int64) __ptr64+279
6: 00007FF783D4EAE4 public: virtual char const * __ptr64 __cdecl disasm::NameConverter::NameOfXMMRegister(int)const __ptr64+17108
7: 00007FF7847752F0 public: void __cdecl v8::internal::compiler::Schedule::AddGoto(class v8::internal::compiler::BasicBlock * __ptr64,class v8::internal::compiler::BasicBlock * __ptr64) __ptr64+48
8: 00007FF7848E47D2 private: void __cdecl v8::internal::compiler::Scheduler::ComputeSpecialRPONumbering(void) __ptr64+3490
9: 00007FF7848E3B88 private: void __cdecl v8::internal::compiler::Scheduler::ComputeSpecialRPONumbering(void) __ptr64+344
10: 00007FF7848E752D private: static void __cdecl v8::internal::compiler::Scheduler::PropagateImmediateDominators(class v8::internal::compiler::BasicBlock * __ptr64)+3101
11: 00007FF7848E29A5 private: void __cdecl v8::internal::compiler::Scheduler::BuildCFG(void) __ptr64+277
12: 00007FF7848E380E public: static class v8::internal::compiler::Schedule * __ptr64 __cdecl v8::internal::compiler::Scheduler::ComputeSchedule(class v8::internal::Zone * __ptr64,class v8::internal::compiler::Graph * __ptr64,class v8::base::Flags<enum v8::internal::compiler::Scheduler::Flag,int>,class v8::internal::TickCounter * __ptr64,class v8::internal::ProfileDataFromFile const * __ptr64)+270
13: 00007FF7847A2CA9 public: bool __cdecl v8::internal::compiler::LoopPeeler::CanPeel(class v8::internal::compiler::LoopTree::Loop * __ptr64) __ptr64+185
14: 00007FF7847A83FB public: class v8::internal::compiler::LifetimePosition __cdecl v8::internal::compiler::LiveRange::NextStart(void)const __ptr64+2043
15: 00007FF7847A3BC1 public: class v8::internal::compiler::LifetimePosition __cdecl v8::internal::compiler::LiveRange::End(void)const __ptr64+177
16: 00007FF78433CFF1 public: enum v8::internal::CompilationJob::Status __cdecl v8::internal::OptimizedCompilationJob::ExecuteJob(class v8::internal::RuntimeCallStats * __ptr64,class v8::internal::LocalIsolate * __ptr64) __ptr64+49
17: 00007FF78430DF49 private: void __cdecl v8::internal::OptimizingCompileDispatcher::CompileNext(class v8::internal::OptimizedCompilationJob * __ptr64,class v8::internal::LocalIsolate * __ptr64) __ptr64+57
18: 00007FF78430EA1A public: void __cdecl v8::internal::OptimizingCompileDispatcher::QueueForOptimization(class v8::internal::OptimizedCompilationJob * __ptr64) __ptr64+714
19: 00007FF783A1682D public: class std::basic_ostream<char,struct std::char_traits<char> > & __ptr64 __cdecl std::basic_ostream<char,struct std::char_traits<char> >::operator<<(__int64) __ptr64+75741
20: 00007FF783B46EDD uv_poll_stop+557
21: 00007FF784960120 public: class v8::internal::compiler::Operator const * __ptr64 __cdecl v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor(enum v8::internal::compiler::IrOpcode::Value) __ptr64+146416
22: 00007FF80C834ED0 BaseThreadInitThunk+16
23: 00007FF80D26E39B RtlUserThreadStart+43
One More:
<--- Last few GCs --->
[6640:000001CC0BCB3FC0] 21334 ms: Scavenge 87.4 (107.8) -> 78.6 (111.3) MB, 9.5 / 0.0 ms (average mu = 0.992, current mu = 0.990) allocation failure
[6640:000001CC0BCB3FC0] 21397 ms: Scavenge 91.4 (111.8) -> 82.6 (115.5) MB, 21.7 / 0.0 ms (average mu = 0.992, current mu = 0.990) allocation failure
[6640:000001CC0BCB3FC0] 21783 ms: Scavenge 95.6 (116.0) -> 86.7 (119.5) MB, 158.3 / 0.0 ms (average mu = 0.992, current mu = 0.990) allocation failure
<--- JS stacktrace --->
<--- Last few GCs --->
[6640:000001CC0BCB3FC0] 21334 ms: Scavenge 87.4 (107.8) -> 78.6 (111.3) MB, 9.5 / 0.0 ms (average mu = 0.992, current mu = 0.990) allocation failure
[6640:000001CC0BCB3FC0] 21397 ms: Scavenge 91.4 (111.8) -> 82.6 (115.5) MB, 21.7 / 0.0 ms (average mu = 0.992, current mu = 0.990) allocation failure
[6640:000001CC0BCB3FC0] 21783 ms: Scavenge 95.6 (116.0) -> 86.7 (119.5) MB, 158.3 / 0.0 ms (average mu = 0.992, current mu = 0.990) allocation failure
<--- JS stacktrace --->
<--- Last few GCs --->
[6640:000001CC0BD4A7B0] 21478 ms: Scavenge 56.2 (76.6) -> 47.2 (80.1) MB, 17.0 / 0.0 ms (average mu = 0.996, current mu = 0.996) allocation failure
[6640:000001CC0BD4A7B0] 22725 ms: Scavenge 60.5 (80.8) -> 51.3 (82.6) MB, 1020.4 / 0.0 ms (average mu = 0.996, current mu = 0.996) allocation failure
[6640:000001CC0BD4A7B0] 27389 ms: Scavenge 62.7 (83.1) -> 54.9 (85.6) MB, 1839.8 / 0.0 ms (average mu = 0.996, current mu = 0.996) allocation failure
<--- JS stacktrace --->
##[error]EXEC(0,0): Error : MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
EXEC : FATAL error : MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory [D:\a\1\s\BehaviorLive\BehaviorLive.Web\BehaviorLive.Web.csproj]
##[error]EXEC(0,0): Error : MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
EXEC : FATAL error : MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory [D:\a\1\s\BehaviorLive\BehaviorLive.Web\BehaviorLive.Web.csproj]
##[error]EXEC(0,0): Error : MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
You didn't jot it down so I ask. Did you use command like:
cross-env NODE_OPTIONS=--max-old-space-size=8192 npm build
It depends on the Azure Agent that you use for deployment. The memory available for the agent might not be enough to complete the build, especially when the agent is trying to process parallel jobs.
You can debug and find exactly what amount of memory will get the job done in your local system.
You can start by keeping
max_old_space_size=1024
And increase size untill build succeeds. I found ours work with 2048.
Now if you can get a dedicated agent which can allocate atleast that amount of memory to run the job, then you will have a reliable solution.
EDIT - Another solution would be to try a different build tool. I assume you are using webpack, there could be other bundlers out there that are more efficient.
I am using Node.js 14 LTS and Express 4.17.1. I am trying to build a global queue that can take JSON payload, store it in the queue and dispatch as a group when the queue is a given size.
The issue that I'm facing is that the garbage collection time vastly increases, which will mean that the Node.js app will eventually become non responsive and maybe crash. No matter if I use the code below, simple arrays, or use contained classes, it is the same thing.
Essentially this is a memory leak even though I'm clearing the arrays, classes or singleton array each time it is dispatched.
I believe this is due to the reference in the root of the file so Node's garbage collection can't release it from memory and therefore grows.
I wonder if you have an alternative that can accomplish similarly but not increase the HEAP?
Here is a simplified example of what I'm trying to get working without leaks:
"use strict";
import Singleton from '../services/Singleton.js'
import Queue from '../services/Queue.js'
import dotenv from 'dotenv'
dotenv.config()
const bulkESNumber = parseInt(process.env.BULK_ES_NUMER)
let ts_queue = null
let ts_singleton = null
router.post('/:es_indice/:id?', async (req, res) => {
if (!ts_queue) {
ts_queue = new Queue();
ts_singleton = new Singleton(ts_queue);
ts_queue = null
}
let ts_q = ts_singleton.getInstance()
if ( bulkESNumber > await ts_q.size()) {
await ts_q.add(req)
console.log(await ts_q.size())
} else {
console.log('dispatche t2_s queue')
ts_q.clear()
}
})
The Queue Class:
"use strict";
class Queue {
queue = null
constructor() {
this.queue = []
}
async add(item) {
this.queue.push(item)
}
async clear() {
this.queue = []
}
async get() {
return this.queue
}
async size() {
return this.queue.length
}
}
export default Queue
The Singleton Class:
class Singleton {
constructor(obj_instance) {
if (!Singleton.instance) {
Singleton.instance = obj_instance;
}
}
getInstance() {
return Singleton.instance;
}
}
export default Singleton
If you are brave enough to scroll through this next section below you'll see that the time between garbage collection and HEAP size both keep increasing throughout the process. This means that the app becomes less responsive to the POSTs and eventually will crash.
I trigger this by posting to an endpoint that POSTs a json payload that should be stored in the queue
This is how I start the server with a heap trace:
>$ node --trace_gc --use_strict index.js
server running on port: http://127.0.0.1:3232
[14071:0x110008000] 376 ms: Mark-sweep 16.6 (28.7) -> 11.1 (28.8) MB, 2.3 / 0.1 ms (+ 0.1 ms in 3 steps since start of marking, biggest step 0.1 ms, walltime since start of marking 16 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via task GC in old space requested
1
2021-06-14T22:53:20.233Z info: Event elasped time 3ms
2
2021-06-14T22:53:21.717Z info: Event elasped time 1ms
3
2021-06-14T22:53:22.659Z info: Event elasped time 1ms
4
2021-06-14T22:53:23.649Z info: Event elasped time 1ms
...
2021-06-14T22:53:25.307Z info: Event elasped time 1ms
[14101:0x110008000] 8531 ms: Mark-sweep 12.4 (29.0) -> 11.4 (14.3) MB, 7.7 / 0.1 ms (+ 0.8 ms in 4 steps since start of marking, biggest step 0.5 ms, walltime since start of marking 35 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via task GC in old space requested
7
2021-06-14T22:53:26.035Z info: Event elasped time 2ms
[14101:0x110008000] 9168 ms: Mark-sweep 11.5 (14.3) -> 11.4 (14.3) MB, 5.1 / 0.0 ms (+ 0.5 ms in 4 steps since start of marking, biggest step 0.3 ms, walltime since start of marking 32 ms) (average mu = 0.998, current mu = 0.991) finalize incremental marking via task GC in old space requested
8
2021-06-14T22:53:28.032Z info: Event elasped time 5ms
...
9
2021-06-14T22:53:38.027Z info: Event elasped time 0ms
10
2021-06-14T22:53:38.643Z info: Event elasped time 1ms
dispatche t2_s queue
2021-06-14T22:53:39.316Z info: Event elasped time 1ms
1
2021-06-14T22:53:40.029Z info: Event elasped time 1ms
2
[14101:0x110008000] 23392 ms: Scavenge 12.5 (14.3) -> 11.7 (14.3) MB, 0.7 / 0.0 ms (average mu = 0.998, current mu = 0.991) allocation failure
2021-06-14T22:53:40.753Z info: Event elasped time 2ms
...
5
2021-06-14T22:53:49.131Z info: Event elasped time 1ms
[14101:0x110008000] 32322 ms: Scavenge 12.5 (14.3) -> 11.8 (14.3) MB, 0.9 / 0.0 ms (average mu = 0.998, current mu = 0.991) allocation failure
I have a MongoDB collection with 14 thousand documents, 100 thousand sub-documents, and 11 MB in the BSON backup. I am unable to run this NodeJS code on it:
VisitorSchema.statics.getAllVisitors = async function() {
try {
// Return all users.
return Visitor.find()
.exec();
} catch (err) {
console.log(err);
}
};
VisitorSchema.statics.checkRecentVisits = async function() {
console.log("Starting...");
let uvs = await this.getAllVisitors();
console.log("Loaded!");
}
The script takes a few seconds to run and throws:
Starting...
<--- Last few GCs --->
fa[1355:0x00] 12006 ms: Mark-sweep 255.4 (257.0) -> 254.7 (257.2) MB, 507.2 / 0.0 ms (+ 0.0 ms in 12 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 529 ms) (average mu = 0.124, current mu = 0.042) allocation fail[1355:0x39dd420] 12456 ms: Mark-sweep 255.6 (257.2) -> 255.0 (257.5) MB, 371.5 / 0.0 ms (+ 58.6 ms in 13 steps since start of marking, biggest step 17.2 ms, walltime since start of marking 451 ms) (average mu = 0.087, current mu = 0.046) allocation fa
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x00]
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
...
Aborted (core dumped)
This thread mentions a heap failure for 14 million records and I am still far from that.
How can I debug the problem or work around it?
You can double the heap size of a task with:
NODE_OPTIONS=--max_old_space_size=8192 node task.js
In the past, if using Windows, I used:
https://www.npmjs.com/package/increase-memory-limit
also, if you are using the latest version of node you might like to read this:
https://github.com/endel/increase-memory-limit#readme
Check this too:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
when I run polymer build after implementing more components, it runs out of memory.
Error message
<--- Last few GCs --->
122257 ms: Mark-sweep 1364.3 (1422.6) -> 1364.3 (1438.6) MB, 2263.8 /
0.0 ms [allocation failure] [GC in old space requested].
124485 ms: Mark-sweep 1364.3 (1438.6) -> 1364.3 (1438.6) MB, 2227.9 /
0.0 ms [allocation failure] [GC in old space requested].
126853 ms: Mark-sweep 1364.3 (1438.6) -> 1372.2 (1422.6) MB, 2367.6 /
0.0 ms [last resort gc].
129104 ms: Mark-sweep 1372.2 (1422.6) -> 1380.1 (1422.6) MB, 2251.1 /
0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x2e91707cfb39 <JS Object>
2: NewPromiseCapability(aka NewPromiseCapability) [native promise.js:175] [pc=0x34d60057703] (this=0x2e9170704381 <undefined>,O=0x2e91707c3041 <JS Function Promise (SharedFunctionInfo 0x2e9170774f51)>)
3: all [native promise.js:269] [pc=0x34d600e1f88] (this=0x2e91707c3041 <JS Function Promise (SharedFunctionInfo 0x2e9170774f51)>,S=0x1c7a750742a9 <JS Array[0]>)
4: /* anonymous */(aka...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap
out of memory
1: node::Abort() [polymer]
2: 0x10d6aac [polymer]
3: v8::Utils::ReportApiFailure(char const*, char const*) [polymer]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool)
[polymer]
5: v8::internal::Factory::NewFixedArray(int,
v8::internal::PretenureFlag) [polymer]
6: 0x99c0eb [polymer]
Aborted (core dumped)
If we increase --max-old-space-size it might solve the issue!! But how to set --max-old-space-size while building?
solved with this command, I'm using Linux
node --max-old-space-size=2048 /usr/local/bin/polymer build
I want to do a simple text classification. I've tried different packages, but all of them do the stuff in memory.
It works pretty smoothly for small inputs, but the bigger the input is, the slower is becomes.
"use strict";
const NaturalSynaptic = require("natural-synaptic");
// After getting the data
rows = rows.map(c => (c.content && c.category_name ? {
input: c.content
, output: c.category_name
} : null)).filter(Boolean);
var classifier = new NaturalSynaptic();
// This part is relatively fast
rows.forEach((c, i) => {
classifier.addDocument(c.input, c.output);
});
// It gets stuck here
classifier.train();
After training, I'd like to predict the outputs using the classifier.classify('did the tests pass?').
When it gets stuck, one of the CPUs is jumps to 100%. I suspect it's because of the for loops from the library.
What's the right way to do this? How to handle so much data as input?
An after waiting enough time, I ended with this, like I expected:
<--- Last few GCs --->
1300704 ms: Mark-sweep 1194.3 (1458.1) -> 1194.3 (1458.1) MB, 238.2 / 0 ms [allocation failure] [scavenge might not succeed].
1300955 ms: Mark-sweep 1194.3 (1458.1) -> 1194.3 (1458.1) MB, 251.7 / 0 ms [allocation failure] [scavenge might not succeed].
1301199 ms: Mark-sweep 1194.3 (1458.1) -> 1194.3 (1458.1) MB, 244.0 / 0 ms [last resort gc].
1301432 ms: Mark-sweep 1194.3 (1458.1) -> 1194.3 (1458.1) MB, 232.9 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1326850e3ac1 <JS Object>
2: textToFeatures [/home/ionicabizau/.../node_modules/natural/lib/natural/classifiers/classifier.js:~82] [pc=0x3204073474c8] (this=0xd98447d4ab1 <JS Object>,observation=0x2eb16ebfc7d9 <JS Array[36]>)
3: train [/home/ionicabizau/.../node_modules/natural/lib/natural/classifiers/classifier.js:101] [pc=0x32040734600d]...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Aborted (core dumped)