I have a Openjdk 11.0.17 + Spring Boot application and this is my GC Configuration --XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Xms512m -Xmx8400m. When the application is running and the heap usage increases as per the incoming web traffic. But I am noticing that when there is no incoming traffic and no processing going on the total heap usage stays the same (high say 7gigs out of 8gig max heap) for longer times. When I take a heap dump the total usage reduces to less than 2% (say 512m). If the heap usage increases it kills the application with OOM (out of memory) error.
How to make G1GC garbage collector run more frequently ? or Is there a way to tune the GC to not make the application oom killed.
But I am noticing that when there is no incoming traffic and no processing going on the total heap usage stays the same (high say 7gigs out of 8gig max heap) for longer times.
By default G1GC only triggers a collection when some of its regions exceed some thresholds, i.e. only tiggers while its allocating objects. Lowering the InitiatingHeapOccupancyPercent can trigger those earlier.
Alternatively, if you upgrade to JDK12 or newer you can use G1PeriodicGCInterval for time-triggered collections in addition to allocation-triggered ones.
Another option is to switch to ZGC or Shenandoah which have options to trigger idle/background GCs to reduce heap size during idle periods.
or Is there a way to tune the GC to not make the application oom killed.
That problem generally is separate from high heap occupancy during idle periods. The GCs will try to perform emergency collections before issuing an OOM. So even if there was lots of floating garbage before the attempted allocation it would have been collected before throwing that error.
One possibility is that there's enough heap for steady-state operation but some load spike caused a large allocation that exceeded the available memory.
But you should enable GC logging to figure out the exact cause and what behavior lead up to it. There are multiple things that can cause OOMs, a few aren't even heap-related.
Related
I have a NodeJS server running on a small VM with 256MB of RAM and I notice the memory usage keeps growing as the server receives new requests. I read that an issue on small environments is that Node doesn't know about the memory constraints and therefore doesn't try to garbage collect until much later (so for instance, maybe it would only want to start garbage collecting once it reaches 512MB of used RAM), is it really the case?
I also tried using various flags such as --max-old-space-size but didn't see much change so I'm not sure if I have an actual memory leak or if Node just doesn't GC as soon as possible?
This might not be a complete answer, but it's coming from experience and might provide some pointers. Memory leak in NodeJS is one of the most challenging bugs that most developers could ever face.
But before we talk about memory leak, to answer your question - unless you explicitly configure --max-old-space-size, there are default memory limits that would take over. Since certain phases of Garbage collection in node are expensive (and sometimes blocking) steps, depending upon how much memory is available to it, it would delay (e.g. mark-sweep collection) some of the expensive GC cycles. I have seen that in a Machine with 16 GB of memory it would easily let the memory go as high as 800 MB before significant Garbage Collections would happen. But I am sure that doesn't make ~800 MB any special limit. It would really depend on how much available memory it has and what kind of application are you running. E.g. it is totally possible that if you have some complex computations, caches (e.g. big DB Connection Pools) or buggy logging libraries - they would themselves always take high memory.
If you are monitoring your NodeJs's memory footprint - sometime after the the server starts-up, everything starts to warm up (express loads all the modules and create some startup objects, caches warm up and all of your high memory consuming modules became active), it might appear as if there is a memory leak because the memory would keep climbing, sometimes as high as ~1 gb. Then you would see that it stabilizes (this limit used to be lesser in <v8 versions).
But sometimes there are actual memory leaks (which might be hard to spot if there is no specific pattern to it).
In your case, 256 MB seems to be meeting just the minimum RAM requirements for nodejs and might not really be enough. Before you start getting anxious of memory leak, you might want to pump it up to 1.5 GB and then monitor everything.
Some good resources on NodeJS's memory model and memory leak.
Node.js Under the Hood
Memory Leaks in NodeJS
Can garbage collection happen while the main thread is
busy?
Understanding and Debugging Memory Leaks in Your Node.js Applications
Some debugging tools to help spot the memory leaks
Node inspector |
Chrome
llnode
gcore
I have a NodeJS application running on a k8s pod. The actual size of the pod is 2GB, but in environment variables, we set this value to 4GB --max-old-space-size=4096 (which will not be true in my case - for some tenants we do allocate 4GB but most pods have 2GB).
Now I tried 2 ways to detect the memory usage and the total memory and both are providing different stats.
I'm fetching memory usage from this system file: /sys/fs/cgroup/memory/memory.usage_in_bytes and total memory from this file : /sys/fs/cgroup/memory/memory.limit_in_bytes
limit_in_bytes is returning 2GB correctly, but the value of usage_in_bytes is fluctuating too much, It's around 1gb in few mins and spikes to 2GB in next minute even though nothing changed in that minute(no stress on the system).
Stats of a process
Memory Usage POD: 2145124352
shlog - memLimit 214748364
The second option I tried is using this V8 built in node js library to get heap statistics : https://nodejs.org/api/v8.html#v8getheapstatistics.
Usage:
const initialStats = v8.getHeapStatistics();
console.log("heap_size_limit: ", (initialStats.heap_size_limit)); // total memory
console.log("total_heap_size: ", (initialStats.total_heap_size)); // current usage
Now in total memory, it's return 4G, which is not right in my case. but the current usage here seems much true.
Stats of the same process
total_heap_size: 126312448,
heap_size_limit: 4320133120,
Complete response of v8 getHeapStatistics method:
HeapStats: {
total_heap_size: 126312448,
total_heap_size_executable: 1097728,
total_physical_size: 124876920,
total_available_size: 4198923736,
used_heap_size: 121633632,
heap_size_limit: 4320133120,
malloced_memory: 73784,
peak_malloced_memory: 9831240,
does_zap_garbage: 0,
number_of_native_contexts: 1,
number_of_detached_contexts: 0
}
My goal is to detect the memory usage according to the total memory of the pod, and so some throttling when the memory consumption reached 85%. I'm willing to use the first method but please tell me why there is so much difference in the memory usage and how to get the accurate memory usage of the pod.
Really looking forward to get some help on this. Thank you.
To get overall memory consumption of a process, look to (and trust) the operating system's facilities.
Node's v8.getHeapStatistics tell you about the managed (a.k.a. garbage-collected) heap where all the JavaScript objects live. But there can be quite a bit of other, non-garbage-collected memory in the process, for example Node buffers and certain strings, and various general infrastructure that isn't on the managed heap. In the average Chrome renderer process, the JavaScript heap tends to be around a third of total memory consumption, but with significant outliers in both directions; for Node apps it depends a lot on what your app is doing.
Setting V8's max heap size (which, again, is just the garbage-collected part of the process' overall memory usage) to a value larger than the amount of memory available to you doesn't make much sense: it invites avoidable crashes, because V8 won't spend as much time on garbage collection when it thinks there's lots of available memory left, but the OS/pod as a whole could already be facing an out-of-memory situation at that point. That's why I linked the other answer: you very likely want to set the max heap size to something a little less than available memory, to give the garbage collector the right hints about when to work harder to stay under the limit.
Im using java melody to monitor memory usage in production environment.
The requirement is memory not should exceed 256MB/512MB .
I have done maximum of code optimized but still the usage is 448MB/512MB but when i executed garbage collector in java melody manually the memory consumption is 109MB/512MB.
You can invoke the garbage collection using one of these two calls in your code (they're equivalent):
System.gc();
Runtime.getRuntime().gc();
It's better if you place the call in a Runnable that gets invoked periodically, depending on how fast your memory limit is reached.
Why you actually care about heap usage? as long as you set XMS (maximum heap) you are fine. Let java invoke GC when it seems fit. As long as you have free heap it is no point doing GC and freeing heap just for sake of having a lot of free heap.
If you want to limit memory allocated by process XMX is not enough. You should also limit native memory.
What you should care about is
Memory leaks, consecutive Full GCs, GC starvation
GC KPIs: Latency, throughput, footprint
Object creation rate, promotion rate, reclamation rate…
GC Pause time statistics: Duration distribution, average, count, average interval, min/max, standard deviation
GC Causes statistics: Duration, Percentage, min/max, total
GC phases related statistics: Each GC algorithm has several sub-phases. Example for G1: initial-mark, remark, young, full, concurrent mark, mixed
See https://blog.gceasy.io/2017/05/30/improving-your-performance-reports/ https://blog.gceasy.io/2017/05/31/gc-log-analysis-use-cases/ for more technical details. You could also analyze your GC logs using https://blog.gceasy.io/ it will help you understand how your JVM is using memory.
Our clusters were running great from days and all sudden today morning hints are high. And they will not go down naturally.
What we are seeing is, gc is taking longer time than 200ms which is causing nodes to be DOWN and thereby leading to increase in hints. This is a vicious circle, and I am not sure how to fix it.
Machine config: 128GB RAM, 2TB hard disk, 24 core machine. JVM heap size 16Gb, GCpausemillisecods 200ms, parallelgcthreads 8.
Please let me know, how can I tune GC config to break this gc pause to hint loop?
We increased the size of JVM heap to 32GB, and took memtable off heap for write heavy customers.
So i have a Node application which slow down every minute.
I used some profilers (nodetime, StrongLoop, memwatch) to find where this degraded performance come from : garbage collection operations takes longer after each request.
According to StrongLoop, my heap size and memory post V8 full GC are almost constant, heap instance count and heap memory usage do not grow.
However, the RSS (Resident Set Size) never stop growing. I believe this is the trigger for the nearly 120 cycles of GC / HTTP request, consuming nearly all CPU.
Any idea where this RSS can come from, and if this has anything to do with increased GC cycles?
graphs