I have a real-time application in NodeJS that listen to multiple websockets and reacts to its events by placing HTTPS requests; it runs continuously. I noticed that the response time, at many many times during execution, was much higher than merely the expected network latency, which led me to investigate the problem. It turns out that the Garbage Collector was running multiple times in a row adding significant latency (5s, up to 30s) to the run time.
The problem seems to be related to the frequent scheduling of a Scavenger round to free up resources, due to allocation failure. Although each round takes less than 100ms to execute, executing thousands of times in a row does add up to many seconds. My only guess is that at some point in my application the allocated heap is close to its limit and little memory is actually freed in each GC round which keeps triggering GC in a long loop. The application seems not to be leaking memory because memory allocation never indeed fails. It just seems to be hitting an edge that triggers GC madness.
Could someone with knowledge/experience shed some tips on how to use the GC options that V8 provides in order to overcome such situation? I have tried --max_old_space_size and --nouse-idle-notification (as suggested by the few articles that tackle this problem) but I do not fully understand the internals of the node GC implementation and the effects of the available options. I would like to have more control over when Scavenger round should run or at least increase the interval between successive rounds so that it becomes more efficient.
I have not been able to find the type of GC that System.gc() recommends the JVM to do. Here, it is said that "the System.gc() method ... forces major collections", but this post implies a full gc is requested.
Could anyone clarify or point to documentation that spells this out? In other words, is it a major or full gc that is requested?
That depends on the JVM you're using.
Assuming you're using hotspot the behavior varies based on flags passed to it. By default it triggers a full stop-the-world GC, which will show up as the gc cause [Full GC (System.gc)] in the logs. With DisableExplicitGC it won't invoke any GC at all. If G1 or CMS are used then ExplicitGCInvokesConcurrent will change that behavior to initiate a concurrent old gen collection instead.
The major and minor terminology is not very useful anymore since GC cycles have become more nuanced.
If in doubt, enable GC logging and see for yourself.
If my nodejs memory is reached upto 1.5GB and then i am not applying any load on it and the system is idle for 30 minutes then also garbage collector is not freeing the memory
It's impossible to say anything specific, like "why gc is not collecting the garbage" as you ask in the comments, when you say nothing about what your program is doing or how the code looks like.
I can only point you to good explanation of how GC works in Node and explain how to run GC manually to see if that helps. When you run node with the --expose-gc flag, then you can use:
global.gc();
in your code. You can try to run that code in a timeout, on a regular interval or at any other specific moment and see if that frees your memory. If it does, that would mean that the GC was indeed not running and that was the problem. If that doesn't free your memory that could mean that the problems is not with the GC not running, but rather not being able to free anything.
Memory not being freed after manual GC invocation would mean that you have some memory leak or that you use so much memory that cannot be freed by the GC.
If the memory is freed after running GC manually it could mean that it is not running by itself, maybe because you are doing a very long, blocking operation that doesn't give the event loop any chance to run. For example having a long running for or while loop could give such an effect.
Not knowing anything about your code or what it does, it's not possible to give you any more specific solution to your problem.
If you want to know how and when the GC in Node works, then there is some good documentation online.
There is a nice article by StrongLoop about how the GC works:
Node.js Performance Tip of the Week: Managing Garbage Collection
Also this article by Daniel Khan is worth reading:
Understanding Garbage Collection and hunting Memory Leaks in Node.js
It's possible that the GC is running but it can't free any memory because you have some memory leak. Without seeing an actual code or even an explanation of what it does it's realy impossible to say more.
Recently I have tried to use G1GC from jdk1.7.0-17 in my java processor which is processing a lot of similar messages received from an MQ (about 15-20 req/sec). Every message is processed in the separate thread (about 100 threads in stable state) that serviced by Java limited thread pool. Surprisingly, I detected the strange behaviour - as soon as GC starts the full gc cycle it begins to use significant processing time (up to 100% CPU and even more). I was doing refactoring of the code several times having a goal to optimizing it and doing it more lightweight. But without any significant result - the behaviour is the same. I use the 4-core 64-bit machine with Debian OS (2.6.32-5 kernel). May someone help me to understand and resolve the situation?
Below are depicted some illustrations for listed above issue.
Surprisingly, I detected the strange behaviour - as soon as GC starts
the full gc cycle...
Unfortunately, this is not a surprise because for the G1 GC implemented within the JVM uses just one hardware thread (vCPU) to execute the Full GC so the idea is to minimize the number of Full GCs. Please, you should keep in mind this collector is recommended for configurations with several cores (of course it does not impact on the Full GC, but impacts on allocation and parallel collections) and big heaps I think bigger than 8GB.
According to Oracle:
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/g1_gc.html
The Garbage-First (G1) garbage collector is a server-style garbage
collector, targeted for multiprocessor machines with large memories.
It attempts to meet garbage collection (GC) pause time goals with high
probability while achieving high throughput. Whole-heap operations,
such as global marking, are performed concurrently with the
application threads. This prevents interruptions proportional to heap
or live-data size.
In this article there is an explanation about the Full GC single thread in this collector.
https://www.redhat.com/en/blog/part-1-introduction-g1-garbage-collector
Finally and unfortunately, G1 also has to deal with the dreaded Full
GC. While G1 is ultimately trying to avoid Full GC’s, they are still a
harsh reality especially in improperly tuned environments. Given that
G1 is targeting larger heap sizes, the impact of a Full GC can be
catastrophic to in-flight processing and SLAs. One of the primary
reasons is that Full GCs are still a single-threaded operation in G1.
Looking at causes, the first, and most avoidable, is related to
Metaspace.
By the way, it seems to be the newest version of Java (10) is going to include a G1 with the capability of executing Full GCs in parallel.
https://www.opsian.com/blog/java-10-with-g1/
Java 10 reduces Full GC pause times by iteratively improving on its
existing algorithm. Until Java 10 G1 Full GCs ran in a single thread.
That’s right - your 32 core server and it’s 128GB will stop and pause
until a single thread takes out the garbage.
Perhaps, you should tune the metaspace or increase the heap or you can use other collector such as the parallel GC.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Most of the modern languages have built in garbage collection (GC). e.g. Java, .NET languages, Ruby, etc. Indeed GC simplifies application development in many ways.
I am interested to know the limitations / disadvantages of writing applications in GCed languages. Assuming GC implemenation is optimal, I am just wondering we may be limited by GC to take some optimization decisions.
The main disadvantages to using a garbage collector, in my opinion, are:
Non-deterministic cleanup of resources. Sometimes, it is handy to say "I'm done with this, and I want it cleaned NOW". With a GC, this typically means forcing the GC to cleanup everything, or just wait until it's ready - both of which take away some control from you as a developer.
Potential performance issues which arise from non-deterministic operation of the GC. When the GC collects, it's common to see (small) hangs, etc. This can be particularly problematic for things such as real-time simulations or games.
Take it from a C programmer ... it is about cost/benefit and appropriate use
The garbage collection algorithms such as tri-color/mark-and-sweep there is often significant latency between a resource being 'lost' and the physical resource being freed. In some runtimes the GC will actually pause execution of the program to perform garbage collection.
Being a long time C programmer, I can tell you:
a) Manual free() garbage collection is hard -- This is because there is usually a greater error rate in human placement of free() calls than GC algorithms.
b) Manual free() garbage collection costs time -- Does the time spent debugging outweigh the millisecond pauses of a GC? It may be beneficial to use garbage collection if you are writing a game than say an embedded kernel.
But, when you can't afford the runtime disadvantage (right resources, real-time constraints) then performing manual resource allocation is probably better. It may take time but can be 100% efficient.
Try and imagine an OS kernel written in Java? or on the .NET runtime with GC ... Just look at how much memory the JVM accumulates when running simple programs. I am aware that projects exist like this ... they just make me feel a bit sick.
Just bear in mind, my linux box does much the same things today with 3GB RAM than it did when it had 512MB ram years ago. The only difference is I have mono/jvm/firefox etc running.
The business case for GC is clear, but it still makes me uncomfortable alot of the time.
Good books:
Dragon book (recent edition), Modern Compiler Implementation in C
For .NET, there are two disadvantages that I can see.
1) People assume that the GC knows best, but that's not always the case. If you make certain types of allocations, you can cause yourself to experience some really nasty program deaths without direct invokation of the GC.
2) Objects larger than 85k go onto the LOH, or Large Object Heap. That heap is currently NEVER compacted, so again, your program can experience out-of-memory exceptions when really the LOH is not compacted enough for you to make another allocation.
Both of these bugs are shown in code that I posted in this question:
How do I get .NET to garbage collect aggressively?
I am interested to know the limitations / disadvantages of writing applications in GCed languages. Assuming GC implemenation is optimal, I am just wondering we may be limited by GC to take some optimization decisions.
My belief is that automatic memory management imposes a glass ceiling on efficiency but I have no evidence to substantiate that. In particular, today's GC algorithms offer only high throughput or low latency but not both simultaneously. Production systems like .NET and the HotSpot JVM incur significant pauses precisely because they are optimized for throughput. Specialized GC algorithms like Staccato offer much lower latency but at the cost of much lower minimum mutator utilisation and, therefore, low throughput.
If you are confident(good) about your memory management skills, there is no advantage.
The concept was introduced to minimize the time of development and due to lack of experts in programming who thoroughly understood memory.
The biggest problem when it comes to performance (especially on or real-time systems) is, that your program may experience some unexpected delays when GC kicks in. However, modern GC try to avoid this and can be tuned for real time purposes.
Another obvious thing is that you cannot manage your memory by yourself (for instance, allocate on numa local memory), which you may need to do when you implement low-level software.
It is almost impossible to make a non-GC memory manager work in a multi-CPU environment without requiring a lock to be acquired and released every time memory is allocated or freed. Each lock acquisition or release will require a CPU to coordinate its actions with other CPUs, and such coordination tends to be rather expensive. A garbage-collection-based system can allow many memory allocations to occur without requiring any locks or other inter-CPU coordination. This is a major advantage. The disadvantage is that many steps in garbage collection require that the CPU's coordinate their actions, and getting good performance generally requires that such steps be consolidated to a significant degree (there's not much benefit to eliminating the requirement of CPU coordination on each memory allocation if the CPUs have to coordinate before each step of garbage collection). Such consolidation will often cause all tasks in the system to pause for varying lengths of time during collection; in general, the longer the pauses one is willing to accept, the less total time will be needed for collection.
If processors were to return to a descriptor-based handle/pointer system (similar to what the 80286 used, though nowadays one wouldn't use 16-bit segments anymore), it would be possible for garbage collection to be done concurrently with other operations (if a handle was being used when the GC wanted to move it, the task using the handle would have to be frozen while the data was copied from its old address to its new one, but that shouldn't take long). Not sure if that will ever happen, though (Incidentally, if I had my druthers, an object reference would be 32 bits, and a pointer would be an object reference plus a 32-bit offset; I think it will be awhile before there's a need for over 2 billion objects, or for any object over 4 gigs. Despite Moore's Law, if an application would have over 2 billion objects, its performance would likely be improved by using fewer, larger, objects. If an application would need an object over 4 gigs, its performance would likely be improved by using more, smaller, objects.)
Typically, garbage collection has certain disadvantages:
Garbage collection consumes computing resources in deciding what memory is to be freed, reconstructing facts that may have been known to the programmer. The penalty for the convenience of not annotating object lifetime manually in the source code is overhead, often leading to decreased or uneven performance. Interaction with memory hierarchy effects can make this overhead intolerable in circumstances that are hard to predict or to detect in routine testing.
The point when the garbage is actually collected can be unpredictable, resulting in stalls scattered throughout a session. Unpredictable stalls can be unacceptable in real-time environments such as device drivers, in transaction processing, or in interactive programs.
Memory may leak despite the presence of a garbage collector, if references to unused objects are not themselves manually disposed of. This is described as a logical memory leak.[3] For example, recursive algorithms normally delay release of stack objects until after the final call has completed. Caching and memoizing, common optimization techniques, commonly lead to such logical leaks. The belief that garbage collection eliminates all leaks leads many programmers not to guard against creating such leaks.
In virtual memory environments typical of modern desktop computers, it can be difficult for the garbage collector to notice when collection is needed, resulting in large amounts of accumulated garbage, a long, disruptive collection phase, and other programs' data swapped out.
Perhaps the most significant problem is that programs that rely on garbage collectors often exhibit poor locality (interacting badly with cache and virtual memory systems), occupy more address space than the program actually uses at any one time, and touch otherwise idle pages. These may combine in a phenomenon called thrashing, in which a program spends more time copying data between various grades of storage than performing useful work. They may make it impossible for a programmer to reason about the performance effects of design choices, making performance tuning difficult. They can lead garbage-collecting programs to interfere with other programs competing for resources