We are trying to get an HP UX server up for running some Batch Jobs in Java. The problem in the nutshell is this:
These Batch Jobs are running on the Jeus Application Server (a Made in Korea product). Although most of the jobs are running on time, some Batch Jobs have reported the famous PermGen Out of Memory Error. We got a heap dump and started looking at the JVM options. They are as follows: Xms - 2048m Xmx - 2048m PermSize - 512m MaxPermSize - 512m. The old GC collector is CMS and the young generation collector is a Parallel Collector.
We noticed that the actual Heap size is quite fine and that the PErm Gen ran out of all its 512m of memory. HPJmeter showed that this was a gradual rise in the memory and hence, we suspected a memory leak. We looked at all the Class Loaders and there were about 168 instances of the core Jeus App Server ClassLoader. The memory leak estimator showed that the this Class Loader was leaking memory. It showed that the number of bytes held were only around 3MB.
We looked at the number of classes/types loaded by this class loader and there were around 512 items. The memory leak also showed some other objects as being involved in memory leaks.
I have the following questions:
Now that the Classloader which is leaking has been identified, how do we identify the exact reference(s) which is causing the problem?
If one class loader with a specific object id is shown as leaking memory, is it possible that the other instances of the same class loader leak the some varying amount of memory?
Any guidance as to what to do next would be highly appreciated.
Aditya.
Are you sure it's the class loader that's "leaking", or are the classes loaded by the loader still referenced?
Understand that a "user class loader" is typically used to load classes that may need to be unloaded at a later time. Eg, if classes are dynamically generated as some sort of web page generation scheme, these classes need to be loaded within their own class loader, used, and then all traces of references to the classes and their loaders need to be eliminated. When those references are gone the loaders and loaded classes will be collected.
If some sort of cache is hanging on to references to the classes (eg, by referencing objects of the classes, or by linking to the classes from other classes in other loaders) then collection doesn't occur.
Related
I'm a compiler researcher and I'm currently researching the impact of different compilation strategies in the final size of an executable. For instance, if a strategy is to generate multiple versions of the same function that bake in some of the parameters (constant propagation), it's to be expected that the amount of code will increase; dead code elimination, on the other hand, decrease it. Similarly, if some optimization relies on using a static memory pool whose size is determined at compile-time, I'd like to know how much space it is taking.
Most tools I've looked into (e.g. valgrind) seem to focus on stack and heap usage, but don't say much about static memory. Is there some other tool I'm missing?
Try using newrelic monitoring tool it will help you to collect garbage data of code or which api is using. In linux there is only physical memory and swap memory,application level they use heap memory for that new relic will be helpful.
Newrelic link to get stack details :
https://newrelic.com/ --> 100GB free trail is available to understand your requirement.
A piece of software from a vendor eats an unholy amount of memory over time.
This is a Java application and I've inspected heap and metaspace inside the JRE, everything ok there (uses < 1G RAM). The issue rather seems to be some native memory allocation within the code (which I don't have). So I've used jemalloc to profile and the following is the result.
Is it safe to say that I should ask the vendor to fix their code or what could be the issue here? How can the output of jemalloc be interpreted?
Some Context: We have upgraded the environment of a web application from running on Java 7 to running on Java 8 and Tomcat 8 (64-bit arch, Heap size about 2 GB, PermGen size=256 MB, no constraints on metaspace size). After a while, we started getting the following error:
java.lang.OutOfMemoryError: Compressed class space
which means that the space needed for UseCompressedClassPointers exceeded CompressedClassSpaceSize. At that moment VisualVM showed a 2 GB metaspace size.
Now with the VisualVM tool, we can see the Metaspace size is constatnly increasing with every request about 3 MB, however the heap does not seem to do so. The heap usage has a saw zigzag shape going back to the same low point after every GC.
I can tell that the application is leaking Metadata only when using a Java JAXB operation, but I couldn't prove it with VisualVM.
The application depends on webservices-rt-1.4 as a JAXB implementation provider. The application uses marshalling, unmarshalling. The class generation from XSD is done with maven-jaxb2-plugin-0.13.1.
Update:
After tracing class loading and unloading, I found out that the same JAXB classes is loaded into memory by WebAppClassLoader multiple times but never cleaned up. Moreover, there are no instances to them in the heap. I debugged and I saw that JDK is calls the method
javax.xml.bind.JAXBContext com.sun.xml.bind.v2.ContextFactory.createContext by reflection and that's when the class are created.
I though the classes are cleaned up by GC. Is it the responsibility of the classLoader to clean up?
Questions: Is there a way to analyze the metaspace objects? Why do I have a leak in metaspace but not in heap? aren't they related? Is that even possible?
Why would the app work fine with PermGen but not Metaspace?
I am facing similar issue.
In my case, memory leak was caused by JAXBContext.newInstance(...) invoke.
Solutions:
wrap this new instance as singleton (https://github.com/javaee/jaxb-v2/issues/581) or
use -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true VM parameter, like in answer Old JaxB and JDK8 Metaspace OutOfMemory Issue
I had similar issue and adding -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true in setenv.sh as JVM OPT arguments it resolve OOM metaspace issue.
My project has started using java 8 from java 7.
After switching to java 8, we are seeing issues like the memory consumed is getting higher with time.
Here are the investigations that we have done :
Issues comes only after migrating from java7 and from java8
As metaspace is the only thing related to memory which is changes from hava 7 to java 8. We monitored metaspace and this does not grow more then 20 MB.
Heap also remains consistent.
Now the only path left is to analyze how the memory gets distributes to process in java 7 and java 8, specifically private byte memory. Any thoughts or links here would be appreciated.
NOTE: this javaw application is a swing based application.
UPDATE 1 : After analyzing the native memory with NMT tool and generated a diff of memory occupied as compare to baseline. We found that the heap remained same but threads are leaking all this memory. So as no change in Heap, I am assuming that this leak is because of native code.
So challenge remains still open. Any thoughts on how to analyze the memory occupied by all the threads will be helpful here.
Below are the snapshots taken from native memory tracking.
In this pic, you can see that 88 MB got increased in threads. Where arena and resource handle count had increased a lot.
in this picture you can see that 73 MB had increased in this Malloc. But no method name is shown here.
So please throw some info in understanding these 2 screenshot.
You may try another GC implementation like G1 introduced in Java 7 and probably the default GC in Java 9. To do so just launch your Java apps with:
-XX:+UseG1GC
There's also an interesting functionality with G1 GC in Java 8u20 that can look for duplicated Strings in the heap and "deduplicate" them (this only works if you activate G1, not with the default Java 8's GC).
-XX:+UseStringDeduplication
Be aware to test thoroughly your system before going to production with such a change!!!
Here you can find a nice description of the diferent GCs you can use
I encountered the exact same issue.
Heap usage constant, only metaspace increase, NMT diffs showed a slow but steady leak in the memory used by threads specifically in the arena allocation. I had tried to fix it by setting the MALLOC_ARENAS_MAX=1 env var but that was not fruitful. Profiling native memory allocation with jemalloc/jeprof showed no leakage that could be attributed to client code, pointing instead to a JDK issue as the only smoking gun there was the memory leak due to malloc calls which, in theory, should be from JVM code.
Like you, I found that upgrading the JDK fixed the problem. The reason I am posting an answer here is because I know the reason it fixes the issue - it's a JDK bug that was fixed in JDK8 u152: https://bugs.openjdk.java.net/browse/JDK-8164293
The bug report mentions Class/malloc increase, not Thread/arena, but a bit further down one of the comments clarifies that the bug reproduction clearly shows increase in Thread/arena.
consider optimising the JVM options
Parallel Collector(throughput collector)
-XX:+UseParallelGC
concurrent collectors (low-latency collectors)
-XX:+UseConcMarkSweepGC
use String Duplicates remover
-XX:+UseStringDeduplication
optimise compact ratio
-XXcompactRatio:
and refer
link1
link2
In this my answer you can see information and references how to profile native memory of JVM to find memory leaks. Shortly, see this.
UPDATE
Did you use -XX:NativeMemoryTracking=detail option? The results are straightforward, they show that the most memory allocated by malloc. :) It's a little bit obviously. Your next step is to profile your application. To analyze native methods and Java I use (and we use on production) flame graphs with perf_events. Look at this blog post for a good start.
Note, that your memory increased for threads, likely your threads grow in application. Before perf I recommend analyze thread dumps before/after to check does Java threads number grow and why. Thread dumps you can get with jstack/jvisualvm/jmc, etc.
This issue does not come with Java 8 update 152. The exact root cause of why it was coming with earlier versions is still not clearly identified.
I have a base class and a derived class.
Things work well with this setup. I have added another class in base class, so its a nested class.
On allocation of memory for the new nested class, I see some memory corruption.
I want to know when we have nested class, does the size of base class increase ?
Odds are the changes are hiding an application code problem, just as certain memory bugs don't corrupt a running process the same way when run under a debugger, which also changes the environment.
mallopt is used to frob the value however, with this doc:
MALLOC_ARENA_SIZE
The size of the arena, a chunk of memory that the memory allocator allocates
and deallocates from the system. This value must be a multiple of 4 KB, and
currently is limited to being less than 256 KB. Environment variable:
MALLOC_ARENA_SIZE.
But like I said, if changes this looks like it fixes the corruption, don't believe it. Better to set the arena size to whatever value exposes the application bug, then fix the bug itself.