I encountered a weird phenomenon when testing my app on JVM. I found that after an evening's lasting running, the JVM's Minor gc will not be triggered unless both Eden and one of survior areas is full. Just like photos below.
jstat -gcutil1
jstat -gcutil2
I'm using Alibaba DragonWell Open JDK:
openjdk version "1.8.0_322"
OpenJDK Runtime Environment (Alibaba Dragonwell 8.10.11) (build 1.8.0_322-b01)
OpenJDK 64-Bit Server VM (Alibaba Dragonwell 8.10.11) (build 25.322-b01, mixed mode)
So are there some experts could explain the reasons for this problem. Thanks a lot
Related
Our production server went OOM because permgen is full. Using jmap -permstat to check the permgen area, we found there were many classes loaded by com.sun.xml.ws.client.WSSServiceDelegatingLoader. The loaded classes are com.sun.proxy.$ProxyXXX, where XXX is an int sequence.
the stacktrace for these classloading is as follow:
eventually, the JVM went OOM, full gc can't reclaim any permgen memory.
What is strange is that if I click System GC in VisualVM, the classes are unloaded and the usage of permgen goes down.
Our JDK version is 1.7.0.80， we have added CMSClassUnloadingEnabled.
-XX:+ExplicitGCInvokesConcurrent
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=60
-XX:+UseParNewGC
-XX:+CMSParalledlRemarkEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:+CMSFullGCsBeforeCompaction=0
-XX:+CMSCLassUnloadingEnabled
-XX:MaxTenuringThreshold=18
-XX:+UseCMSInitialtingOccupancyOnly
-XX:SurvivorRatio=4
-XX:ParallecGCThreads=16
Our code has been running for a long time. The most recent operation is a WebLogic patch. This really confused me. Could someone give me some help with this issue, many thanks!
This is a known bug https://github.com/javaee/metro-jax-ws/issues/1161
Every time a JAX-WS client is created, for instance, using library JAX-WS RI 2.2 which is bundled in Weblogic Server 12.1.3
com.sun.xml.ws.client.WSServiceDelegate$DelegatingLoader#1
Client proxy classes are being loaded into classloader:
([Loaded com.sun.proxy.$Proxy979 from com.sun.xml.ws.client.WSServiceDelegate$DelegatingLoader] )
Solution/Workaround:
Replace JAX-WS client where this bug is solved.
I have a websocket server that hoards memory during days, till the point that Kubernetes eventually kills it. We monitor it using prometheous-net.
# dotnet --info
Host (useful for support):
Version: 2.1.6
Commit: 3f4f8eebd8
.NET Core SDKs installed:
No SDKs were found.
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
But when I connect remotely and take a memory dump (using createdump), suddently the memory drops... without the service stopping, restarting or loosing any connected user. See the green line in the picture.
I can see in the graphs, that GC is collecting regularly in all generations.
GC Server is disabled using:
<PropertyGroup>
<ServerGarbageCollection>false</ServerGarbageCollection>
</PropertyGroup>
Before disabling GC Server, the service used to grow memory way faster. Now it takes two weeks to get into 512Mb.
Other services using ASP.NET Core on request/response fashion do not show this problem. This uses Websockets, where each connection last usually around 10 minutes... so I guess everything related with the connection survives till Gen 2 easily.
Note that there are two pods, showing the same behaviour, and then one (the green) drops suddenly in memory ussage due the taking of the memory dump.
The pods did not restart during the taking of the memory dump:
No connection was lost or restarted.
Heap:
(lldb) eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00007F8481C8D0B0
generation 1 starts at 0x00007F8481C7E820
generation 2 starts at 0x00007F852A1D7000
ephemeral segment allocation context: none
segment begin allocated size
00007F852A1D6000 00007F852A1D7000 00007F853A1D5E90 0xfffee90(268430992)
00007F84807D0000 00007F84807D1000 00007F8482278000 0x1aa7000(27947008)
Large object heap starts at 0x00007F853A1D7000
segment begin allocated size
00007F853A1D6000 00007F853A1D7000 00007F853A7C60F8 0x5ef0f8(6222072)
Total Size: Size: 0x12094f88 (302600072) bytes.
------------------------------
GC Heap Size: Size: 0x12094f88 (302600072) bytes.
(lldb)
Free objects:
(lldb) dumpheap -type Free -stat
Statistics:
MT Count TotalSize Class Name
00000000010c52b0 219774 10740482 Free
Total 219774 objects
Is there any explanation to this behaviour?
The problem was the connection to RabbitMQ. Because we were using sort lived channels, the "auto-reconnect" feature of the RabbitMQ.Client was keeping a lot of state about dead channels. We switched off this configuration, since we do not need the "perks" of the "auto-reconnect" feature, and everything start working normally. It was a pain, but we basically had to setup a Windows deploy and do the usual memory analysis process with Windows tools (Jetbrains dotMemory in this case). Using lldb is not productive at all.
Disclaimer: I am no .NET Wizard.
But you should do two things to go with Kubernetes best practices:
Define sensible resource limits for your app. If the app does not need more than 200MB memory define a resource limit to prevent the app from consuming all available host memory. But be aware that the Unix API to get available memory is not capable of processing the cgroup the process has and always outputs the host memory no matter what your cgroup says.
Tell your app what this resource limit is. It seems like your app does not "feel the need" to free memory as there is plenty. Almost all applications, and frameworks as well, have a switch to define the maximum memory to be consumed. Tell your app this limit, and it will "see" memory pressure and perform a full GC (what I guess could be the problem here)
I've been having trouble with my webapp. My heap memory peak up to nearly to max size for about 30 mins and the it crashes my system.
I have googled and tried nearly everything. I have been monitoring my heap memory using Java VisualVM, jconsole and Oracle Java Mission Control(I know it's outdated).
So what I have tried until know:
Monitored heap memory to see if there is a specific thread running at specific time and it peak the memory. (This is not the case as it doesn't specifically peak at specific times.)
2.Increased my heap memory size.
Followed instructions from:
http://karunsubramanian.com/websphere/top-4-java-heap-related-issues-and-how-to-fix-them/
So my questions are:
Is there any tool that can help me see if I have a memory leak and from where?
Has anyone experienced the same issue.
Any pointers on how to manage this kind of problems.
Btw I am quite new in this area so please be kind.
Tomcat 7 on Windows Server 2012
JAVA 7
If you need more information please comment.
You need to configure the jvm to create a heapdump when an outofmemory occurs.
-XX:+HeapDumpOnOutOfMemoryError
Then analyze the heap dump to find what classes are using the memory
What is the impact to put -d64 into the Liferay 5.2, weblogic 10 over Solaris environment?
What is the impact of choosing Xms=Xmx=8GB and PermGen=1g, in that setup.
[Hands on experience: I see some Null exceptions occasionally, but the application is on and I don't see problems data, or visually. It's a test environment, not any production. ]
Thank you for any comment
These options impact performance so there will not be any functional impact to your application.
-d64 explained : What impact, if any, does the -d64 switch have on Sun JVM resident memory usage?
Xms=Xmx=8GB, PermGen=1g : These are Heap Size settings, these will influence the GC algorithm running time.
I have a web application that is eventually running out of memory when it runs on IIS 7 Windows Server 2008. When I attempt to run a memory profiler against the application to determine the leak, it is not reproducible on my development workstation...Windows Vista.
The GC collection cycles are not consistent between the server and the workstation and it appears the server's collection is not reclaiming all of its memory and is eventually running out. The server becomes non responsive and throws out of memory exceptions.
We have tried setting objects that are surviving too many generations to null...Some improvement was noticed.
Any assistance/recommendations would be greatly appreciated
Tess Ferrandez's blog has some great information on debugging memory leaks using Windbg.
By taking a dump of the running application and then analysing it in Windbg, you should be able to find the source of the leaks you are seeing.
The following entries are probably a good starting point:
Setup (including links to configuring Windbg
Memory Leak Lab 1
Memory Leak Lab 2
Good luck!