We see multiple stuck threads in weblogic while using apache poi to write data to excel.
"[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" Id=2734 RUNNABLE
at javax.xml.namespace.QName.equals(QName.java:369)
at org.apache.xmlbeans.impl.store.Locale.count(Locale.java:2062)
at org.apache.xmlbeans.impl.store.Xobj.count_elements(Xobj.java:2068)
at org.openxmlformats.schemas.spreadsheetml.x2006.main.impl.CTMergeCellsImpl.sizeOfMergeCellArray(Unknown Source)
- locked org.apache.xmlbeans.impl.store.Locale#4e4edf28
at org.apache.poi.xssf.usermodel.XSSFSheet.getNumMergedRegions(XSSFSheet.java:816)
Looks like the issue is with xmlbeans. Restarting weblogic server temporarily fixes the issue. Looking for some help in debugging.
Related
Our production server went OOM because permgen is full. Using jmap -permstat to check the permgen area, we found there were many classes loaded by com.sun.xml.ws.client.WSSServiceDelegatingLoader. The loaded classes are com.sun.proxy.$ProxyXXX, where XXX is an int sequence.
the stacktrace for these classloading is as follow:
eventually, the JVM went OOM, full gc can't reclaim any permgen memory.
What is strange is that if I click System GC in VisualVM, the classes are unloaded and the usage of permgen goes down.
Our JDK version is 1.7.0.80， we have added CMSClassUnloadingEnabled.
-XX:+ExplicitGCInvokesConcurrent
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=60
-XX:+UseParNewGC
-XX:+CMSParalledlRemarkEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:+CMSFullGCsBeforeCompaction=0
-XX:+CMSCLassUnloadingEnabled
-XX:MaxTenuringThreshold=18
-XX:+UseCMSInitialtingOccupancyOnly
-XX:SurvivorRatio=4
-XX:ParallecGCThreads=16
Our code has been running for a long time. The most recent operation is a WebLogic patch. This really confused me. Could someone give me some help with this issue, many thanks!
This is a known bug https://github.com/javaee/metro-jax-ws/issues/1161
Every time a JAX-WS client is created, for instance, using library JAX-WS RI 2.2 which is bundled in Weblogic Server 12.1.3
com.sun.xml.ws.client.WSServiceDelegate$DelegatingLoader#1
Client proxy classes are being loaded into classloader:
([Loaded com.sun.proxy.$Proxy979 from com.sun.xml.ws.client.WSServiceDelegate$DelegatingLoader] )
Solution/Workaround:
Replace JAX-WS client where this bug is solved.
I have a linux trusty on aws m4.xlarge so 4 CPU, 16 GB RAM. It's running a java application on tomcat7 and oracle java 8.
Very frequently the app will hang and won't accept any other connection. Status cake will report it as down since the response times out. Datadog will show threads are maxed out. But there is no increase in CPU (barely 10% of usage). RAM usage remains unchanged during that period.
Only a tomcat restart fixes the problem temporarily(12h approx ). So I have taken a thread dump and seen so many threads in a waiting state. Since this is very new to me, I am blind even with the data.
I was hoping I could get help here and eventually master the art of ciphering a thread dump file. I have attached it here and I have as well uploaded it to fastthread.io and it says there is no problem. I have also uploaded the full threadump on zerobin
I would be very grateful if anyone here can shed some lights on this and I hope it will help others in the same situation. Thanks in advance.
Lots of threads are in WAITING state, and it's absolutely ok for them. For example, there are thread which have the following stack trace:
...
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
...
This only means threads are waiting for any tasks to do.
However, other stacks do not look good.
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.**awaitAvailable**(BasicResourcePool.java:1414)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:606)
- locked <0x000000055c2d3ce0> (a com.mchange.v2.resourcepool.BasicResourcePool)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:526)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:755)
Those threads are waiting for connection to be free in the pool. C3P0 is a pool of database connections. Instead of creating a new connection every time, they are cached in the pool. Upon closing, the connection itself is not closed, but only returned to the pool. So, if hibernate for some reason (or other user) do not close connection after releasing it, then pool can get exhausted.
In order to resolve an issue, you have to find out why some connections are not closed after using. Try to look at your code to do this.
The other option is to temporarily go without C3P0 (pooling). This is not forever, but at least you can check whether this guess is right.
error:org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
serious: The web application appears to have started a thread named [QuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
serious: The web application appears to have started a thread named [QuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
I am not been able to understand the cause of this problem. Can someone suggest me how to resolve this issue? thanks.
Most likely that you didn't call shutdown() method on your scheduler
We have installed java application on tomcat with linux server.
When we shut down tomcat server I am getting following error messages in logs.
created a ThreadLocal with key of type [null] (value [null]) and a value of type [org.apache.lucene.index.SegmentTermEnum] (value [org.apache.lucene.index.SegmentTermEnum#19a419a4]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Can any one help me
How I stop these threads?
And
How prevent these error should not appear again?
Please reply as early as possible we are facing this issue in our production which cause slowing our web application.
Thanks in advance..
I am using below code in my spring webflow application running on weblogic server.
RequestContextHolder.getRequestContext()
.getExternalContext().getNativeRequest()).getSession()
I know that RequestContextHolder uses ThreadLocal to store the requestContext during start and resume steps.
Now coming to the question is there any remote possibility that one thread may access requestContext of other thread (may be because thread is being reused by weblogic or beacuse of some other reason) ?
Upgrading to SWF 2.3.0 solved the issue (data shared between threads)
The SWF version (2.2.1) had a bug. Its ExpressionParser was not thread-safe (JIRA).