I am getting a stuck thread and need help to find out the root cause.
Below is a thread dumps:
"AD Thread Pool-Global15" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#2537cac0 TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.DelayQueue.take(DelayQueue.java:220)
com.singularity.ee.util.javaspecific.scheduler.v.c(v.java:755)
com.singularity.ee.util.javaspecific.scheduler.i.run(i.java:725)
java.lang.Thread.run(Thread.java:745)
"DmsThread-2" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#5b51514f TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)
java.security.AccessController.doPrivileged(Native Method)
oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)
java.lang.Thread.run(Thread.java:745)
"oracle.dfw.common.DiagnosticsEventManager.NotifierThread" waiting for lock java.util.LinkedList#3eb2e717 WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
oracle.dfw.common.DiagnosticsEventManager$NotifierThread.run(DiagnosticsEventManager.java:298)
java.lang.Thread.run(Thread.java:745)
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
weblogic.work.ExecuteThread.run(ExecuteThread.java:233)
"app-producer-network-thread | producer-1" RUNNABLE native
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollsecImpl.doSelect(EPollsecImpl.java:79)
sun.nio.ch.secImpl.lockAndDoSelect(secImpl.java:87)
sun.nio.ch.secImpl.select(secImpl.java:98)
log.app.sec.select(sec.java:672)
log.app.sec.poll(sec.java:396)
log.app.net.poll(net.java:460)
log.app.pro.Send.run(Send.java:239)
app.pro.Send.run(Send.java:163)
java.lang.Thread.run(Thread.java:745)
"logback-1" waiting for lock java.util.concurrent.SynchronousQueue$TransferStack#6edc7ff0 WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
"CacheCleaner" waiting for lock oracle.ias.cache.TimeQ#67322757 TIMED_WAITING
java.lang.Object.wait(Native Method)
oracle.ias.cache.TimeQ.waitForWork(TimeQ.java:178)
oracle.ias.cache.CacheCleaner.run(CacheCleaner.java:116)
"Worker-2" waiting for lock oracle.ias.cache.TaskQ#db4f5af WAITING
java.lang.Object.wait(Native Method)
oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
oracle.ias.cache.WorkerThread.run(ThreadPool.java:303
"Worker-0" waiting for lock oracle.ias.cache.TaskQ#db4f5af WAITING
java.lang.Object.wait(Native Method)
oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)
"RepolessTargetManager" waiting for lock oracle.sysman.emdrep.repoless.RepolessTargetManager#7a62d12e TIMED_WAITING
java.lang.Object.wait(Native Method)
oracle.sysman.emdrep.repoless.RepolessTargetManager.waitNewAgent(RepolessTargetManager.java:483)
oracle.sysman.emdrep.repoless.RepolessTargetManager.run(RepolessTargetManager.java:920)
java.lang.Thread.run(Thread.java:745)
"HealthMonitor" waiting for lock oracle.sysman.util.healthMonitor.HealthMonitor#5dabda0f TIMED_WAITING
java.lang.Object.wait(Native Method)
oracle.sysman.util.healthMonitor.HealthMonitor.run(HealthMonitor.java:282)
java.lang.Thread.run(Thread.java:745)
"Java2D Disposer" waiting for lock java.lang.ref.ReferenceQueue$Lock#7dea38d3 WAITING
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
sun.java2d.Disposer.run(Disposer.java:145)
java.lang.Thread.run(Thread.java:745)
"[STANDBY] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread#2bf20a04 WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
weblogic.work.ExecuteThread.run(ExecuteThread.java:233)
"DmsThread-1" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#5b51514f WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)
java.security.AccessController.doPrivileged(Native Method)
oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)
java.lang.Thread.run(Thread.java:745)
"[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread#55a3e52f WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
weblogic.work.ExecuteThread.run(ExecuteThread.java:233)
"com.bea.osb.transports.jms.TimerThread" waiting for lock java.util.concurrent.Semaphore$NonfairSync#37b1485f WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
com.bea.wli.sb.transports.jms.JmsAsyncMessageTable$TimeoutThread.run(JmsAsyncMessageTable.java:250
"AD Thread-Hotspot Processor0" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#f080685 TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.DelayQueue.take(DelayQueue.java:220)
com.singularity.ee.util.javaspecific.scheduler.v.c(v.java:755)
com.singularity.ee.util.javaspecific.scheduler.v.take(v.java:695)
com.singularity.ee.util.javaspecific.scheduler.w.a(w.java:500)
com.singularity.ee.util.javaspecific.scheduler.i.run(i.java:725)
java.lang.Thread.run(Thread.java:745)
"[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread#722e5eff WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
weblogic.work.ExecuteThread.run(ExecuteThread.java:233)
"[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread#4014b4d WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205
weblogic.work.ExecuteThread.run(ExecuteThread.java:233)
"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread#57a9acd WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)
weblogic.work.ExecuteThread.run(ExecuteThread.java:233)
"Thread-60" RUNNABLE native
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:152)
java.net.SocketInputStream.read(SocketInputStream.java:122)
java.net.SocketInputStream.read(SocketInputStream.java:210)
oracle.ons.InputBuffer.getNextString(InputBuffer.java:137)
oracle.ons.ReceiverThread.run(ReceiverThread.java:295)
"OracleTimeoutPollingThread" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
oracle.jdbc.driver.OracleTimeoutPollingThread.run(OracleTimeoutPollingThread.java:155)
"oracle.dfw.incident.IncidentCreatorThread" waiting for lock java.util.ArrayList#596c1061 WAITING
oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl$IncidentCreatorThread.run(DiagnosticsDataExtractorImpl.java:2144)
"oracle.dfw.impl.incident.ADRHelper - purge incidents" waiting for lock java.util.TaskQueue#45a77e24 TIMED_WAITING
java.util.TimerThread.mainLoop(Timer.java:552)
"oracle.dfw.impl.incident - ADR Executor " waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#774a9135 WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.lang.Thread.run(Thread.java:745)
"DFW Diagnostic Dump Sampling - JavaClassHistogram" waiting for lock [B#40836bc2 TIMED_WAITING
java.lang.Object.wait(Native Method)
oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:547)
"DFW Diagnostic Dump Sampling - JVMThreadDump" waiting for lock [B#7417aea4 TIMED_WAITING
java.lang.Object.wait(Native Method)
oracle.dfw.sampling.DumpSampling.run(DumpSampling.java:547)
"LogFlusher" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
"LogFlusher" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
"LogFlusher" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
oracle.core.ojdl.BufferedLogWriter$Flusher.run(BufferedLogWriter.java:409)
"Timer-4" waiting for lock java.util.TaskQueue#56183c4b TIMED_WAITING
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
"Thread-20" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#2c7ee04e WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)
weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)
java.lang.Thread.run(Thread.java:745)
"Timer-3" waiting for lock java.util.TaskQueue#9512a14 WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
"Runtime Authorization Event Notifier" waiting for lock oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl#16e2e213 WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl.run(RuntimeEventNotifierImpl.java:230)
java.lang.Thread.run(Thread.java:745)
"JMX Framework document pooling thread" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
oracle.as.config.notification.filesystem.WatchingDocumentChangeNotifier.run(WatchingDocumentChangeNotifier.java:252)
java.lang.Thread.run(Thread.java:745)
"JPS FileChangeNotifier<clean-only>" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#2ca6ce6a TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
"DoSManager" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
com.octetstring.vde.DoSManager.run(DoSManager.java:433)
"VDE Transaction Processor Thread" waiting for lock com.octetstring.vde.backend.standard.TransactionProcessor#2c568d5b WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)
com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" waiting for lock weblogic.socket.PosixSocketMuxer$1#3968b775 BLOCKED
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" waiting for lock weblogic.socket.PosixSocketMuxer$1#3968b775 BLOCKED
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" RUNNABLE native
weblogic.socket.PosixSocketMuxer.poll(Native Method)
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" waiting for lock weblogic.socket.PosixSocketMuxer$1#3968b775 BLOCKED
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"Thread-12" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#34fb382c WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)
weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)
java.lang.Thread.run(Thread.java:745)
"Timer-2" waiting for lock java.util.TaskQueue#6f9ecd9 TIMED_WAITING
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
"weblogic.timers.TimerThread" waiting for lock java.lang.Object#118edefb TIMED_WAITING
java.lang.Object.wait(Native Method)
weblogic.timers.internal.TimerThread.doWait(TimerThread.java:114)
weblogic.timers.internal.TimerThread.access$000(TimerThread.java:22)
weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:290)
"weblogic.time.TimeEventGenerator" waiting for lock weblogic.time.common.internal.TimeTable#19aa5071 TIMED_WAITING
java.lang.Object.wait(Native Method)
weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)
weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)
java.lang.Thread.run(Thread.java:745)
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE
java.util.ArrayList.fastRemove(ArrayList.java:523)
java.util.ArrayList.remove(ArrayList.java:511)
com.bea.adaptive.harvester.jmx.RegistrationManager.instanceDeletedInternal(RegistrationManager.java:290)
com.bea.adaptive.harvester.jmx.RegistrationManager.instanceDeleted(RegistrationManager.java:257)
com.bea.adaptive.mbean.typing.JMXNotificationListener.deletedInstance(JMXNotificationListener.java:291)
com.bea.adaptive.mbean.typing.JMXNotificationListener.processRegistrationEvent(JMXNotificationListener.java:313)
com.bea.adaptive.mbean.typing.JMXNotificationListener.access$300(JMXNotificationListener.java:37)
com.bea.adaptive.mbean.typing.JMXNotificationListener$MBeanRegistrationWork.run(JMXNotificationListener.java:108)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
"Timer-1" waiting for lock java.util.TaskQueue#5f09fd6d TIMED_WAITING
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
"Timer-0" waiting for lock java.util.TaskQueue#717095c1 WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
"AD Thread-Top Backends Processor0" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#1b2ca3e0 TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
java.util.concurrent.DelayQueue.take(DelayQueue.java:220)
com.singularity.ee.util.javaspecific.scheduler.v.c(v.java:755)
com.singularity.ee.util.javaspecific.scheduler.v.take(v.java:695)
com.singularity.ee.util.javaspecific.scheduler.w.a(w.java:500)
com.singularity.ee.util.javaspecific.scheduler.i.run(i.java:725)
java.lang.Thread.run(Thread.java:745)
"AD Thread-Metric Reporter1" waiting for lock java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#32fa1581 WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
java.util.concurrent.DelayQueue.take(DelayQueue.java:215)
com.singularity.ee.util.javaspecific.scheduler.v.c(v.java:755)
com.singularity.ee.util.javaspecific.scheduler.v.take(v.java:695)
com.singularity.ee.util.javaspecific.scheduler.w.a(w.java:500
com.singularity.ee.util.javaspecific.scheduler.i.run(i.java:725)
java.lang.Thread.run(Thread.java:745)
"AD Thread-Config Poller" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
com.singularity.util.org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:101)
"Thread-1" TIMED_WAITING
java.lang.Thread.sleep(Native Method)
com.singularity.util.org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:101)
"Signal Dispatcher" RUNNABLE
"Finalizer" waiting for lock java.lang.ref.ReferenceQueue$Lock#3272350 WAITING
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" waiting for lock java.lang.ref.Reference$Lock#5b22899c WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
"main" waiting for lock weblogic.t3.srvr.T3Srvr#3ecacc87 WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:503)
weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:983)
weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:492)
weblogic.Server.main(Server.java:71)
`
Firstly never troubleshoot stuck threads using only 1 dump, always take 3-4 dumps each spaced about 3-4 secs apart..
The thread dump you have provided seems to be just fine, the one thread marked stuck seems to one where you are trying to uninstall a composite, something uninstall does take time and cause 1 thread to be stuck, this has little or no impact whatsoever on the servers functionality.
Related
I'm having a problem in a glassfish 3 server. The log is throwing messages
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 256
I did a couple of thread dumps and found that there are 16 threads waiting for the same lock, for example:
"__ejb-thread-pool1" daemon prio=6 tid=0x39657c00 nid=0x1c08 waiting on condition [0x3297f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x117b2cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
"__ejb-thread-pool2" daemon prio=6 tid=0x38408c00 nid=0x1a3c waiting on condition [0x3ad3f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x117b2cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
...
is this normal? I thought I was going to find a thread owning the lock in the thread dump, but there isn't. Please help.. I'm new to thread analysis.
If some executor service has 16 theads doing nothing, then yes it's perfectly normal that they are all waiting for something to show in the task queue.
The rejected executions likely occured at another moment than when you snapshot threads. It only means there was a spike in submissions and the queue overflowed earlier.
Careful though, these rejected executions may not even be on the executor service you are looking at. It depends how many other executor services there are elsewhere.
There are many Timer threads in my jvm. From thread dump, i get this:
"Timer-19" - Thread t#159
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <134c08ad> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
In my program i don't use Timer, it confused me a lot.
So please help!
attach a debugger
set a breakpoint on the Timer constructor
wait until it triggers
inspect the call stack to find the caller
I have this deadlock scenario which brings the application to a complete halt, not sure whats going on, if you could show some light. Thank you
This is the consumer configuration
<jms:message-driven-channel-adapter
id="InBoundZFlow"
connection-factory="wmqConnFactory"
destination="ResponseQueue"
channel="responseInChannel"
auto-startup="false"
max-messages-per-task="20"
receive-timeout="10000"
concurrent-consumers="2"
task-executor="receivingTaskExecutor"
max-concurrent-consumers="20"
idle-consumer-limit="1"
idle-task-execution-limit="1"
recovery-interval="60000"/>
receivingTaskExecutor has 20 threads.
This is what i see for some of the blocking threads -
"receivingTaskExecutor-19" - Thread t#226
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1142)
- waiting to lock <1f7cd479> (a java.lang.Object) owned by "receivingTaskExecutor-11" t#111
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
"rceivingTaskExecutor-16" - Thread t#184
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.AbstractJmsListeningContainer.isRunning(AbstractJmsListeningContainer.java:347)
- waiting to lock <3c616415> (a java.lang.Object) owned by "receivingTaskExecutor-7" t#69
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:995)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <712a95c7> (a java.util.concurrent.ThreadPoolExecutor$Worker)
----------------------------------------------------
"mqReceivingTaskExecutor-7" - Thread t#69
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1142)
- waiting to lock <1f7cd479> (a java.lang.Object) owned by "mqReceivingTaskExecutor-11" t#111
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1043)
- locked <3c616415> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <a01ec0b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
---------------------------------------------------------
"receivingTaskExecutor-11" - Thread t#111
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- waiting to lock <1df1bdab> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by "receivingTaskExecutor-5" t#65
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock(RemoteConnection.java:2842)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConnection.java:1812)
at com.ibm.mq.jmqi.remote.impl.RemoteSession.sendTSH(RemoteSession.java:758)
at com.ibm.mq.jmqi.remote.impl.RemoteSession.exchangeTSH(RemoteSession.java:1237)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.spiNotify(RemoteFAP.java:4612)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiNotify(RemoteFAP.java:4449)
at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.jmqiNotify(JmqiInterceptAdapter.java:579)
at com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.setRunning(WMQSyncConsumerShadow.java:462)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.stop(WMQMessageConsumer.java:535)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.stopUnconditional(JmsMessageConsumerImpl.java:706)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.stop(JmsSessionImpl.java:2014)
- locked <2a02832d> (a java.util.Collections$SynchronizedSet)
- locked <af214e7> (a java.lang.Object)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.close(JmsSessionImpl.java:315)
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.close(JmsConnectionImpl.java:300)
- locked <1b3a4f4e> (a com.ibm.msg.client.jms.internal.State)
at com.ibm.mq.jms.MQConnection.close(MQConnection.java:98)
at org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:395)
- locked <1f7cd479> (a java.lang.Object)
at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:885)
at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:861)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1013)
- locked <112c97d9> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <33d65cd0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- locked <6a5305e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"receivingTaskExecutor-5" - Thread t#65
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1142)
- waiting to lock <1f7cd479> (a java.lang.Object) owned by "receivingTaskExecutor-11" t#111
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <19f71b53> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- locked <1df1bdab> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
This looks like a bug in the mq client library.
receivingTaskExecutor-5 is holding a lock owned by the client library but has no client library stack frames on its stack.
receivingTaskExecutor-11 is trying to acquire that lock (while holding a DMLC lock; the other threads are waiting for that).
So something happened in the client library on receivingTaskExecutor-5 that caused it to leave the lock locked.
I suggest you contact IBM support.
I have noticed that after few days running this stopped consuming messages from the queue with no known reason or trace, i have to restart it to continue processing.
Are there any known reasons behind this, also how to find out what happened in such cases and ways to fix it.
I am using WMQ and need to maintain transaction while processing a message. Thanks You.
<jms:message-driven-channel-adapter
acknowledge="transacted"
destination="inboundQueue"
channel="inboundChannel"
auto-startup="false"
max-concurrent-consumers="5"
transaction-manager="transactionManager"
recovery-interval="60000"/>
Also i will like to turn on debug on this consumer thread(s) to print that its trying to get a message, is there a way to do that.
EDITED*
I see that this had deadlocked, please let me know the cause and possible resolution, thanks.
"receivingTaskExecutor-12" - Thread t#961
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.AbstractJmsListeningContainer.isRunning(AbstractJmsListeningContainer.java:348)
- waiting to lock <66a37d72> (a java.lang.Object) owned by "mqReceivingTaskExecutor-10" t#878
at org.springframework.jms.listener.DefaultMessageListenerContainer.scheduleNewInvokerIfAppropriate(DefaultMessageListenerContainer.java:715)
at org.springframework.jms.listener.DefaultMessageListenerContainer.messageReceived(DefaultMessageListenerContainer.java:692)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:996)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <5a9e40d2> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"receivingTaskExecutor-10" - Thread t#878
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <35212499> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:328)
at org.springframework.integration.util.CallerBlocksPolicy.rejectedExecution(CallerBlocksPolicy.java:58)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:822)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1373)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:254)
at org.springframework.jms.listener.DefaultMessageListenerContainer.doRescheduleTask(DefaultMessageListenerContainer.java:682)
at org.springframework.jms.listener.AbstractJmsListeningContainer.rescheduleTaskIfNecessary(AbstractJmsListeningContainer.java:519)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1036)
- locked <66a37d72> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <78ff22ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Most likely is some problem up-stack (hung in user code somewhere). Use jstack <pid> to see what the container thread is doing when you hit this condition.
I've got a deadlock in my application, but there is no obvious locking instance in the stack trace. How is this possible? Is this a bug?
jstack -l output
Full thread dump OpenJDK 64-Bit Server VM (19.0-b09 mixed mode):
"Attach Listener" daemon prio=10 tid=0x000000000120d000 nid=0x34a9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"queue_1" prio=10 tid=0x0000000001043800 nid=0x2ecc waiting for monitor entry [0x00000000426f2000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.wordstat.AbstractWordstatEngine.startQueueProcessing(AbstractWordstatEngine.java:56)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.act(WordstatEngineManager.java:155)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.run(WordstatEngineManager.java:145)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_2" prio=10 tid=0x0000000001043000 nid=0x2ecb waiting for monitor entry [0x00000000413a5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.wordstat.AbstractWordstatEngine.startQueueProcessing(AbstractWordstatEngine.java:56)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.act(WordstatEngineManager.java:155)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.run(WordstatEngineManager.java:145)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_1" prio=10 tid=0x0000000001a7c800 nid=0xdb8 waiting for monitor entry [0x00000000433ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_2" prio=10 tid=0x0000000003f24800 nid=0xdb5 waiting for monitor entry [0x00000000431fd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_3" prio=10 tid=0x0000000001578000 nid=0xdb2 waiting for monitor entry [0x00000000424f0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_4" prio=10 tid=0x0000000003f20000 nid=0xdad waiting for monitor entry [0x0000000042ffb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_5" prio=10 tid=0x0000000001091000 nid=0xda3 waiting for monitor entry [0x00000000428f4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_6" prio=10 tid=0x0000000001592000 nid=0xd9d waiting for monitor entry [0x0000000042af6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_7" prio=10 tid=0x00000000030f5000 nid=0xd98 waiting for monitor entry [0x0000000042efa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_8" prio=10 tid=0x0000000001094000 nid=0xd92 waiting for monitor entry [0x0000000042bf7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_9" prio=10 tid=0x0000000001485800 nid=0xd5c waiting for monitor entry [0x0000000041e13000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_10" prio=10 tid=0x0000000003f26800 nid=0xd42 waiting for monitor entry [0x00000000432fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_11" prio=10 tid=0x00000000032ed000 nid=0xd34 waiting for monitor entry [0x00000000425f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_12" prio=10 tid=0x00000000011b8000 nid=0xd05 waiting for monitor entry [0x00000000427f3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_13" prio=10 tid=0x000000000104b800 nid=0xcb7 waiting for monitor entry [0x00000000422ee000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_14" prio=10 tid=0x00000000014f9800 nid=0xb3c waiting for monitor entry [0x00000000411f0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_15" prio=10 tid=0x0000000001213000 nid=0xb25 waiting for monitor entry [0x00000000429f5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_16" prio=10 tid=0x00000000032fa800 nid=0xabe waiting for monitor entry [0x00000000423ef000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_17" prio=10 tid=0x0000000003f22000 nid=0xaaf waiting for monitor entry [0x00000000430fc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_18" prio=10 tid=0x0000000001595800 nid=0xa9f waiting for monitor entry [0x0000000042cf8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_3" prio=10 tid=0x00000000014f2000 nid=0x890 waiting for monitor entry [0x0000000040b05000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_4" prio=10 tid=0x0000000001039000 nid=0x88f waiting for monitor entry [0x000000004081b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_5" prio=10 tid=0x0000000001786000 nid=0x88e waiting for monitor entry [0x000000004071a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_6" prio=10 tid=0x0000000001f96000 nid=0x88d waiting for monitor entry [0x00000000421ed000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_7" prio=10 tid=0x00000000021a3800 nid=0x88c waiting for monitor entry [0x0000000041d12000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_8" prio=10 tid=0x00000000016b9800 nid=0x88b waiting on condition [0x0000000040a04000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:136)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"DestroyJavaVM" prio=10 tid=0x0000000000cf8800 nid=0x86d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-4" prio=10 tid=0x0000000001495800 nid=0x88a runnable [0x0000000041c11000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00007f9fd5878c78> (a sun.nio.ch.Util$1)
- locked <0x00007f9fd5878c90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00007f9fd5878c00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:307)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"server-timer" daemon prio=10 tid=0x0000000001613000 nid=0x889 in Object.wait() [0x0000000040fcc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd5878d78> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00007f9fd5878d78> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"InternetCheckerThread" daemon prio=10 tid=0x0000000001310800 nid=0x884 waiting on condition [0x000000004032c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.utility.InternetCheckerThread.run(InternetCheckerThread.java:61)
Locked ownable synchronizers:
- None
"ConsoleThread" prio=10 tid=0x00000000015e3800 nid=0x883 runnable [0x0000000040ecb000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:236)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00007f9fd54b01c0> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:282)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:176)
- locked <0x00007f9fd56e8a48> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:153)
at java.io.BufferedReader.readLine(BufferedReader.java:316)
- locked <0x00007f9fd56e8a48> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:379)
at org.fs.jcs.JcsCliThread.run(JcsCliThread.java:97)
Locked ownable synchronizers:
- None
"WordstScheduleThread" prio=10 tid=0x000000000172d800 nid=0x882 waiting on condition [0x0000000041f68000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.engines.wordstat.WordstatEngineManager.launchAndWait(WordstatEngineManager.java:104)
at org.seosrv.schedule.WordstatSchedule.fullCheck(WordstatSchedule.java:64)
at org.seosrv.schedule.WordstatSchedule.run(WordstatSchedule.java:100)
Locked ownable synchronizers:
- None
"PosScheduleThread" prio=10 tid=0x00000000012d1800 nid=0x881 waiting on condition [0x000000004197c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.engines.search.SearchEngineManager.launchAndWait(SearchEngineManager.java:121)
at org.seosrv.schedule.PositionSchedule.fullCheck(PositionSchedule.java:135)
at org.seosrv.schedule.PositionSchedule.run(PositionSchedule.java:326)
Locked ownable synchronizers:
- None
"net.sf.ehcache.CacheManager#45c1f5b2" daemon prio=10 tid=0x0000000001787000 nid=0x880 in Object.wait() [0x0000000040c97000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd56e8e20> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00007f9fd56e8e20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x0000000000d8e800 nid=0x876 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread1" daemon prio=10 tid=0x0000000000d89000 nid=0x875 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread0" daemon prio=10 tid=0x0000000000d87000 nid=0x874 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x0000000000d85800 nid=0x873 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x0000000000d60800 nid=0x870 in Object.wait() [0x00000000416d2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd54b0550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00007f9fd54b0550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000000d5f000 nid=0x86f in Object.wait() [0x000000004049a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd54b0630> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00007f9fd54b0630> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000000d58000 nid=0x86e runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000000d91800 nid=0x877 waiting on condition
JNI global references: 1608
ConnectionManagerImpl is a singleton, it's guaranteed that there is no explicit synchronization on it via synchronized (conManager) {...} within the hand-written code. Here is it's source:
public class ConnectionManagerImpl implements ConnectionManager {
private int maxCons = 30;
private int lastGrantedConnectionId = 0;
private final List <Integer> allConnections = new ArrayList <Integer>();
#Override
public synchronized int getConnectionId(){
if (allConnections.size() >= maxCons) return -1;
if (!InternetChecker.waitForInternet()) { //unsynch
return -1;
}
int newId = ++lastGrantedConnectionId;
allConnections.add(newId);
return newId;
}
#Override
public synchronized void releaseConnectionId(int connectionId){
if (!allConnections.remove(Integer.valueOf(connectionId))) {
LogUtils.logError(new InternalException("Finished thread never was registred: " + "networkId = " + connectionId + ", active network IDs: " + allConnections));
}
}
}
VPS configuration: CentOS 5 x64, OpenJDK 64-Bit 19.0-b09
App runned with
-XX:HeapDumpPath=path_here
-XX:+HeapDumpOnOutOfMemoryError
-XX:-PrintClassHistogram
-XX:-UseCompressedOops
I removed a InternetChecker.waitForInternet() call like Udi Cohen proposed and here is what I got:
Exception in thread "queue_1" java.lang.IllegalMonitorStateException
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Exception in thread "queue_2" java.lang.IllegalMonitorStateException
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (synchronizer.cpp:1954), pid=15269, tid=1104439616
# guarantee(mid->header()->is_neutral()) failed: invariant
#
# JRE version: 6.0_20-b20
# Java VM: OpenJDK 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 )
# Derivative: IcedTea6 1.9.8
# Distribution: CentOS release 5.6 (Final), package rhel-1.22.1.9.8.el5_6-x86_64
# An error report file with more information is saved as:
# /hs_err_pid15269.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
I've already noticed the IllegalMonitorStateException under Oracle JDK 1.6.0_27 for c3p0 thread, but I thought that was a c3p0 problem. Now i've tested it better and got similiar error with IBM SDK.
It seems like this is a virtualization bug. Thank you for your efforts.
(Disclaimer: my company develops JProfiler)
Obviously, the jstack output is missing something in this case.
I would recommend to consult a different tool that gets this information from the JVMTI profiling interface. In JProfiler, you get a view like this:
Do you have any specific reason for using OpenJDK?
If not you can switch to another JDK to make sure this is not a bug in the JDK implementation.