I have noticed that after few days running this stopped consuming messages from the queue with no known reason or trace, i have to restart it to continue processing.
Are there any known reasons behind this, also how to find out what happened in such cases and ways to fix it.
I am using WMQ and need to maintain transaction while processing a message. Thanks You.
<jms:message-driven-channel-adapter
acknowledge="transacted"
destination="inboundQueue"
channel="inboundChannel"
auto-startup="false"
max-concurrent-consumers="5"
transaction-manager="transactionManager"
recovery-interval="60000"/>
Also i will like to turn on debug on this consumer thread(s) to print that its trying to get a message, is there a way to do that.
EDITED*
I see that this had deadlocked, please let me know the cause and possible resolution, thanks.
"receivingTaskExecutor-12" - Thread t#961
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.AbstractJmsListeningContainer.isRunning(AbstractJmsListeningContainer.java:348)
- waiting to lock <66a37d72> (a java.lang.Object) owned by "mqReceivingTaskExecutor-10" t#878
at org.springframework.jms.listener.DefaultMessageListenerContainer.scheduleNewInvokerIfAppropriate(DefaultMessageListenerContainer.java:715)
at org.springframework.jms.listener.DefaultMessageListenerContainer.messageReceived(DefaultMessageListenerContainer.java:692)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1102)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:996)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <5a9e40d2> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"receivingTaskExecutor-10" - Thread t#878
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <35212499> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:328)
at org.springframework.integration.util.CallerBlocksPolicy.rejectedExecution(CallerBlocksPolicy.java:58)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:822)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1373)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:254)
at org.springframework.jms.listener.DefaultMessageListenerContainer.doRescheduleTask(DefaultMessageListenerContainer.java:682)
at org.springframework.jms.listener.AbstractJmsListeningContainer.rescheduleTaskIfNecessary(AbstractJmsListeningContainer.java:519)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1036)
- locked <66a37d72> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <78ff22ed> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Most likely is some problem up-stack (hung in user code somewhere). Use jstack <pid> to see what the container thread is doing when you hit this condition.
Related
I'm having a problem in a glassfish 3 server. The log is throwing messages
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 256
I did a couple of thread dumps and found that there are 16 threads waiting for the same lock, for example:
"__ejb-thread-pool1" daemon prio=6 tid=0x39657c00 nid=0x1c08 waiting on condition [0x3297f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x117b2cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
"__ejb-thread-pool2" daemon prio=6 tid=0x38408c00 nid=0x1a3c waiting on condition [0x3ad3f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x117b2cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
...
is this normal? I thought I was going to find a thread owning the lock in the thread dump, but there isn't. Please help.. I'm new to thread analysis.
If some executor service has 16 theads doing nothing, then yes it's perfectly normal that they are all waiting for something to show in the task queue.
The rejected executions likely occured at another moment than when you snapshot threads. It only means there was a spike in submissions and the queue overflowed earlier.
Careful though, these rejected executions may not even be on the executor service you are looking at. It depends how many other executor services there are elsewhere.
There are many Timer threads in my jvm. From thread dump, i get this:
"Timer-19" - Thread t#159
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <134c08ad> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
In my program i don't use Timer, it confused me a lot.
So please help!
attach a debugger
set a breakpoint on the Timer constructor
wait until it triggers
inspect the call stack to find the caller
I have this deadlock scenario which brings the application to a complete halt, not sure whats going on, if you could show some light. Thank you
This is the consumer configuration
<jms:message-driven-channel-adapter
id="InBoundZFlow"
connection-factory="wmqConnFactory"
destination="ResponseQueue"
channel="responseInChannel"
auto-startup="false"
max-messages-per-task="20"
receive-timeout="10000"
concurrent-consumers="2"
task-executor="receivingTaskExecutor"
max-concurrent-consumers="20"
idle-consumer-limit="1"
idle-task-execution-limit="1"
recovery-interval="60000"/>
receivingTaskExecutor has 20 threads.
This is what i see for some of the blocking threads -
"receivingTaskExecutor-19" - Thread t#226
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1142)
- waiting to lock <1f7cd479> (a java.lang.Object) owned by "receivingTaskExecutor-11" t#111
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
"rceivingTaskExecutor-16" - Thread t#184
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.AbstractJmsListeningContainer.isRunning(AbstractJmsListeningContainer.java:347)
- waiting to lock <3c616415> (a java.lang.Object) owned by "receivingTaskExecutor-7" t#69
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:995)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <712a95c7> (a java.util.concurrent.ThreadPoolExecutor$Worker)
----------------------------------------------------
"mqReceivingTaskExecutor-7" - Thread t#69
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1142)
- waiting to lock <1f7cd479> (a java.lang.Object) owned by "mqReceivingTaskExecutor-11" t#111
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1043)
- locked <3c616415> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <a01ec0b> (a java.util.concurrent.ThreadPoolExecutor$Worker)
---------------------------------------------------------
"receivingTaskExecutor-11" - Thread t#111
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- waiting to lock <1df1bdab> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by "receivingTaskExecutor-5" t#65
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.requestSendLock(RemoteConnection.java:2842)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConnection.java:1812)
at com.ibm.mq.jmqi.remote.impl.RemoteSession.sendTSH(RemoteSession.java:758)
at com.ibm.mq.jmqi.remote.impl.RemoteSession.exchangeTSH(RemoteSession.java:1237)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.spiNotify(RemoteFAP.java:4612)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiNotify(RemoteFAP.java:4449)
at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.jmqiNotify(JmqiInterceptAdapter.java:579)
at com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.setRunning(WMQSyncConsumerShadow.java:462)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.stop(WMQMessageConsumer.java:535)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.stopUnconditional(JmsMessageConsumerImpl.java:706)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.stop(JmsSessionImpl.java:2014)
- locked <2a02832d> (a java.util.Collections$SynchronizedSet)
- locked <af214e7> (a java.lang.Object)
at com.ibm.msg.client.jms.internal.JmsSessionImpl.close(JmsSessionImpl.java:315)
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.close(JmsConnectionImpl.java:300)
- locked <1b3a4f4e> (a com.ibm.msg.client.jms.internal.State)
at com.ibm.mq.jms.MQConnection.close(MQConnection.java:98)
at org.springframework.jms.connection.ConnectionFactoryUtils.releaseConnection(ConnectionFactoryUtils.java:80)
at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:395)
- locked <1f7cd479> (a java.lang.Object)
at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:885)
at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:861)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1013)
- locked <112c97d9> (a java.lang.Object)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <33d65cd0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- locked <6a5305e> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"receivingTaskExecutor-5" - Thread t#65
java.lang.Thread.State: BLOCKED
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1142)
- waiting to lock <1f7cd479> (a java.lang.Object) owned by "receivingTaskExecutor-11" t#111
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1002)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <19f71b53> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- locked <1df1bdab> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
This looks like a bug in the mq client library.
receivingTaskExecutor-5 is holding a lock owned by the client library but has no client library stack frames on its stack.
receivingTaskExecutor-11 is trying to acquire that lock (while holding a DMLC lock; the other threads are waiting for that).
So something happened in the client library on receivingTaskExecutor-5 that caused it to leave the lock locked.
I suggest you contact IBM support.
When I use
jstack -l pid
to see the thread's dump info, I get result as follow:
"Attach Listener" daemon prio=10 tid=0x01e4a800 nid=0x109c waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
My questions are as follow:
What's the function of "Attach Listener" thread?
When does this thread be created? Where? I can't find it.
Thank you very much.
See here http://openjdk.java.net/groups/hotspot/docs/Serviceability.html
Find "Dynamic Attachâ
Dynamic attach has an attach listener thread in the target JVM. This
is a thread that is started when the first attach request occurs.
I've got a deadlock in my application, but there is no obvious locking instance in the stack trace. How is this possible? Is this a bug?
jstack -l output
Full thread dump OpenJDK 64-Bit Server VM (19.0-b09 mixed mode):
"Attach Listener" daemon prio=10 tid=0x000000000120d000 nid=0x34a9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"queue_1" prio=10 tid=0x0000000001043800 nid=0x2ecc waiting for monitor entry [0x00000000426f2000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.wordstat.AbstractWordstatEngine.startQueueProcessing(AbstractWordstatEngine.java:56)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.act(WordstatEngineManager.java:155)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.run(WordstatEngineManager.java:145)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_2" prio=10 tid=0x0000000001043000 nid=0x2ecb waiting for monitor entry [0x00000000413a5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.wordstat.AbstractWordstatEngine.startQueueProcessing(AbstractWordstatEngine.java:56)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.act(WordstatEngineManager.java:155)
at org.seosrv.engines.wordstat.WordstatEngineManager$EngineQueueXable.run(WordstatEngineManager.java:145)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_1" prio=10 tid=0x0000000001a7c800 nid=0xdb8 waiting for monitor entry [0x00000000433ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_2" prio=10 tid=0x0000000003f24800 nid=0xdb5 waiting for monitor entry [0x00000000431fd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_3" prio=10 tid=0x0000000001578000 nid=0xdb2 waiting for monitor entry [0x00000000424f0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_4" prio=10 tid=0x0000000003f20000 nid=0xdad waiting for monitor entry [0x0000000042ffb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_5" prio=10 tid=0x0000000001091000 nid=0xda3 waiting for monitor entry [0x00000000428f4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_6" prio=10 tid=0x0000000001592000 nid=0xd9d waiting for monitor entry [0x0000000042af6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_7" prio=10 tid=0x00000000030f5000 nid=0xd98 waiting for monitor entry [0x0000000042efa000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_8" prio=10 tid=0x0000000001094000 nid=0xd92 waiting for monitor entry [0x0000000042bf7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_9" prio=10 tid=0x0000000001485800 nid=0xd5c waiting for monitor entry [0x0000000041e13000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_10" prio=10 tid=0x0000000003f26800 nid=0xd42 waiting for monitor entry [0x00000000432fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_11" prio=10 tid=0x00000000032ed000 nid=0xd34 waiting for monitor entry [0x00000000425f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_12" prio=10 tid=0x00000000011b8000 nid=0xd05 waiting for monitor entry [0x00000000427f3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_13" prio=10 tid=0x000000000104b800 nid=0xcb7 waiting for monitor entry [0x00000000422ee000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_14" prio=10 tid=0x00000000014f9800 nid=0xb3c waiting for monitor entry [0x00000000411f0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_15" prio=10 tid=0x0000000001213000 nid=0xb25 waiting for monitor entry [0x00000000429f5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_16" prio=10 tid=0x00000000032fa800 nid=0xabe waiting for monitor entry [0x00000000423ef000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_17" prio=10 tid=0x0000000003f22000 nid=0xaaf waiting for monitor entry [0x00000000430fc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"actor_18" prio=10 tid=0x0000000001595800 nid=0xa9f waiting for monitor entry [0x0000000042cf8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.releaseConnectionId(ConnectionManagerImpl.java:29)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.releaseConnectionId(ConnectionUtils.java:19)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.act(SearchEngineContainer.java:352)
at org.seosrv.engines.search.SearchEngineContainer$PositionCheckerXable.run(SearchEngineContainer.java:266)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_3" prio=10 tid=0x00000000014f2000 nid=0x890 waiting for monitor entry [0x0000000040b05000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_4" prio=10 tid=0x0000000001039000 nid=0x88f waiting for monitor entry [0x000000004081b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_5" prio=10 tid=0x0000000001786000 nid=0x88e waiting for monitor entry [0x000000004071a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_6" prio=10 tid=0x0000000001f96000 nid=0x88d waiting for monitor entry [0x00000000421ed000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_7" prio=10 tid=0x00000000021a3800 nid=0x88c waiting for monitor entry [0x0000000041d12000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
- waiting to lock <0x00007f9fd56f4ab8> (a org.seosrv.manager.ConnectionManagerImpl)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"queue_8" prio=10 tid=0x00000000016b9800 nid=0x88b waiting on condition [0x0000000040a04000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:136)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"DestroyJavaVM" prio=10 tid=0x0000000000cf8800 nid=0x86d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-4" prio=10 tid=0x0000000001495800 nid=0x88a runnable [0x0000000041c11000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00007f9fd5878c78> (a sun.nio.ch.Util$1)
- locked <0x00007f9fd5878c90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00007f9fd5878c00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:307)
at java.lang.Thread.run(Thread.java:636)
Locked ownable synchronizers:
- None
"server-timer" daemon prio=10 tid=0x0000000001613000 nid=0x889 in Object.wait() [0x0000000040fcc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd5878d78> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00007f9fd5878d78> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"InternetCheckerThread" daemon prio=10 tid=0x0000000001310800 nid=0x884 waiting on condition [0x000000004032c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.utility.InternetCheckerThread.run(InternetCheckerThread.java:61)
Locked ownable synchronizers:
- None
"ConsoleThread" prio=10 tid=0x00000000015e3800 nid=0x883 runnable [0x0000000040ecb000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:236)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00007f9fd54b01c0> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:282)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:176)
- locked <0x00007f9fd56e8a48> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:153)
at java.io.BufferedReader.readLine(BufferedReader.java:316)
- locked <0x00007f9fd56e8a48> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:379)
at org.fs.jcs.JcsCliThread.run(JcsCliThread.java:97)
Locked ownable synchronizers:
- None
"WordstScheduleThread" prio=10 tid=0x000000000172d800 nid=0x882 waiting on condition [0x0000000041f68000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.engines.wordstat.WordstatEngineManager.launchAndWait(WordstatEngineManager.java:104)
at org.seosrv.schedule.WordstatSchedule.fullCheck(WordstatSchedule.java:64)
at org.seosrv.schedule.WordstatSchedule.run(WordstatSchedule.java:100)
Locked ownable synchronizers:
- None
"PosScheduleThread" prio=10 tid=0x00000000012d1800 nid=0x881 waiting on condition [0x000000004197c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.seosrv.engines.search.SearchEngineManager.launchAndWait(SearchEngineManager.java:121)
at org.seosrv.schedule.PositionSchedule.fullCheck(PositionSchedule.java:135)
at org.seosrv.schedule.PositionSchedule.run(PositionSchedule.java:326)
Locked ownable synchronizers:
- None
"net.sf.ehcache.CacheManager#45c1f5b2" daemon prio=10 tid=0x0000000001787000 nid=0x880 in Object.wait() [0x0000000040c97000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd56e8e20> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:531)
- locked <0x00007f9fd56e8e20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:484)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=10 tid=0x0000000000d8e800 nid=0x876 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread1" daemon prio=10 tid=0x0000000000d89000 nid=0x875 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"CompilerThread0" daemon prio=10 tid=0x0000000000d87000 nid=0x874 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x0000000000d85800 nid=0x873 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x0000000000d60800 nid=0x870 in Object.wait() [0x00000000416d2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd54b0550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00007f9fd54b0550> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x0000000000d5f000 nid=0x86f in Object.wait() [0x000000004049a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f9fd54b0630> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00007f9fd54b0630> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x0000000000d58000 nid=0x86e runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000000d91800 nid=0x877 waiting on condition
JNI global references: 1608
ConnectionManagerImpl is a singleton, it's guaranteed that there is no explicit synchronization on it via synchronized (conManager) {...} within the hand-written code. Here is it's source:
public class ConnectionManagerImpl implements ConnectionManager {
private int maxCons = 30;
private int lastGrantedConnectionId = 0;
private final List <Integer> allConnections = new ArrayList <Integer>();
#Override
public synchronized int getConnectionId(){
if (allConnections.size() >= maxCons) return -1;
if (!InternetChecker.waitForInternet()) { //unsynch
return -1;
}
int newId = ++lastGrantedConnectionId;
allConnections.add(newId);
return newId;
}
#Override
public synchronized void releaseConnectionId(int connectionId){
if (!allConnections.remove(Integer.valueOf(connectionId))) {
LogUtils.logError(new InternalException("Finished thread never was registred: " + "networkId = " + connectionId + ", active network IDs: " + allConnections));
}
}
}
VPS configuration: CentOS 5 x64, OpenJDK 64-Bit 19.0-b09
App runned with
-XX:HeapDumpPath=path_here
-XX:+HeapDumpOnOutOfMemoryError
-XX:-PrintClassHistogram
-XX:-UseCompressedOops
I removed a InternetChecker.waitForInternet() call like Udi Cohen proposed and here is what I got:
Exception in thread "queue_1" java.lang.IllegalMonitorStateException
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
Exception in thread "queue_2" java.lang.IllegalMonitorStateException
at org.seosrv.manager.ConnectionManagerImpl.getConnectionId(ConnectionManagerImpl.java:20)
at org.seosrv.utility.ConnectionUtils.getConnectionId(ConnectionUtils.java:15)
at org.seosrv.engines.search.SearchEngineContainer.startQueueProcessing(SearchEngineContainer.java:134)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.act(SearchEngineManager.java:202)
at org.seosrv.engines.search.SearchEngineManager$EngineQueueXable.run(SearchEngineManager.java:189)
at java.lang.Thread.run(Thread.java:636)
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (synchronizer.cpp:1954), pid=15269, tid=1104439616
# guarantee(mid->header()->is_neutral()) failed: invariant
#
# JRE version: 6.0_20-b20
# Java VM: OpenJDK 64-Bit Server VM (19.0-b09 mixed mode linux-amd64 )
# Derivative: IcedTea6 1.9.8
# Distribution: CentOS release 5.6 (Final), package rhel-1.22.1.9.8.el5_6-x86_64
# An error report file with more information is saved as:
# /hs_err_pid15269.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
I've already noticed the IllegalMonitorStateException under Oracle JDK 1.6.0_27 for c3p0 thread, but I thought that was a c3p0 problem. Now i've tested it better and got similiar error with IBM SDK.
It seems like this is a virtualization bug. Thank you for your efforts.
(Disclaimer: my company develops JProfiler)
Obviously, the jstack output is missing something in this case.
I would recommend to consult a different tool that gets this information from the JVMTI profiling interface. In JProfiler, you get a view like this:
Do you have any specific reason for using OpenJDK?
If not you can switch to another JDK to make sure this is not a bug in the JDK implementation.