Consumres are waiting for run with Java ExecutorService - executorservice

Question

I have one producer vs. multiple consumers sample. The producer put an event to a blocking queue. Consumers take message from the queue and then insert them to DB. This works for many days.
However, it went down several times yesterday and the producer is blocked, when the traffic was very tough.
I checked the stack using jstack and I saw all EnterClassData-Consumer-Service threads are waiting for running while all ConsumeMessageThread are waiting for PUT
Sample stack is followed:
2019-01-29 20:46:39
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.65-b01 mixed mode):
"Attach Listener" #78 daemon prio=9 os_prio=0 tid=0x00007f97f0001000 nid=0x1301 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" #74 prio=5 os_prio=0 tid=0x00007f983c009800 nid=0x558f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"ConsumeMessageThread_2" #57 prio=5 os_prio=0 tid=0x00007f9800025000 nid=0x5640 waiting on condition [0x00007f97980ee000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
at xx.xx.monitor.config.EnterClassMQConsumer.onMessage(EnterClassMQConsumer.java:80)
at xx.xx.vkmq.client.spring.VKMQConsumerInitializer$$Lambda$34/1277678493.onMessage(Unknown Source)
at xx.xx.vkmq.client.consumer.VKMQConsumer.lambda$subscribe$0(VKMQConsumer.java:52)
at xx.xx.vkmq.client.consumer.VKMQConsumer$$Lambda$35/1742448147.consumeMessage(Unknown Source)
at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"ConsumeMessageThread_1" #55 prio=5 os_prio=0 tid=0x00007f9800023800 nid=0x563f waiting on condition [0x00007f97981ef000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f590> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
at xx.xx.monitor.config.EnterClassMQConsumer.onMessage(EnterClassMQConsumer.java:80)
at xx.xx.vkmq.client.spring.VKMQConsumerInitializer$$Lambda$34/1277678493.onMessage(Unknown Source)
at xx.xx.vkmq.client.consumer.VKMQConsumer.lambda$subscribe$0(VKMQConsumer.java:52)
at xx.xx.vkmq.client.consumer.VKMQConsumer$$Lambda$35/1742448147.consumeMessage(Unknown Source)
at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"PullMessageServiceScheduledThread" #56 prio=5 os_prio=0 tid=0x00007f97e0a1a000 nid=0x563e waiting on condition [0x00007f97982f0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006775910a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"NettyClientSelector_1" #41 prio=5 os_prio=0 tid=0x00007f983db61800 nid=0x5630 runnable [0x00007f97a63f2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006773a05b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000677590c20> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000067739dd90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:692)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:352)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
at java.lang.Thread.run(Thread.java:745)
"RebalanceService" #38 prio=5 os_prio=0 tid=0x00007f983e617000 nid=0x562f waiting on condition [0x00007f97a64f3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006773a0050> (a org.apache.rocketmq.common.CountDownLatch2$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at org.apache.rocketmq.common.CountDownLatch2.await(CountDownLatch2.java:114)
at org.apache.rocketmq.common.ServiceThread.waitForRunning(ServiceThread.java:116)
at org.apache.rocketmq.client.impl.consumer.RebalanceService.run(RebalanceService.java:40)
at java.lang.Thread.run(Thread.java:745)
"PullMessageService" #37 prio=5 os_prio=0 tid=0x00007f983e59e800 nid=0x562e waiting on condition [0x00007f97a65f4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000677591100> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.rocketmq.client.impl.consumer.PullMessageService.run(PullMessageService.java:88)
at java.lang.Thread.run(Thread.java:745)
"CleanExpireMsgScheduledThread_1" #40 prio=5 os_prio=0 tid=0x00007f983e580000 nid=0x562d waiting on condition [0x00007f97a66f5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000677592558> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"MQClientFactoryScheduledThread" #39 prio=5 os_prio=0 tid=0x00007f983e533000 nid=0x562c waiting on condition [0x00007f97a67f6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000677591208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"ClientHouseKeepingService" #36 daemon prio=5 os_prio=0 tid=0x00007f983dfc0000 nid=0x562b in Object.wait() [0x00007f97a6af7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000677592090> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"pool-6-thread-1" #34 prio=5 os_prio=0 tid=0x00007f983cb58000 nid=0x5621 waiting on condition [0x00007f97a6df8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006775e6a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-3-thread-2" #33 prio=5 os_prio=0 tid=0x00007f97a0172800 nid=0x5617 waiting on condition [0x00007f97a70f9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000675d44aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Abandoned connection cleanup thread" #32 daemon prio=5 os_prio=0 tid=0x00007f979c135800 nid=0x5613 in Object.wait() [0x00007f97a73fa000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006772ef050> (a java.lang.ref.ReferenceQueue$Lock)
at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Tomcat JDBC Pool Cleaner[1365202186:1548743608273]" #31 daemon prio=5 os_prio=0 tid=0x00007f979c102000 nid=0x5612 in Object.wait() [0x00007f97a74fb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000006772ef098> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"pool-3-thread-1" #30 prio=5 os_prio=0 tid=0x00007f97a045f800 nid=0x5611 waiting on condition [0x00007f97a75fc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000675d44aa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"EnterClassData-Consumer-Service-thread-5" #26 prio=5 os_prio=0 tid=0x00007f983c985800 nid=0x55e9 waiting on condition [0x00007f97f42cf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f2a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"EnterClassData-Consumer-Service-thread-4" #25 prio=5 os_prio=0 tid=0x00007f983c985000 nid=0x55e8 waiting on condition [0x00007f97f43d0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f2a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"EnterClassData-Consumer-Service-thread-3" #24 prio=5 os_prio=0 tid=0x00007f983cc18000 nid=0x55e7 waiting on condition [0x00007f97f44d1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f2a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"EnterClassData-Consumer-Service-thread-2" #23 prio=5 os_prio=0 tid=0x00007f983cc17800 nid=0x55e6 waiting on condition [0x00007f97f45d2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f2a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"EnterClassData-Consumer-Service-thread-1" #22 prio=5 os_prio=0 tid=0x00007f983d031800 nid=0x55e5 waiting on condition [0x00007f97f46d3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067675f2a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"container-0" #21 prio=5 os_prio=0 tid=0x00007f983d70c800 nid=0x55db waiting on condition [0x00007f97f4bd4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer$1.run(TomcatEmbeddedServletContainer.java:177)
"ContainerBackgroundProcessor[StandardEngine[Tomcat]]" #20 daemon prio=5 os_prio=0 tid=0x00007f983cdeb800 nid=0x55da waiting on condition [0x00007f97f4cd5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1355)
at java.lang.Thread.run(Thread.java:745)
"AsyncAppender-Worker-ASYNC_SLOW_SQL" #15 daemon prio=5 os_prio=0 tid=0x00007f983d876000 nid=0x55ae waiting on condition [0x00007f9810bd5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000674ce2a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:264)
"RMI TCP Accept-0" #13 daemon prio=5 os_prio=0 tid=0x00007f983c301000 nid=0x55a2 runnable [0x00007f9810fd7000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-9999" #12 daemon prio=5 os_prio=0 tid=0x00007f983c2f4000 nid=0x55a1 runnable [0x00007f98110d8000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
"RMI TCP Accept-0" #11 daemon prio=5 os_prio=0 tid=0x00007f983c2e0000 nid=0x55a0 runnable [0x00007f98111d9000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f983c1da800 nid=0x559f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f983c1cd000 nid=0x559e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f983c1cb000 nid=0x559d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f983c1c8800 nid=0x559c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f983c1c6800 nid=0x559b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f983c1c5000 nid=0x559a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f983c18d800 nid=0x5599 in Object.wait() [0x00007f981c158000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000674ce2b08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f983c18b000 nid=0x5598 in Object.wait() [0x00007f981c259000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x0000000674f0eee8> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=0 tid=0x00007f983c186000 nid=0x5597 runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f983c01a800 nid=0x5591 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f983c01c000 nid=0x5592 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f983c01e000 nid=0x5593 runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f983c01f800 nid=0x5594 runnable
"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f983c062800 nid=0x5596 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f983c303800 nid=0x55a3 waiting on condition
JNI global references: 462
I also post main part of my code here.
public class EnterClassMQConsumer implements MessageListenerAdapter {
public static final int NUM_WORKING_THREADS=
public static final int MAX_QUEUE_SIZE =
public static ExecutorService executorService = ExecutorsFactory.newNameThreadPool(NUM_WORKING_THREADS, MAX_QUEUE_SIZE, "EnterClassData-Consumer-Service");
private final static int QUEUE_SIZE=100;
private static BlockingQueue<EnterClassRoomStatus> queue = new LinkedBlockingQueue<EnterClassRoomStatus>(QUEUE_SIZE);
#PostConstruct
public void init() {
for(int i=0; i<NUM_WORKING_THREADS; i++ ) {
executorService.submit(new DBUpdateOperator());
}
}
#Override
public boolean onMessage(ReceivedMessage message) {
EnterClassRoomStatus enterClassRoomStatus = EnterClassRoomStatusFactory.getInstance().getRoomStatus();
enterClassRoomStatus.setMessage(message);
try {
queue.put(enterClassRoomStatus);
} catch (InterruptedException e) {
}
return true;
}
class DBUpdateOperator implements Runnable{
Set<EnterClassRoomStatus> pendSet = Sets.newLinkedHashSet();
#Override
public void run() {
while(true) {
EnterClassRoomStatus event = null;
try {
event = queue.poll(1, TimeUnit.SECONDS);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
processAndInsertDB(event);
}
}
}
and the ExecutorService is defined as
public static ThreadPoolExecutor newNameThreadPool(int maxSize, int queueSize, RejectedExecutionHandler rejectedExecutionHandler, String prefix) {
BlockingQueue<Runnable> workQueue = new LinkedBlockingDeque<Runnable>(queueSize);
ThreadFactory threadFactory = new NamedThreadFactory(prefix);
int maxPoolSize = maxSize > DEFAULT_MAX_POOL_SIZE ? maxSize : DEFAULT_MAX_POOL_SIZE;
return new ThreadPoolExecutor(DEFAULT_CORE_POOL_SIZE, maxPoolSize,
DEFAULT_KEEP_ALIVE_TIME, TimeUnit.MILLISECONDS, workQueue, threadFactory, rejectedExecutionHandler); //rejectedExecutionHandler is a reject handler that blocks thread until queue has available space.
}
What might be the reason for EnterClassData-Consumer-Service-thread waiting? Is it related to ExecutorService and how can I improve it?
Thanks in advance.

Answer 1

The reason the call to BlockingQueue#put is waiting is because the queue is full, i.e. it reaches its capacity (which seems to be QUEUE_SIZE in your case). This is the expected behaviour as per the javadoc:
Inserts the specified element into this queue, waiting if necessary for space to become available.
As a result, you will need to implement onMessage differently based on what you want to do when your system undergoes a traffic burst:
Shed requests in excess of queue capacity?
Increase queue size? [not recommended in most cases]
Also, note that you have two explicit queues in your system: one in EnterClassMQConsumer and one for the thread pool executor, resulting in queueing tasks up to QUEUE_SIZE + MAX_QUEUE_SIZE - is this what you really want?