cassandra 4.0.0 driver - PreparedStatement uses 100% of one CPU node? - cassandra

Question

With cassandra driver 4.0.0 (com.datastax.oss / java-driver-core / 4.0.0), a PreparedStatement (just by existing) uses 100% of one CPU node, even when the application is idle:
import com.datastax.oss.driver.api.core.CqlSession;
import java.net.InetSocketAddress;
public class Demo_4_0_0 {
public static void main(String[] args) throws Exception {
CqlSession session = CqlSession.builder().addContactPoint(new InetSocketAddress("localhost", 9042)).withLocalDatacenter("datacenter1").build();
System.out.println("before preparing select");
Thread.sleep(15000);
session.prepare("SELECT value FROM demo.demo WHERE partition = 0;");
System.out.println("after preparing select");
Thread.sleep(15000);
session.close();
}
}
The same PreparedStatement with cassandra driver 3.7.1 (com.datastax.cassandra / cassandra-driver-core / 3.7.1) behaves fine - no CPU load when the application is idle:
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
public class Demo_3_7_1 {
public static void main(String[] args) throws Exception {
Session session = Cluster.builder().addContactPoints("127.0.0.1").build().connect();
System.out.println("before preparing select");
Thread.sleep(15000);
session.prepare("SELECT value FROM demo.demo WHERE partition = 0;");
System.out.println("after preparing select");
Thread.sleep(15000);
session.close();
}
}
For the examples to work, in cassandra first execute:
CREATE KEYSPACE demo WITH REPLICATION = {'class':'SimpleStrategy', 'replication_factor':1};
CREATE TABLE demo.demo (partition INT PRIMARY KEY, value INT);
My environment: Cassandra 3.11.2 (as docker container), jdk1.8.0_111 x64, Windows 10, 8 CPUs.
Any ideas?
Additional information: The thread dump before and after preparing the statement looks identical with one exception, that the "s0-timer-0" thread has appeared:
"s0-timer-0" #12 prio=5 os_prio=0 tid=0x0000000021392000 nid=0xcd0 runnable [0x0000000024f1e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.sleep(Native Method)
at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:579)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:478)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
Here is the full thread dump after the statement has been prepared:
2019-04-06 10:50:54
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode):
"s0-timer-0" #12 prio=5 os_prio=0 tid=0x0000000021392000 nid=0xcd0 runnable [0x0000000024f1e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.sleep(Native Method)
at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:579)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:478)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"JMX server connection timeout 20" #20 daemon prio=5 os_prio=0 tid=0x000000001ff32000 nid=0xb90 in Object.wait() [0x0000000024a1f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076b779218> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
- locked <0x000000076b779218> (a [I)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" #19 daemon prio=5 os_prio=0 tid=0x000000001ff31000 nid=0x13ac waiting on condition [0x000000002491f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076f8c4ca0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Connection(1)-10.0.75.1" #18 daemon prio=5 os_prio=0 tid=0x0000000020be7800 nid=0x27b0 runnable [0x000000002481d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <0x000000076b73dae0> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$180/1667804988.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- <0x000000076b553ee8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Accept-0" #17 daemon prio=5 os_prio=0 tid=0x0000000020b3c000 nid=0x448c runnable [0x000000002461f000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x000000076f400858> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"s0-io-1" #16 prio=5 os_prio=0 tid=0x0000000020b34000 nid=0x33f8 runnable [0x000000002310f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x000000076f413268> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000076f413280> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000076f4131e8> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:786)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:434)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"s0-io-0" #15 prio=5 os_prio=0 tid=0x00000000203cd800 nid=0x368c runnable [0x0000000021aff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x000000076f40b900> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000076f40b918> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000076f40b880> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:786)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:434)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"s0-admin-1" #14 prio=5 os_prio=0 tid=0x00000000202c7000 nid=0x4aac waiting on condition [0x0000000020a2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076f418190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:251)
at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:52)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"s0-admin-0" #13 prio=5 os_prio=0 tid=0x00000000202b5800 nid=0x4158 waiting on condition [0x000000002072e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076f420180> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:238)
at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:52)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000000001eb95000 nid=0x2b30 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread3" #9 daemon prio=9 os_prio=2 tid=0x000000001eb11800 nid=0x1948 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x000000001eb05800 nid=0x56c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000001eafd800 nid=0x4a04 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000001eafa800 nid=0x1e0c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001eaf8800 nid=0x34c4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001cc1e000 nid=0x36f0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000000319e800 nid=0x4acc in Object.wait() [0x000000001efce000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076f438180> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x000000076f438180> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000003195000 nid=0x4118 in Object.wait() [0x000000001eace000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000076f430468> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference.tryHandlePending(Unknown Source)
- locked <0x000000076f430468> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00000000011be000 nid=0x3c90 waiting on condition [0x000000000301f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at bug.Demo_4_0_0.main(Demo_4_0_0.java:14)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=2 tid=0x000000001cc0a000 nid=0x17c8 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00000000030b7800 nid=0x4b9c runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00000000030b9800 nid=0x9c8 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00000000030bb000 nid=0x46e8 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00000000030bd000 nid=0x3234 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00000000030be800 nid=0x3160 runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00000000030bf800 nid=0xc20 runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00000000030c3800 nid=0x4298 runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00000000030c5000 nid=0x429c runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001ebaf000 nid=0x4a20 waiting on condition
JNI global references: 664

Answer 1

I looked into this a little bit today as we've had several reports of this happening now (JAVA-2264).
It appears this is caused by the fact that there is some special logic in netty's HashedWheelTimer timer code to round sleep duration to a factor of 10:
// Check if we run on windows, as if thats the case we will need
// to round the sleepTime as workaround for a bug that only affect
// the JVM if it runs on windows.
//
// See https://github.com/netty/netty/issues/356
if (PlatformDependent.isWindows()) {
sleepTimeMs = sleepTimeMs / 10 * 10;
}
Typically, this would be ok, however the java driver by default configures HashedWheelTimers tick duration to 1ms. As sleepTimeMs is a long, my guess is we're sleeping in a tight loop as 1/10*10 = 0.
We will find a solution for this in the driver, whether it be adjusting the default tick duration overall (maybe 1ms is too aggressive), or just for windows.
Until then you can work around this in one of two ways:
Pass in the system property -Ddatastax-java-driver.advanced.netty.timer.tick-duration="100 millseconds" to your application
Set datastax-java-driver.advanced.netty.timer.tick-duration=100 millseconds in application.conf