Why is cassandra not reading data from the local node? - cassandra

Question

I'm running a three-node cassandra cluster. I created a table with the following configuration:
cassandra#cqlsh> describe docstore;
CREATE KEYSPACE docstore WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE docstore.docs (
docid uuid PRIMARY KEY
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = 'NONE';
I traced a read command via CQL, and I see that a message is being sent from the coordinator node to another node:
Execute CQL3 query | 2022-12-21 22:06:53.438000 | 10.155.32.67 | 0 | 10.155.32.56
Parsing select * from docstore.docs where docId = f8e682f0-b4d0-9b83-bcdb-4a182453d9c9; [Native-Transport-Requests-1] | 2022-12-21 22:06:53.438000 | 10.155.32.67 | 97 | 10.155.32.56
Preparing statement [Native-Transport-Requests-1] | 2022-12-21 22:06:53.438000 | 10.155.32.67 | 182 | 10.155.32.56
Executing single-partition query on roles [ReadStage-3] | 2022-12-21 22:06:53.438000 | 10.155.32.67 | 411 | 10.155.32.56
Acquiring sstable references [ReadStage-3] | 2022-12-21 22:06:53.438000 | 10.155.32.67 | 425 | 10.155.32.56
Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-3] | 2022-12-21 22:06:53.438000 | 10.155.32.67 | 443 | 10.155.32.56
Key cache hit for sstable 1 [ReadStage-3] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 467 | 10.155.32.56
Merged data from memtables and 1 sstables [ReadStage-3] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 533 | 10.155.32.56
Read 1 live rows and 0 tombstone cells [ReadStage-3] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 554 | 10.155.32.56
Executing single-partition query on roles [ReadStage-2] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 763 | 10.155.32.56
Acquiring sstable references [ReadStage-2] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 776 | 10.155.32.56
Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-2] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 785 | 10.155.32.56
Key cache hit for sstable 1 [ReadStage-2] | 2022-12-21 22:06:53.438001 | 10.155.32.67 | 800 | 10.155.32.56
Merged data from memtables and 1 sstables [ReadStage-2] | 2022-12-21 22:06:53.439000 | 10.155.32.67 | 832 | 10.155.32.56
Read 1 live rows and 0 tombstone cells [ReadStage-2] | 2022-12-21 22:06:53.439000 | 10.155.32.67 | 847 | 10.155.32.56
reading data from /10.155.32.56 [Native-Transport-Requests-1] | 2022-12-21 22:06:53.439000 | 10.155.32.67 | 956 | 10.155.32.56
Sending READ message to cassandra-0.cassandra-headless.default.svc.cluster.local/10.155.32.56 [MessagingService-Outgoing-cassandra-0.cassandra-headless.default.svc.cluster.local/10.155.32.56-Small] | 2022-12-21 22:06:53.439000 | 10.155.32.67 | 1015 | 10.155.32.56
REQUEST_RESPONSE message received from /10.155.32.56 [MessagingService-Incoming-/10.155.32.56] | 2022-12-21 22:06:53.440000 | 10.155.32.67 | 1957 | 10.155.32.56
Processing response from /10.155.32.56 [RequestResponseStage-4] | 2022-12-21 22:06:53.440000 | 10.155.32.67 | 1987 | 10.155.32.56
READ message received from /10.155.32.67 [MessagingService-Incoming-/10.155.32.67] | 2022-12-21 22:06:53.451000 | 10.155.32.56 | 4 | 10.155.32.56
Executing single-partition query on docs [ReadStage-3] | 2022-12-21 22:06:53.451000 | 10.155.32.56 | 115 | 10.155.32.56
Acquiring sstable references [ReadStage-3] | 2022-12-21 22:06:53.451000 | 10.155.32.56 | 135 | 10.155.32.56
Merging memtable contents [ReadStage-3] | 2022-12-21 22:06:53.451000 | 10.155.32.56 | 150 | 10.155.32.56
Key cache hit for sstable 1 [ReadStage-3] | 2022-12-21 22:06:53.452000 | 10.155.32.56 | 258 | 10.155.32.56
Read 1 live rows and 0 tombstone cells [ReadStage-3] | 2022-12-21 22:06:53.452000 | 10.155.32.56 | 368 | 10.155.32.56
Enqueuing response to /10.155.32.67 [ReadStage-3] | 2022-12-21 22:06:53.452000 | 10.155.32.56 | 552 | 10.155.32.56
Sending REQUEST_RESPONSE message to /10.155.32.67 [MessagingService-Outgoing-/10.155.32.67-Small] | 2022-12-21 22:06:53.452000 | 10.155.32.56 | 664 | 10.155.32.56
Request complete | 2022-12-21 22:06:53.440087 | 10.155.32.67 | 2087 | 10.155.32.56
Why is a request being sent from the coordinator node to another node, considering the coordinator node contains the data that's being requested? I have a replication factor of 3 on a three-node cluster, so every node contains all the data.

Answer 1

What consistency level do you use for your read? If it's higher than ONE/LOCAL_ONE then the coordinator will request from another node to ensure consistency. I'd try running 'consistency LOCAL_ONE;' before the query with the same trace output to see if it changes.