i have 3 node cassandra 2.0.9 on production and face the issue when count or any specific query with clqsh is always get rpc_timeout the weird is this happen only on cassandra 1, other node with same configuration is fine
[cqlsh 4.1.1 | Cassandra 2.0.9 | CQL spec 3.1.1 | Thrift protocol 19.39.0]
Use HELP for help.
cqlsh> use xdata;
cqlsh:xdata> select count(*) from blobstore limit 100;
Request did not complete within rpc_timeout.
here the log from system.log when execute the query
INFO [MemoryMeter:1] 2022-08-03 10:40:10,910 Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 14.607407883739976 (just-counted was 14.607407407407408). calculation took 2ms for 54 cells
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,061 MessagingService.java (line 857) 1 REQUEST_RESPONSE messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,061 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,062 StatusLogger.java (line 70) MutationStage 0 0 8726 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,062 StatusLogger.java (line 70) RequestResponseStage 0 0 193404 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,062 StatusLogger.java (line 70) ReadRepairStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,062 StatusLogger.java (line 70) ReadStage 0 0 295316 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,062 StatusLogger.java (line 70) ReplicateOnWriteStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) MiscStage 0 0 2582 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) AntiEntropySessions 0 0 1028 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) HintedHandoff 0 0 112 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) FlushWriter 0 0 39 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) MemoryMeter 0 0 50 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) GossipStage 0 0 150208 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,063 StatusLogger.java (line 70) CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) InternalResponseStage 0 0 4112 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) CompactionExecutor 0 0 271 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) ValidationExecutor 0 0 2582 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) MigrationStage 0 0 2 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) commitlog_archiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) AntiEntropyStage 0 0 11332 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,064 StatusLogger.java (line 70) PendingRangeCalculator 0 0 3 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 70) MemtablePostFlusher 0 0 6062 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 79) CompactionManager 0 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 81) Commitlog n/a 0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 93) MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 103) Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 105) KeyCache 13808 104857600 all
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 111) RowCache 0 0 all
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 118) ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 121) system.compaction_history 9,3184
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 121) system.hints 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,065 StatusLogger.java (line 121) system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.paxos 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.peer_events 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.range_xfers 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.peers 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.local 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.sstable_activity 639,23664
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.schema_columns 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,066 StatusLogger.java (line 121) system.batchlog 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,067 StatusLogger.java (line 121) xdata.blobstore 127,54400
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,067 StatusLogger.java (line 121) xdata.document 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,067 StatusLogger.java (line 121) xdata.blobstoremeta 252,121960
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,067 StatusLogger.java (line 121) system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:40:21,067 StatusLogger.java (line 121) system_traces.events 0,0
INFO [ScheduledTasks:1] 2022-08-03 10:42:13,051 GCInspector.java (line 116) GC for ParNew: 261 ms for 1 collections, 2073077016 used; max is 8482586624
A CQL SELECT COUNT() is not a good thing to do in a distributed architecture. When you do an unbounded COUNT(), Cassandra has to do a full table scan to read every single record on every single node.
Counting rows in tables is very expensive as I've explained in detail in Why COUNT() is bad in Cassandra.
You should consider using other software such as Spark, Solr, or tools such as the DataStax Bulk Loader (DSBulk) which are able to count data in Cassandra tables efficiently since they use algorithms optimised for the task. Details in the DBA Stack Exchange post I linked above. Cheers!
resolve by add more node and cleanup some unused data
Related
i have 3 node cassandra cluster in kubernetes.
Deployed cassandra using bitnami/cassandra helm chart.
getting error based on more number of request after sometime later
WARN [GossipTasks:1] 2020-01-09 11:39:33,070 FailureDetector.java:278 - Not marking nodes down due to local pause of 8206335128 > 5000000000
WARN [GossipTasks:1] 2020-01-09 11:39:42,238 FailureDetector.java:278 - Not marking nodes down due to local pause of 6668041401 > 5000000000
WARN [GossipTasks:1] 2020-01-09 11:40:03,341 FailureDetector.java:278 - Not marking nodes down due to local pause of 15041441083 > 5000000000
WARN [PERIODIC-COMMIT-LOG-SYNCER] 2020-01-09 11:41:55,606 NoSpamLogger.java:94 - Out of 1 commit log syncs over the past 0.00s with average duration of 11850.79ms, 1 have exceeded the configured commit interval by an average of 1850.79ms
WARN [GossipTasks:1] 2020-01-09 11:42:20,019 Gossiper.java:783 - Gossip stage has 1 pending tasks; skipping status check (no nodes will be marked down)
NFO [RequestResponseStage-1] 2020-01-09 11:45:36,329 Gossiper.java:1011 - InetAddress /100.96.7.7 is now UP
INFO [RequestResponseStage-1] 2020-01-09 11:45:36,330 Gossiper.java:1011 - InetAddress /100.96.7.7 is now UP
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,931 MessagingService.java:1236 - MUTATION messages were dropped in last 5000 ms: 0 internal and 45 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2874 ms
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,933 StatusLogger.java:47 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,949 StatusLogger.java:51 - MutationStage 0 0 226236 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,950 StatusLogger.java:51 - ViewMutationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,950 StatusLogger.java:51 - ReadStage 0 0 244468 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,951 StatusLogger.java:51 - RequestResponseStage 0 0 341270 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,952 StatusLogger.java:51 - ReadRepairStage 0 0 5395 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,953 StatusLogger.java:51 - CounterMutationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,958 StatusLogger.java:51 - MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,959 StatusLogger.java:51 - CompactionExecutor 0 0 686641 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,960 StatusLogger.java:51 - MemtableReclaimMemory 0 0 689 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,962 StatusLogger.java:51 - PendingRangeCalculator 0 0 9 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,964 StatusLogger.java:51 - GossipStage 0 0 3093860 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,966 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,970 StatusLogger.java:51 - HintsDispatcher 0 0 10 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,973 StatusLogger.java:51 - MigrationStage 0 0 6 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,973 StatusLogger.java:51 - MemtablePostFlush 0 0 717 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,974 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0 0 0 689 0 0
:
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,974 StatusLogger.java:51 - PerDiskMemtableFlushWriter_0 0 0 689 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,974 StatusLogger.java:51 - ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,975 StatusLogger.java:51 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,975 StatusLogger.java:51 - MemtableFlushWriter 0 0 689 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,976 StatusLogger.java:51 - InternalResponseStage 0 0 869 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,977 StatusLogger.java:51 - AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2020-01-09 11:45:55,978 StatusLogger.java:51 - CacheCleanupExecutor 0 0 0 0 0
INFO
INFO [Service Thread] 2020-01-09 12:11:49,292 GCInspector.java:284 - ParNew GC in 659ms. CMS Old Gen: 2056877512 -> 2057740336; Par Eden Space: 671088640 -> 0; Par Survivor Space: 2636992 -> 6187520
Tried to solved based on some of the reference issue but not given for kubernetes
Cassandra Error message: Not marking nodes down due to local pause. Why?
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 245904 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 696906 0 0
MutationStage 0 0 244820 0 0
MemtableReclaimMemory 0 0 697 0 0
PendingRangeCalculator 0 0 9 0 0
GossipStage 0 0 3138625 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 10 0 0
RequestResponseStage 0 0 364305 0 0
Native-Transport-Requests 0 0 11089339 0 241
ReadRepairStage 0 0 5395 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 6 0 0
MemtablePostFlush 0 0 725 0 0
PerDiskMemtableFlushWriter_0 0 0 697 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 697 0 0
InternalResponseStage 0 0 869 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 45
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
From the above "tpstats" metrics looks okay but we can see some mutation there so it indicates that you cluster is going overload. Some requests blocked there too. Commitlogs seem not accepting the write there. You should plan cluster expansion or start debugging why nodes are overloading.
Based on the log entries you posted above, the nodes are overloaded making them unresponsive. Mutations get dropped because the commitlog disks cannot keep up with the writes.
You will need to review the size of your cluster as you might need to add more nodes to increase capacity. Cheers!
Everyone
A few days ago I upgraded our 6 node EC2 cluster from cassandra 2.1.4 to 2.1.5.
Since then, all my nodes have "exploded" in their cpu usage - their at 100% cpu for much of the time and their load average is between 100-300 (!!!).
This did not start immediately after the upgrade. It started a few hours afterwards with one of the nodes, and slowly, more and more nodes began to exhibit the same behavior.
It seems to correlate with the compaction of our largest column family, and after the compaction is complete (~24 hours after it starts) it seems the node goes back to normal. It has only been 2 days or so, so I'm hoping it will not happen again, but I am still monitoring this.
Here are my questions
Is this a bug or an expected behavior?
If it is an expected behavior -
What is the explanation for this issue?
Is it documented somewhere that I have missed?
Should I do upgrades differently? Maybe 1 or 2 nodes at a time every 24 hours or so? What is the best practice?
If it is a bug -
Is it known?
Where should I report this? What data should I add?
Will downgrading back to 2.1.4 work?
Any feedback on this would be great
Thanks
Amir
Update:
This is the structure of the table in question.
CREATE TABLE tbl1 (
key text PRIMARY KEY,
created_at timestamp,
customer_id bigint,
device_id bigint,
event text,
fail_count bigint,
generation bigint,
gr_id text,
imei text,
raw_post text,
"timestamp" timestamp
) WITH COMPACT STORAGE
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = 'NONE';
The logs do not reveal much (at least to me). Here's a snippet of how the logs look
INFO [WRITE-/10.0.1.142] 2015-05-23 05:43:42,577 YamlConfigurationLoader.java:92 - Loading settings from file:/etc/cassandra/cassandra.yaml
INFO [WRITE-/10.0.1.142] 2015-05-23 05:43:42,580 YamlConfigurationLoader.java:135 - Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_rpc_address=10.0.2.145; cas_contention_timeout_in_ms=1000; client_encryption_options=; cluster_name=Gryphonet21 Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/data/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/data/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=16; partitioner=RandomPartitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/data/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=10.0.1.141,10.0.2.145,10.0.3.149}]}]; server_encryption_options=; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO [HANDSHAKE-/10.0.1.142] 2015-05-23 05:43:42,591 OutboundTcpConnection.java:494 - Cannot handshake version with /10.0.1.142
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,713 MessagingService.java:887 - 135 MUTATION messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,713 StatusLogger.java:51 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,714 StatusLogger.java:66 - CounterMutationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,714 StatusLogger.java:66 - ReadStage 5 1 5702809 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,715 StatusLogger.java:66 - RequestResponseStage 0 45 29528010 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,715 StatusLogger.java:66 - ReadRepairStage 0 0 997 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,715 StatusLogger.java:66 - MutationStage 0 31 43404309 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,716 StatusLogger.java:66 - GossipStage 0 0 569931 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,716 StatusLogger.java:66 - AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,716 StatusLogger.java:66 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,717 StatusLogger.java:66 - MigrationStage 0 0 9 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,829 StatusLogger.java:66 - ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,830 StatusLogger.java:66 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,830 StatusLogger.java:66 - MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,831 StatusLogger.java:66 - CommitLogArchiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,831 StatusLogger.java:66 - MemtableFlushWriter 1 1 1756 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,831 StatusLogger.java:66 - PendingRangeCalculator 0 0 11 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,832 StatusLogger.java:66 - MemtableReclaimMemory 0 0 1756 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,832 StatusLogger.java:66 - MemtablePostFlush 1 2 3819 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,832 StatusLogger.java:66 - CompactionExecutor 2 32 742 0 0
INFO [ScheduledTasks:1] 2015-05-23 05:43:42,833 StatusLogger.java:66 - InternalResponseStage 0 0 0 0 0
INFO [HANDSHAKE-/10.0.1.142] 2015-05-23 05:43:45,086 OutboundTcpConnection.java:485 - Handshaking version with /10.0.1.142
UPDATE:
The issue persists. I thought after one compaction on each node finishes the node is back to normal, but it isn't. After a few hours, CPU jumps to 100% and load average in the areas of 100-300.
I am downgrading back to 2.1.4.
UPDATE:
Used phact's dumpThreads script to get stack traces. Also, tried using jvmtop, but it just seemed to hang.
The output is too big to paste here, but you can find it at http://downloads.gryphonet.com/cassandra/.
Username: cassandra
Password: cassandra
try using jvmtop to see what is the cassandra process doing. it has two modes, one to see current running threads and the other to show distribution of cpu per class procedure (--profile), paste both outputs here
Answering my own question -
We are using a very specific thrift API - describe_splits_ex, and that seems to cause the issue.
It is obvious when looking at all the stack traces of all the different threads while the cpu usage goes to 100%.
For us, it was easy to fix, because we use this api as an optimization, not a must, so we just stopped using it, and the issue went away.
However, this api is also used by the cassandra-hadoop connector (at least it used to in earlier versions), so I would test before upgrading to 2.1.5 if you are using the connector.
Not sure what change in 2.1.5 caused the issue, but I know it did not happen in 2.1.4 and happened consistently in 2.1.5.
I am having some problem with cassandra 2.0.8 that after I have restarted it, it cannot join the cluster with the "stream failed" exception, then it stops running. Until that time, node stays in JOINING state.
Do you have any idea why stream fails in cassandra?
Logs:
ERROR [main] 2015-01-17 17:52:10,063 CassandraDaemon.java (line 513) Exception encountered during startup
java.lang.RuntimeException: Error during boostrap: Stream failed
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1035)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:797)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:614)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:504)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:348)
at org.apache.cassandra.streaming.StreamSession.convict(StreamSession.java:617)
at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:237)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:643)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:64)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:170)
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,063 StatusLogger.java (line 70) GossipStage 0 0 537 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,081 StatusLogger.java (line 70) CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,082 StatusLogger.java (line 70) MigrationStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,083 StatusLogger.java (line 70) MemoryMeter 1 1 28 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,084 StatusLogger.java (line 70) ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,085 StatusLogger.java (line 70) FlushWriter 0 0 20 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,086 StatusLogger.java (line 70) InternalResponseStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,087 StatusLogger.java (line 70) AntiEntropyStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,087 StatusLogger.java (line 70) MemtablePostFlusher 0 0 49 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,088 StatusLogger.java (line 70) MiscStage 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,095 StatusLogger.java (line 70) PendingRangeCalculator 0 0 3 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,098 StatusLogger.java (line 70) commitlog_archiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,099 StatusLogger.java (line 70) CompactionExecutor 10 10 142 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,100 StatusLogger.java (line 70) HintedHandoff 0 0 2 0 0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,104 StatusLogger.java (line 79) CompactionManager 10 19
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,105 StatusLogger.java (line 81) Commitlog n/a 2
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,106 StatusLogger.java (line 93) MessagingService n/a 0/1217
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,107 StatusLogger.java (line 103) Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,107 StatusLogger.java (line 105) KeyCache 373840 3221225472 all
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,108 StatusLogger.java (line 111) RowCache 0 3221225472 all
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,108 StatusLogger.java (line 118) ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,109 StatusLogger.java (line 121) system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,109 StatusLogger.java (line 121) system.local 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,109 StatusLogger.java (line 121) system.peers 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,110 StatusLogger.java (line 121) system.batchlog 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,110 StatusLogger.java (line 121) system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,110 StatusLogger.java (line 121) system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,111 StatusLogger.java (line 121) system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,111 StatusLogger.java (line 121) system.compaction_history 20,6271
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,112 StatusLogger.java (line 121) system.paxos 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,112 StatusLogger.java (line 121) system.schema_columns 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,112 StatusLogger.java (line 121) system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,113 StatusLogger.java (line 121) system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,113 StatusLogger.java (line 121) system.range_xfers 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,113 StatusLogger.java (line 121) system.peer_events 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,114 StatusLogger.java (line 121) system.hints 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,114 StatusLogger.java (line 121) system.sstable_activity 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,115 StatusLogger.java (line 121) monitor.metric_data_30min 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,115 StatusLogger.java (line 121) monitor.metric_data_24hour 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,115 StatusLogger.java (line 121) monitor.metric_data_2hour 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,116 StatusLogger.java (line 121) monitor.metric_test 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,116 StatusLogger.java (line 121) monitor.time_data_5min 82442,15530317
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,116 StatusLogger.java (line 121) monitor.time_data_30min 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,117 StatusLogger.java (line 121) monitor.metric_data_5min 155046,91535821
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,117 StatusLogger.java (line 121) system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2015-01-17 17:52:10,118 StatusLogger.java (line 121) system_traces.events 0,0
INFO [StorageServiceShutdownHook] 2015-01-17 17:52:10,167 Gossiper.java (line 1271) Announcing shutdown
INFO [RequestResponseStage:17] 2015-01-17 17:52:10,194 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:18] 2015-01-17 17:52:10,196 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:19] 2015-01-17 17:52:10,199 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:20] 2015-01-17 17:52:10,205 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:21] 2015-01-17 17:52:10,215 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:22] 2015-01-17 17:52:10,219 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:23] 2015-01-17 17:52:10,224 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:24] 2015-01-17 17:52:10,238 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:25] 2015-01-17 17:52:10,244 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:26] 2015-01-17 17:52:10,249 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:28] 2015-01-17 17:52:10,258 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:27] 2015-01-17 17:52:10,258 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:29] 2015-01-17 17:52:10,263 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:30] 2015-01-17 17:52:10,268 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:31] 2015-01-17 17:52:10,275 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:32] 2015-01-17 17:52:10,283 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [RequestResponseStage:17] 2015-01-17 17:52:10,283 Gossiper.java (line 869) InetAddress /x.y.z.t is now UP
INFO [ScheduledTasks:1] 2015-01-17 17:52:12,125 GCInspector.java (line 116) GC for ParNew: 459 ms for 1 collections, 7433903048 used; max is 12759072768
INFO [StorageServiceShutdownHook] 2015-01-17 17:52:12,170 MessagingService.java (line 683) Waiting for messaging service to quiesce
INFO [ACCEPT-cassandra/x.y.z.t] 2015-01-17 17:52:12,175 MessagingService.java (line 923) MessagingService has terminated the accept() thread
It was a little aggressive in 2.0.8 on timeouts for convict. So if the networks a little flakey it can fail like this. There was an update in 2.0.9 that helps: CASSANDRA-7063.
Can work around it possibly by increasing phi_convict in your cassandra.yaml temporarily.
We are experimenting with using Cassandra as our datastore and have run into an issue with nodes failing due to running out of heap space. We're running Datastax Community Edition with Cassandra 2.0.1 on a 9 node cluster running Ubuntu server 13.04 with 16 GB RAM per node. During a data migration, two of our nodes went down unexpectedly due to running out of heap space. The stack traces in the logs were fairly nondescript and varied. Here's an example of one of them:
ERROR [MutationStage:21] 2013-11-01 07:08:39,656 CassandraDaemon.java (line 185) Exception in thread Thread[MutationStage:21,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at org.apache.cassandra.utils.SlabAllocator$Region.init(SlabAllocator.java:178)
at org.apache.cassandra.utils.SlabAllocator.getRegion(SlabAllocator.java:101)
at org.apache.cassandra.utils.SlabAllocator.allocate(SlabAllocator.java:70)
at org.apache.cassandra.utils.Allocator.clone(Allocator.java:30)
at org.apache.cassandra.db.ColumnFamilyStore.internOrCopy(ColumnFamilyStore.java:2220)
at org.apache.cassandra.db.Column.localCopy(Column.java:277)
at org.apache.cassandra.db.Memtable$1.apply(Memtable.java:107)
at org.apache.cassandra.db.Memtable$1.apply(Memtable.java:104)
at org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196)
at org.apache.cassandra.db.Memtable.put(Memtable.java:160)
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:201)
at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Prior to this there are AssertionErrors like this:
ERROR [FlushWriter:6176] 2013-11-01 06:55:48,825 CassandraDaemon.java (line 185) Exception in thread Thread[FlushWriter:6176,5,main]
java.lang.AssertionError
at org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:198)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:186)
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:358)
at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:317)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
as well as a slew of garbage collection status messages like so:
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,923 GCInspector.java (line 116) GC for ConcurrentMarkSweep: 5935 ms for 1 collections, 2963961136 used; max is 3902799872
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,924 StatusLogger.java (line 55) Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,925 StatusLogger.java (line 70) ReadStage 0 3 58646672 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,925 StatusLogger.java (line 70) RequestResponseStage 0 1 22614351 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,925 StatusLogger.java (line 70) ReadRepairStage 0 0 76371 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,926 StatusLogger.java (line 70) MutationStage 7 260 709366463 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,926 StatusLogger.java (line 70) ReplicateOnWriteStage 0 0 104455 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,926 StatusLogger.java (line 70) GossipStage 0 1 3695467 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,953 StatusLogger.java (line 70) AntiEntropyStage 0 0 404 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,954 StatusLogger.java (line 70) MigrationStage 0 0 1178 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,954 StatusLogger.java (line 70) MemtablePostFlusher 1 39 43229 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,955 StatusLogger.java (line 70) MemoryMeter 0 0 668 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,955 StatusLogger.java (line 70) FlushWriter 0 0 23228 0 82
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,955 StatusLogger.java (line 70) MiscStage 0 0 196 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,956 StatusLogger.java (line 70) commitlog_archiver 0 0 0 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,956 StatusLogger.java (line 70) InternalResponseStage 0 0 276 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,956 StatusLogger.java (line 70) HintedHandoff 0 0 13 0 0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,956 StatusLogger.java (line 79) CompactionManager 3 11
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,957 StatusLogger.java (line 81) Commitlog n/a 261
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,957 StatusLogger.java (line 93) MessagingService n/a 1,0
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,957 StatusLogger.java (line 103) Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,957 StatusLogger.java (line 105) KeyCache 41783700 104857600 all
INFO [ScheduledTasks:1] 2013-11-01 06:59:14,975 StatusLogger.java (line 111) RowCache 0 0 all
...
Considering this happened after only 4 hours of data ingest, we're wondering why this happened and what we can do prevent it from happening again. Thanks in advance.
Been running Cassandra on Ubuntu in production for some years and it's really sensitive to the RAM settings. In general, don't store more than 1TB of data per node, and avoid running with less than 8GB of max heap.
See the "MAX_HEAP_SIZE" setting in /etc/cassandra/cassandra-env.sh.
When you initially import data, it goes into RAM, then gets compacted. It's usually a good idea to set the max heap higher for the initial start-up, then restart with a smaller heap once the cluster is fully up.
INFO 19:07:42,273 GC for ParNew: 2182 ms, 27013384 reclaimed leaving 215461536
used; max is 1171062784
INFO 19:07:44,382 Pool Name Active Pending
INFO 19:07:44,960 ReadStage 0 0
INFO 19:07:44,976 RequestResponseStage 0 0
INFO 19:07:44,976 ReadRepairStage 0 0
INFO 19:07:45,007 MutationStage 0 0
INFO 19:07:45,007 GossipStage 0 0
INFO 19:07:45,007 AntiEntropyStage 0 0
INFO 19:07:45,007 MigrationStage 0 0
INFO 19:07:45,007 StreamStage 0 0
INFO 19:07:45,022 MemtablePostFlusher 0 0
INFO 19:07:45,022 FlushWriter 0 0
INFO 19:07:45,022 MiscStage 0 0
INFO 19:07:45,022 FlushSorter 0 0
INFO 19:07:45,038 InternalResponseStage 0 0
INFO 19:07:45,038 HintedHandoff 0 0
INFO 19:07:45,085 CompactionManager n/a 0
INFO 19:07:45,101 MessagingService n/a 0,0
INFO 19:07:45,116 ColumnFamily Memtable ops,data Row cache siz
/cap Key cache size/cap
INFO 19:07:45,288 system.LocationInfo 0,0
0/0 1/1
INFO 19:07:45,304 system.HintsColumnFamily 0,0
0/0 0/1
INFO 19:07:45,319 system.Migrations 0,0
0/0 0/1
INFO 19:07:45,319 system.Schema 0,0
0/0 0/1
INFO 19:07:45,319 system.IndexInfo 0,0
0/0 0/1
After this my installation process does not proceed. It generally hangs showing:
Listening for thrift clients....
Thats exactly what you want to see if your cassandra node/cluster is up and running.
Schildmeijer is correct - this is a normal log output from running Cassandra after a successful installation.
If you are unsure, then try running the Cassandra CLI (see http://wiki.apache.org/cassandra/CassandraCli) and execute some commands to check that the server node responds.
Cassandra runs as a server process - you don't interact directly with it, only via the CLI or another client tool.