I am seeing below exception in my cassandra logs(/var/log/cassandra/system.log)
INFO [ScheduledTasks:1] 2014-02-13 13:13:57,641 GCInspector.java (line 119) GC for ParNew: 273 ms for 1 collections, 2319121816 used; max is 445
6448000
INFO [ScheduledTasks:1] 2014-02-13 13:14:02,695 GCInspector.java (line 119) GC for ParNew: 214 ms for 1 collections, 2315368976 used; max is 445
6448000
INFO [OptionalTasks:1] 2014-02-13 13:14:08,093 MeteredFlusher.java (line 64) flushing high-traffic column family CFS(Keyspace='comsdb', ColumnFa
mily='product_update') (estimated 213624220 bytes)
INFO [OptionalTasks:1] 2014-02-13 13:14:08,093 ColumnFamilyStore.java (line 626) Enqueuing flush of Memtable-product_update#1067619242(31239028/
213625108 serialized/live bytes, 222393 ops)
INFO [FlushWriter:94] 2014-02-13 13:14:08,127 Memtable.java (line 400) Writing Memtable-product_update#1067619242(31239028/213625108 serialized/
live bytes, 222393 ops)
INFO [ScheduledTasks:1] 2014-02-13 13:14:08,696 GCInspector.java (line 119) GC for ParNew: 214 ms for 1 collections, 2480175160 used; max is 445
6448000
INFO [FlushWriter:94] 2014-02-13 13:14:10,836 Memtable.java (line 438) Completed flushing /cassandra1/data/comsdb/product_update/comsdb-product_
update-ic-416-Data.db (15707248 bytes) for commitlog position ReplayPosition(segmentId=1391568233618, position=13712751)
ERROR [Thrift:13] 2014-02-13 13:15:45,694 CustomTThreadPoolServer.java (line 213) Thrift error occurred during processing of message.
org.apache.thrift.TException: Negative length: -2147418111
at org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:388)
at org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:20304)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:21)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
ERROR [Thrift:103] 2014-02-13 13:21:25,719 CustomTThreadPoolServer.java (line 213) Thrift error occurred during processing of message.
org.apache.thrift.TException: Negative length: -2147418111
Below is my cassandra version and hector client version, which is being used currently
Cassandra-version: 1.2.11
Hector-client: 1.0-2
Any lead would be appreciated though we are planning to move cassandra 2.0 version with java-driver but it may take some time meanwhile need to find the root cause and resolve this issue.
Related
I need to pull data from oracle to Hive. My job is completed in 24 hours.
I am using spark JDBC API to pull the data.How to tune this job?
Oracle table info:
No of blocks:54014592
Memory in MB:421989
DRIVER_MEMORY :25 gb
EXECUTOR_CORES :5
EXECUTOR_INSTANCES :25
EXECUTOR_MEMORY :20 gb
spark Parallel Degree:25
Table has 29 partitions and max partition size is 93 GB
Logs:It has many Garbage Collection
18/12/03 05:11:55 INFO scheduler.TaskSetManager: Finished task 16.0 in stage 4.0 (TID 144) in 1448053 ms on bdgtr004d02h1u.nam.nsroot.net (executor 18) (36/64)
1837.013: [Full GC (System.gc()) 549M->296M(25G), 0.5553646 secs]
18/12/03 05:22:11 INFO storage.BlockManagerInfo: Added rdd_10_44 in memory on bdgtr015d07h2u.nam.nsroot.net:36517 (size: 498.7 MB, free: 10.3 GB)
18/12/03 05:58:59 INFO scheduler.TaskSetManager: Finished task 38.0 in stage 4.0 (TID 166) in 4271907 ms on bdgtr007d17i2u.nam.nsroot.net (executor 5) (59/64)
18/12/03 06:16:17 INFO storage.BlockManagerInfo: Added rdd_10_22 in memory on bdgtr006d20i2u.nam.nsroot.net:34124 (size: 705.2 MB, free: 8.4 GB)
5437.013: [Full GC (System.gc()) 1121M->297M(25G), 0.6317014 secs]
18/12/03 06:17:00 INFO scheduler.TaskSetManager: Finished task 22.1 in stage 4.0 (TID 192) in 2686834 ms on bdgtr006d20i2u.nam.nsroot.net (executor 9) (60/64)
7237.013: [Full GC (System.gc()) 1112M->297M(25G), 0.7000144 secs]
18/12/03 07:02:15 INFO storage.BlockManagerInfo: Added rdd_10_63 in memory on bdgtr007d17i2u.nam.nsroot.net:43841 (size: 318.9 MB, free: 9.0 GB)
18/12/03 07:02:39 INFO scheduler.TaskSetManager: Finished task 63.0 in stage 4.0 (TID 191) in 8091801 ms on bdgtr007d17i2u.nam.nsroot.net (executor 5) (61/64)
9037.014: [Full GC (System.gc()) 1097M->297M(25G), 0.6828210 secs]
18/12/03 07:17:57 INFO storage.BlockManagerInfo: Added rdd_10_58 in memory on bdgtr002d16i2u.nam.nsroot.net:41262 (size: 247.2 MB, free: 9.6 GB)
18/12/03 07:18:17 INFO scheduler.TaskSetManager: Finished task 58.0 in stage 4.0 (TID 186) in 9030124 ms on bdgtr002d16i2u.nam.nsroot.net (executor 25) (62/64)
18/12/03 07:21:11 INFO storage.BlockManagerInfo: Added rdd_10_0 in memory on bdgtr001d01h1u.nam.nsroot.net:41190 (size: 515.8 MB, free: 10.0 GB)
18/12/03 07:21:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 128) in 9241836 ms on bdgtr001d01h1u.nam.nsroot.net (executor 8) (63/64)
10837.013: [Full GC (System.gc()) 1095M->297M(25G), 0.7272104 secs]
18/12/03 07:51:01 INFO storage.BlockManagerInfo: Added rdd_10_59 in memory on bdgtr009d08i2u.nam.nsroot.net:44716 (size: 287.4 MB, free: 9.4 GB)
I'm experiencing node crashes where system.logfile is showing bunch of 'ReadTimeoutException' hitting 500ms.
cassandra.yaml file has setting for [read_request_timeout_in_ms: 10000]
can you folks please share how i can address these timeout! Thanks in advance!
error stack:
ERROR [SharedPool-Worker-241] 2017-02-01 13:18:27,663 Message.java:611 - Unexpected exception during request; channel = [id: 0x5d8abf33, /172.18.30.62:47580 => /216.12.225.9:9042]
java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:497) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:306) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.ClientState.login(ClientState.java:269) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:79) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-2.2.8.jar:2.2.8]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.8.jar:2.2.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:110) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:147) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1441) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1365) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1282) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:176) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:505) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:493) ~[apache-cassandra-2.2.8.jar:2.2.8]
... 13 common frames omitted
INFO [ScheduledTasks:1] 2017-02-01 13:18:27,682 MessagingService.java:946 - READ messages were dropped in last 5000 ms: 149 for internal timeout and 0 for cross node timeout
INFO [Service Thread] 2017-02-01 13:18:27,693 StatusLogger.java:106 - enterprise.t_sf_venue_test 0,0
INFO [ScheduledTasks:1] 2017-02-01 13:18:27,699 MessagingService.java:946 - REQUEST_RESPONSE messages were dropped in last 5000 ms: 7 for internal timeout and 0 for cross node timeout
INFO [Service Thread] 2017-02-01 13:18:27,699 StatusLogger.java:106 - enterprise.alestnstats 0,0
INFO [ScheduledTasks:1] 2017-02-01 13:18:27,699 MessagingService.java:946 - RANGE_SLICE messages were dropped in last 5000 ms: 116 for internal timeout and 0 for cross node timeout
As you see in your logs, actually the failing query is not the one you are trying to execute.
the failing query is internal to cassandra:
"SELECT * FROM system_auth.roles;"
These internal cassandra queries(misc queries) does not use 'read_request_timeout_in_ms'. Instead, it uses 'request_timeout_in_ms'.
I am trying to install cassandra on amazon centOS6.0 for that i follow from Caassandra
After successful install i am able to start the cassandra service but when trying to check status it throws "cassandra dead but pid file exists" error.
Any help.
Cassandra LOG:
INFO 18:09:06,422 Logging initialized
INFO 18:09:06,785 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 18:09:07,753 Data files directories: [/var/lib/cassandra/data]
INFO 18:09:07,770 Commit log directory: /var/lib/cassandra/commitlog
INFO 18:09:07,771 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 18:09:07,771 disk_failure_policy is stop
INFO 18:09:07,772 commit_failure_policy is stop
INFO 18:09:07,799 Global memtable threshold is enabled at 72MB
INFO 18:09:08,232 Not using multi-threaded compaction
INFO 18:09:08,834 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 18:09:08,872 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 18:09:08,896 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_71
INFO 18:09:08,896 Heap size: 301727744/302776320
INFO 18:09:08,897 Code Cache Non-heap memory: init = 2555904(2496K) used = 642560(627K) committed = 2555904(2496K) max = 50331648(49152K)
INFO 18:09:08,897 Eden Space Heap memory: init = 61341696(59904K) used = 46198280(45115K) committed = 61341696(59904K) max = 61341696(59904K)
INFO 18:09:08,897 Survivor Space Heap memory: init = 7602176(7424K) used = 0(0K) committed = 7602176(7424K) max = 7602176(7424K)
INFO 18:09:08,901 CMS Old Gen Heap memory: init = 232783872(227328K) used = 0(0K) committed = 232783872(227328K) max = 233832448(228352K)
INFO 18:09:08,902 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 13255544(12944K) committed = 21757952(21248K) max = 85983232(83968K)
INFO 18:09:08,902 Classpath: /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.17.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.17.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.17.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
System LOG:
INFO [main] 2016-02-16 18:09:06,422 CassandraDaemon.java (line 141) Logging initialized
INFO [main] 2016-02-16 18:09:06,785 YamlConfigurationLoader.java (line 81) Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO [main] 2016-02-16 18:09:07,753 DatabaseDescriptor.java (line 143) Data files directories: [/var/lib/cassandra/data]
INFO [main] 2016-02-16 18:09:07,770 DatabaseDescriptor.java (line 144) Commit log directory: /var/lib/cassandra/commitlog
INFO [main] 2016-02-16 18:09:07,771 DatabaseDescriptor.java (line 184) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2016-02-16 18:09:07,771 DatabaseDescriptor.java (line 198) disk_failure_policy is stop
INFO [main] 2016-02-16 18:09:07,772 DatabaseDescriptor.java (line 199) commit_failure_policy is stop
INFO [main] 2016-02-16 18:09:07,799 DatabaseDescriptor.java (line 269) Global memtable threshold is enabled at 72MB
INFO [main] 2016-02-16 18:09:08,232 DatabaseDescriptor.java (line 410) Not using multi-threaded compaction
INFO [main] 2016-02-16 18:09:08,834 YamlConfigurationLoader.java (line 81) Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO [main] 2016-02-16 18:09:08,872 YamlConfigurationLoader.java (line 81) Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO [main] 2016-02-16 18:09:08,896 CassandraDaemon.java (line 217) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_71
INFO [main] 2016-02-16 18:09:08,896 CassandraDaemon.java (line 245) Heap size: 301727744/302776320
INFO [main] 2016-02-16 18:09:08,897 CassandraDaemon.java (line 247) Code Cache Non-heap memory: init = 2555904(2496K) used = 642560(627K) committed = 2555904(2496K) max = 50331648(49152K)
INFO [main] 2016-02-16 18:09:08,897 CassandraDaemon.java (line 247) Eden Space Heap memory: init = 61341696(59904K) used = 46198280(45115K) committed = 61341696(59904K) max = 61341696(59904K)
INFO [main] 2016-02-16 18:09:08,897 CassandraDaemon.java (line 247) Survivor Space Heap memory: init = 7602176(7424K) used = 0(0K) committed = 7602176(7424K) max = 7602176(7424K)
INFO [main] 2016-02-16 18:09:08,901 CassandraDaemon.java (line 247) CMS Old Gen Heap memory: init = 232783872(227328K) used = 0(0K) committed = 232783872(227328K) max = 233832448(228352K)
INFO [main] 2016-02-16 18:09:08,902 CassandraDaemon.java (line 247) CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 13255544(12944K) committed = 21757952(21248K) max = 85983232(83968K)
INFO [main] 2016-02-16 18:09:08,902 CassandraDaemon.java (line 248) Classpath: /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.17.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.17.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.17.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
When i type cassandra -f
[root#ip-10-147-234-46 ~]# cassandra -f
INFO 18:31:32,468 Logging initialized
INFO 18:31:32,828 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 18:31:33,727 Data files directories: [/var/lib/cassandra/data]
INFO 18:31:33,738 Commit log directory: /var/lib/cassandra/commitlog
INFO 18:31:33,739 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 18:31:33,739 disk_failure_policy is stop
INFO 18:31:33,739 commit_failure_policy is stop
INFO 18:31:33,761 Global memtable threshold is enabled at 72MB
INFO 18:31:34,150 Not using multi-threaded compaction
INFO 18:31:34,749 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 18:31:34,787 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO 18:31:34,809 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_71
INFO 18:31:34,810 Heap size: 301727744/302776320
INFO 18:31:34,811 Code Cache Non-heap memory: init = 2555904(2496K) used = 642752(627K) committed = 2555904(2496K) max = 50331648(49152K)
INFO 18:31:34,811 Eden Space Heap memory: init = 61341696(59904K) used = 46225792(45142K) committed = 61341696(59904K) max = 61341696(59904K)
INFO 18:31:34,811 Survivor Space Heap memory: init = 7602176(7424K) used = 0(0K) committed = 7602176(7424K) max = 7602176(7424K)
INFO 18:31:34,815 CMS Old Gen Heap memory: init = 232783872(227328K) used = 0(0K) committed = 232783872(227328K) max = 233832448(228352K)
INFO 18:31:34,816 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 13255312(12944K) committed = 21757952(21248K) max = 85983232(83968K)
INFO 18:31:34,816 Classpath: /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.17.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.17.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.17.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
Killed
I found the issue the day i asked this question.
The issue is on amazon instance that was configured for me was very small below the micro instance. So once i check the instance and update it to large instance issue goes away immediately. Any way thanks for other suggestion.
Hi may be this is a stupid question, but I did not find the answer via google.
So what I have:
java 1.7
cassandra 1.2.8 running in single node with -Xmx1G and -Xms1G without any changes to yaml file
I've created next test column family:
CREATE COLUMN FAMILY TEST_HUGE_SF
WITH comparator = UTF8Type
AND key_validation_class=UTF8Type;
Then I try to insert rows in this column family.
I use astyanax lib to access cassandra:
final long START = 1;
final long MAX_ROWS_COUNT = 1000000000; // 1 Billion
Keyspace keyspace = AstyanaxProvider.getAstyanaxContext().getClient();
ColumnFamily<String, String> cf = new ColumnFamily<>(
"TEST_HUGE_SF",
StringSerializer.get(),
StringSerializer.get());
MutationBatch mb = keyspace.prepareMutationBatch()
.withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
for (long i = START; i<MAX_ROWS_COUNT; i++) {
long t = i % 1000;
if (t == 0) {
System.out.println("pushed: " + i);
mb.execute();
Thread.sleep(1);
mb = keyspace.prepareMutationBatch()
.withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
}
ColumnListMutation<String> clm = mb.withRow(cf, String.format("row_%012d", i));
clm.putColumn("col1", i);
clm.putColumn("col2", t);
}
mb.execute();
So as you can see from code, I try to insert 1 Billion rows, each one contains two columns, each column contains simple long value.
After inserting ~ 122 million rows, - cassandra crashed with OutOfMemoryError.
In logs there is next:
INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,334 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941,]. 865 252 169 bytes to 901 723 715 (~104% of original) in 922 963ms = 0,931728MB/s. 26 753 257 total rows, 26 753 257 unique. Row merge counts were {1:26753257, 2:0, 3:0, 4:0, }
INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,337 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1571] 2014-08-08 08:31:46,167 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1571,1,main]
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
INFO [CompactionExecutor:1570] 2014-08-08 08:31:46,994 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-3213,]. 34 773 524 bytes to 35 375 883 (~101% of original) in 44 162ms = 0,763939MB/s. 1 151 482 total rows, 1 151 482 unique. Row merge counts were {1:1151482, 2:0, 3:0, 4:0, }
INFO [CompactionExecutor:1570] 2014-08-08 08:31:47,105 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1570] 2014-08-08 08:31:47,110 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1570,1,main]
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
As I see cassandra crashes during sstables compaction.
Does this mean that to handle more rows cassandra needs more heap space?
I expected that lack of heap space will only affect performance. Can someone describe, why my expectations are wrong?
Someone else noted this - 1GB heap is very small. With Cassandra 2.0, you could look into this tuning guide for further information:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html
Another consideration is how garbage collecting is being handled. In the cassandra log directory, there should be also GC logs indicating how often and how long the collections were. You can monitor them live using jvisualvm, if you want.
Linux x64, Cassandra 2, Sun Java 1.7. After some reads - a bit more intensive than usual - cassandra simply dies with nothing helpful in the logfile.
This is how it is started (I used ps aux | grep cassandra):
109 29705 14.1 24.5 102272904 12122668 ? SLl 10:38 0:42 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar:/usr/share/cassandra/apache-cassandra-2.0.6.jar:/usr/share/cassandra/apache-cassandra-thrift-2.0.6.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar: -XX:HeapDumpPath=/var/lib/cassandra/java_1400575116.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1400575116.log org.apache.cassandra.service.CassandraDaemon
This is the log (full!):
INFO [ScheduledTasks:1] 2014-05-19 17:01:09,489 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-compaction_history#1762312178(253/2530 serialized/live bytes, 9 ops)
INFO [FlushWriter:2] 2014-05-19 17:01:09,489 Memtable.java (line 331) Writing Memtable-compaction_history#1762312178(253/2530 serialized/live bytes, 9 ops)
INFO [ScheduledTasks:1] 2014-05-19 17:01:09,584 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-schema_keyspaces#141563727(113/1130 serialized/live bytes, 4 ops)
INFO [ScheduledTasks:1] 2014-05-19 17:01:09,613 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-schema_columnfamilies#1997162460(49256/492560 serialized/live bytes, 1151 ops)
INFO [ScheduledTasks:1] 2014-05-19 17:01:09,647 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-schema_columns#1140062610(66556/665560 serialized/live bytes, 1863 ops)
INFO [FlushWriter:2] 2014-05-19 17:01:09,843 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/compaction_history/system-compaction_history-jb-415-Data.db (238 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
INFO [FlushWriter:2] 2014-05-19 17:01:09,845 Memtable.java (line 331) Writing Memtable-schema_keyspaces#141563727(113/1130 serialized/live bytes, 4 ops)
INFO [FlushWriter:2] 2014-05-19 17:01:10,083 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-108-Data.db (135 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
INFO [FlushWriter:2] 2014-05-19 17:01:10,084 Memtable.java (line 331) Writing Memtable-schema_columnfamilies#1997162460(49256/492560 serialized/live bytes, 1151 ops)
INFO [CompactionExecutor:23] 2014-05-19 17:01:10,084 CompactionTask.java (line 115) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-105-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-108-Data.db')]
INFO [CompactionExecutor:23] 2014-05-19 17:01:10,618 CompactionTask.java (line 275) Compacted 4 sstables to [/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-109,]. 793 bytes to 334 (~42% of original) in 531ms = 0,000600MB/s. 7 total partitions merged to 4. Partition merge counts were {1:2, 2:1, 3:1, }
INFO [FlushWriter:2] 2014-05-19 17:01:10,631 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-107-Data.db (10262 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
INFO [FlushWriter:2] 2014-05-19 17:01:10,632 Memtable.java (line 331) Writing Memtable-schema_columns#1140062610(66556/665560 serialized/live bytes, 1863 ops)
INFO [FlushWriter:2] 2014-05-19 17:01:10,926 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-96-Data.db (14011 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
INFO [CompactionExecutor:17] 2014-05-19 17:01:10,927 CompactionTask.java (line 115) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-93-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-95-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-94-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-96-Data.db')]
INFO [CompactionExecutor:17] 2014-05-19 17:01:11,267 CompactionTask.java (line 275) Compacted 4 sstables to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-97,]. 23 264 bytes to 14 011 (~60% of original) in 337ms = 0,039650MB/s. 10 total partitions merged to 4. Partition merge counts were {2:3, 4:1, }
INFO [ScheduledTasks:1] 2014-05-19 18:01:09,489 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-compaction_history#683113558(661/6610 serialized/live bytes, 21 ops)
INFO [FlushWriter:3] 2014-05-19 18:01:09,490 Memtable.java (line 331) Writing Memtable-compaction_history#683113558(661/6610 serialized/live bytes, 21 ops)
INFO [FlushWriter:3] 2014-05-19 18:01:09,704 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/compaction_history/system-compaction_history-jb-416-Data.db (372 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=227709)
INFO [CompactionExecutor:31] 2014-05-19 20:01:10,471 AutoSavingCache.java (line 250) Saved KeyCache (63 items) in 1192 ms
INFO [CompactionExecutor:32] 2014-05-20 00:01:10,040 AutoSavingCache.java (line 250) Saved KeyCache (63 items) in 761 ms
INFO [CompactionExecutor:33] 2014-05-20 04:01:10,280 AutoSavingCache.java (line 250) Saved KeyCache (163 items) in 1000 ms
INFO [CompactionExecutor:34] 2014-05-20 08:01:10,704 AutoSavingCache.java (line 250) Saved KeyCache (220 items) in 1424 ms
INFO [StorageServiceShutdownHook] 2014-05-20 10:36:12,376 ThriftServer.java (line 141) Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2014-05-20 10:36:12,389 Server.java (line 181) Stop listening for CQL clients
INFO [StorageServiceShutdownHook] 2014-05-20 10:36:12,390 Gossiper.java (line 1251) Announcing shutdown
INFO [StorageServiceShutdownHook] 2014-05-20 10:36:14,391 MessagingService.java (line 667) Waiting for messaging service to quiesce
INFO [ACCEPT-localhost/127.0.0.1] 2014-05-20 10:36:14,391 MessagingService.java (line 907) MessagingService has terminated the accept() thread
As you see, it shows "Gossiper.java (line 1251) Announcing shutdown" - but why?! This is really annoying and since it happens in production, this is business-critical =(
this could happen because of an out of memory error. trying running dmesg and see actually why it is killed.
I've had this happen to me, but this with an image deployed onto a VMWare Player. It turned out that it was because the SSD drive on which the Player was hosted on ran out of disk space. There were no related / helpful warnings in the log files in my case, either. Everything was fine after I cleared up some space.
I added auto_bootstrap:false in my cassandra.yaml file and it works for me after restarting Cassandra.