i am using cassandra 3.0.0.8 on centos i got the below error in the logs after that metable flushes are not happening
ERROR [MemtableFlushWriter:7735] 2017-01-18 23:00:02,101 CassandraDaemon.java:201 - Exception in thread Thread[MemtableFlushWriter:7735,5,RMI Runtime]
java.lang.StackOverflowError: null
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) ~[apache-cassandra-3.0.8.jar:3.0.8]
This can be due to data corruption.
Try to run repair on the node.
If it doesn't help, try to run rebuild on the node.
Does it happens on 1 node or all nodes in the cluster?
Related
I am using ubuntu 14.04 with apache cassandra 3.7. I am trying to start it but get the following error message:
ERROR [main] 2016-07-15 15:22:10,627 CassandraDaemon.java:731 - Cannot start node if snitch's data center (dc1) differs from previous data center (datacenter1). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
I know I can set -Dcassandra.ignore_dc=true, BUT that is not a fix, its a band-aid and for development use only, this is suppose to be in production. I tried to clear out all the files and folders in /var/lib/cassandra, I MEAN every SINGLE FILE AND FOLDER, started apache cassandra again, AND STILL THE SAME ERROR MESSAGE... any other idea??
change in file:
/etc/cassandra/cassandra-rackdc.properties
entry from dc1 to datacenter1
on all nodes
and then do a rolling restart of nodes.
If have just switched to GossipingPropertyFileSnitch, start Cassandra with the option
-Dcassandra.ignore_dc=true
If it starts successfully, execute:
nodetool repair
nodetool cleanup
Afterwards, Cassandra should be able to start normally without the ignore option.
I faced the issue while upgrading my Apache cassandra from 3.11.1 to 3.11.4 .
cassandra.yaml
old_Config : endpoint_snitch: GossipingPropertyFileSnitch
New_Config: endpoint_snitch: SimpleSnitch
{changed it to GossipingPropertyFileSnitch}
cassandra-rackdc.properties
old_version_config: dc:Dc1 rack:Rack1
New_version_config: dc:dc rack:rack (changed this to Dc1 and Rack1)
this resolves my issue
I have a 6 node cluster running 1.1.5. I added a new node running the same version. The new node joined the cluster without any issues. But we are seeing the below error messages in the system.log
ERROR 10:15:53,455 Exception in thread Thread[ReadStage:320,5,main]
java.lang.AssertionError: Unknown keyspace
Any thoughts?
I have problems to get a existing Cassandra node to join the cluster again after reboot (on a new virtual machine instance).
I had a running Cassandra cluster with 4 nodes all in state "up and normal" according to nodetool status. The nodes are running on virtual machines in Azure. I changed the instance type of the virtual machine for 10.0.0.6, which returned in a reboot of this machine. The machine stayed on 10.0.0.6.
After the reboot I am unable to start Cassandra again. I am getting this exception:
INFO 22:39:07 Handshaking version with /10.0.0.4
INFO 22:39:07 Node /10.0.0.6 is now part of the cluster
INFO 22:39:07 Node /10.0.0.5 is now part of the cluster
INFO 22:39:07 Handshaking version with cassandraprd001/10.0.0.6
INFO 22:39:07 Node /10.0.0.9 is now part of the cluster
INFO 22:39:07 Handshaking version with /10.0.0.5
INFO 22:39:07 Node /10.0.0.4 is now part of the cluster
INFO 22:39:07 InetAddress /10.0.0.6 is now UP
INFO 22:39:07 Handshaking version with /10.0.0.9
INFO 22:39:07 InetAddress /10.0.0.4 is now UP
INFO 22:39:07 InetAddress /10.0.0.9 is now UP
INFO 22:39:07 InetAddress /10.0.0.5 is now UP
ERROR 22:39:08 Exception encountered during startup
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546) [apache-cassandra-2.1.0.jar:2.1.0]
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
Exception encountered during startup: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
INFO 22:39:08 Announcing shutdown
I am using Cassandra 2.1.0. I am not replaying a dead node - I am just trying to get the old node up and running again. According to nodetool status (on the other nodes) all nodes are "up and normal" except 10.0.0.6 which is "down and normal".
How do I get this node up and running again?
First, on another node, use
nodetool status
the results show you list of nodes in the cluster. Find your node with ip which fail to start, get its id and fill to command:
nodetool removenode <node_id>
then start cassandra.
Best,
Quick answer, if the node's ip is 10.200.10.200
add this
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.200.10.200"
to the end of your
cassandra-env.sh
Don't forget to remove it once your done.
You can look this blog, http://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html.
It works for me, this is a bug for Cassandra. If your node's host_id changed, but use old IP, will throw this exception.
If you use Cassandra 2.x.x, you should modify cassandra/conf/cassandra-env.sh.
Finally, don't forget to REMOVE modifications on cassandra-env.sh after the complete bootstrap!
I have a cluster running on datastax-cassandra 1.2.5, it works fine, because of vnodes adn leveled compaction strategy issue i tried promoting it to 1.2.6.
So upgrading involved -
1 - stopping all the nodes
2 - deleting 1.2.5 rpm
3 - installing 1.2.6 rpm
4 - fixing cassandra.yaml
5 - starting cassandra.
Problem Statement - The problem now is that all the nodes are up and running, but not in one cluster. They all are operating in their own cluster even though the seeds in yaml points to the original seed.
nodetool status also just shows the one node (the node on which we are on)
system log shows one error
ERROR [WRITE-/10.93.3.46] 2013-10-21 19:43:29,101 CassandraDaemon.java (line 192)
Exception in thread Thread[WRITE-/10.10.10.10,5,main]
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:351)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:143)
**** 10.10.10.10 is the seed ip
Any help on how to pass through it
Try to set the internode_compression to none. It will disable compression between nodes, which is failing because snappy cannot initialize
internode_compression: none
I am unable to connect to to the cassandra. By this I mean that when I try cassandra-CLI, or pycassaShell I get the following error:
Exception connecting to localhost/9160. Reason: Connection refused.
I have tried using a few non-defaul ports but to no joy.
I am running cassandra version 1.0.7
I have tried bounding the service, using restart, stop, start, and force-restart.
Run jps command under root user and kill CassandraDaemon if you will see it. After this you will start Cassandra again.
My problem was cassandra was not even starting. Solution was to start Cassandra!
I had the same issue in cassandra, After rename var/lib/cassandra/data to var/lib/cassandra/data1 then cassandra is working fine, If cassandra alloted maximum memory is full then it will throw connection refuse exception