I have a 6 node cluster running 1.1.5. I added a new node running the same version. The new node joined the cluster without any issues. But we are seeing the below error messages in the system.log
ERROR 10:15:53,455 Exception in thread Thread[ReadStage:320,5,main]
java.lang.AssertionError: Unknown keyspace
Any thoughts?
Related
We have a 15 node cassandra setup across 3 DC's.
Using cassandra 3.0.9
One of the nodes in one of our DCs has died with the below startup error
CassandraDaemon.java:709 - Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF 111111-111111111-11111111
What we have tried -
Bootstrap replacing the node -https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsReplaceNode.html
This works for a while and then the bootstrap process hangs with the SAME error in the logs -
CassandraDaemon.java:709 - Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF
Provision a new blank node,
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddNodeToCluster.html
and try to join this to the cluster (with the intention of removing the dead node after). It refuses to start, with the same error.
has anyone ever ran across this before?
i am using cassandra 3.0.0.8 on centos i got the below error in the logs after that metable flushes are not happening
ERROR [MemtableFlushWriter:7735] 2017-01-18 23:00:02,101 CassandraDaemon.java:201 - Exception in thread Thread[MemtableFlushWriter:7735,5,RMI Runtime]
java.lang.StackOverflowError: null
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) ~[apache-cassandra-3.0.8.jar:3.0.8]
This can be due to data corruption.
Try to run repair on the node.
If it doesn't help, try to run rebuild on the node.
Does it happens on 1 node or all nodes in the cluster?
I have problems to get a existing Cassandra node to join the cluster again after reboot (on a new virtual machine instance).
I had a running Cassandra cluster with 4 nodes all in state "up and normal" according to nodetool status. The nodes are running on virtual machines in Azure. I changed the instance type of the virtual machine for 10.0.0.6, which returned in a reboot of this machine. The machine stayed on 10.0.0.6.
After the reboot I am unable to start Cassandra again. I am getting this exception:
INFO 22:39:07 Handshaking version with /10.0.0.4
INFO 22:39:07 Node /10.0.0.6 is now part of the cluster
INFO 22:39:07 Node /10.0.0.5 is now part of the cluster
INFO 22:39:07 Handshaking version with cassandraprd001/10.0.0.6
INFO 22:39:07 Node /10.0.0.9 is now part of the cluster
INFO 22:39:07 Handshaking version with /10.0.0.5
INFO 22:39:07 Node /10.0.0.4 is now part of the cluster
INFO 22:39:07 InetAddress /10.0.0.6 is now UP
INFO 22:39:07 Handshaking version with /10.0.0.9
INFO 22:39:07 InetAddress /10.0.0.4 is now UP
INFO 22:39:07 InetAddress /10.0.0.9 is now UP
INFO 22:39:07 InetAddress /10.0.0.5 is now UP
ERROR 22:39:08 Exception encountered during startup
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546) [apache-cassandra-2.1.0.jar:2.1.0]
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
Exception encountered during startup: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
INFO 22:39:08 Announcing shutdown
I am using Cassandra 2.1.0. I am not replaying a dead node - I am just trying to get the old node up and running again. According to nodetool status (on the other nodes) all nodes are "up and normal" except 10.0.0.6 which is "down and normal".
How do I get this node up and running again?
First, on another node, use
nodetool status
the results show you list of nodes in the cluster. Find your node with ip which fail to start, get its id and fill to command:
nodetool removenode <node_id>
then start cassandra.
Best,
Quick answer, if the node's ip is 10.200.10.200
add this
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.200.10.200"
to the end of your
cassandra-env.sh
Don't forget to remove it once your done.
You can look this blog, http://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html.
It works for me, this is a bug for Cassandra. If your node's host_id changed, but use old IP, will throw this exception.
If you use Cassandra 2.x.x, you should modify cassandra/conf/cassandra-env.sh.
Finally, don't forget to REMOVE modifications on cassandra-env.sh after the complete bootstrap!
I have a cluster running on datastax-cassandra 1.2.5, it works fine, because of vnodes adn leveled compaction strategy issue i tried promoting it to 1.2.6.
So upgrading involved -
1 - stopping all the nodes
2 - deleting 1.2.5 rpm
3 - installing 1.2.6 rpm
4 - fixing cassandra.yaml
5 - starting cassandra.
Problem Statement - The problem now is that all the nodes are up and running, but not in one cluster. They all are operating in their own cluster even though the seeds in yaml points to the original seed.
nodetool status also just shows the one node (the node on which we are on)
system log shows one error
ERROR [WRITE-/10.93.3.46] 2013-10-21 19:43:29,101 CassandraDaemon.java (line 192)
Exception in thread Thread[WRITE-/10.10.10.10,5,main]
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:351)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:143)
**** 10.10.10.10 is the seed ip
Any help on how to pass through it
Try to set the internode_compression to none. It will disable compression between nodes, which is failing because snappy cannot initialize
internode_compression: none
I am unable to connect to to the cassandra. By this I mean that when I try cassandra-CLI, or pycassaShell I get the following error:
Exception connecting to localhost/9160. Reason: Connection refused.
I have tried using a few non-defaul ports but to no joy.
I am running cassandra version 1.0.7
I have tried bounding the service, using restart, stop, start, and force-restart.
Run jps command under root user and kill CassandraDaemon if you will see it. After this you will start Cassandra again.
My problem was cassandra was not even starting. Solution was to start Cassandra!
I had the same issue in cassandra, After rename var/lib/cassandra/data to var/lib/cassandra/data1 then cassandra is working fine, If cassandra alloted maximum memory is full then it will throw connection refuse exception