I have an existing 3 node Cassandra cluster. Added 3 new nodes to the cluster. The nodes are still "bootstrapping", but I added the nodes 3 days ago. I'm really concerned about this process.
1) How long does bootstrapping typically take? We have about 40GB on each node.
2) All of the new nodes have died at least once during bootstrapping with no cause given in the logs. Are there any known issues around this?
Using Cassandra 2.0.6 on Ubuntu 12.04. Any help or advice would be greatly appreciated.
Usually Bootstrapping is process to communicate the brand new node to the existing nodes in a cluster. It is recommended to allow 2-3 minutes to complete the bootstrapping process.
Do the new nodes have the same cassandra configuration ? While adding a new node, from the console you can run nodetool cfstats command to check either any streams going on.
Related
We have a 6 node cassandra 3.11.3 cluster with ubuntu 16.04. These are virtual machines.
We are switching to physical machines on brand (8!) new servers that will have debian 11 and presumably cassandra 3.11.12.
Since the main version is always 3.11.x and ubuntu 16.04 is out of support, the question is: can we just let the new machines join the old cluster and then decommission the outdated?
I hope to get a tips about this becouse intuitively it seems fine but we are not too sure about that.
Thank you.
We have a 6 node cassandra 3.11.3 cluster with ubuntu 16.04. These are virtual machines. We are switching to physical machines on brand (8!)
Quick tip here; but it's a good idea to build your clusters in multiples of your RF. Not sure what your RF is, but if RF=3, I'd either stay with six or get one more and go to nine. It's all about even data distribution.
can we just let the new machines join the old cluster and then decommission the outdated?
In short, no. You'll want to upgrade the existing nodes to 3.11.12, first. I can't recall if 3.11.3 and 3.11.12 are SSTable compatible, but I wouldn't risk it.
Secondly, the best way to do this, is to build your new (physical) nodes in the cluster as their own logical data center. Start them up empty, and then run a nodetool rebuild on each. Once that's complete, then decommission the old nodes.
There is a bit simpler solution - move data from each virtual machine into a physical server, as following:
Prepare Cassandra installation on a physical machine, configure the same cluster name, etc.
1.Stop Cassandra in a virtual machine & make sure that it won't start
Copy all Cassandra data /var/lib/cassandra or something like from VM to the physical server
Start Cassandra process on a physical server
Repeat that process for all VM nodes, at some point, updating seeds, etc. After process is finished, you can add two physical servers that are left. Also, to speedup process, you can do initial copy of the data before stopping Cassandra in the VM, and after it's stopped, re-sync data with rsync or something like. This way you can minimize the downtime.
This approach would be much faster compared to the adding a new node & decommissioning the old one as we won't need to stream data twice. This works because after node is initialized, Cassandra identify nodes by assigned UUID, not by IP address.
Another approach is to follow instructions on replacement of the dead node. In this case streaming of data will happen only once, but it could be a bit slower compared to the direct copy of the data.
I have a Cassandra cluster with 3 nodes running on v3.11.4
I want to add 3 more nodes to this cluster. Now, Cassandra v4 is available so i have installed it on the new nodes.
When i restart Cassandra, the new nodes are unable to join the cluster.
Error: There are nodes in the cluster with a different schema version than us
I even tried added skip_schema options in jvm-server.options file but still the nodes could not join.
Please help me how can i add the new nodes in the existing cluster. I want to keep v4 for new nodes so i don't have to update these when upgrading older nodes to v4.
It isn't possible to add nodes running a new major version to the cluster. You will only be able to add nodes running Cassandra 3.11.
They won't be able to stream data to each other because they have different formats. This is the reason you can't run repairs during an upgrade. You also can't add or decommission nodes in the middle of an upgrade. Cheers!
So the plan forward here, would be to shut down the Cassandra 4.0 nodes. Then upgrade the 3.11 nodes to 4.0. Then adding the new 4.0 nodes should work as expected.
I was trying to change one of my node to a machine with high specs (more memory and CPU). So I run node decommission wait to leave the ring, terminate the machine and add a new machine. After that I configured the cassandra.yaml with
cluster_name
listend_address
rpc_broadcast_address
-seeds: with the ip's machine
After starting the cassandra service, the seed node joined the ring right way and with a low load. Which for me is really strange, since others node took a lot of time to join the ring.
After 1h the seed node is still with the same load.
What should I do to add the seed node?
Thanks in advance.
Yes, many newer users of Cassandra have this idea that a seed node is some mystical master-node equivalent. It's really not anything special. Essentially, a node needs to know about the cluster topology at start-time, and the seeds property provides a list of nodes that should be there.
In theory, you can have a new node designate any existing node as its seed node. And that node could designate another node as its seed node. And so on, and so on. All it does is use that node to figure out the cluster topology.
After starting the cassandra service, the seed node joined the ring right way and with a low load. Which for me is really strange, since others node took a lot of time to join the ring.
After 1h the seed node is still with the same load.
What should I do to add the seed node?
Seed nodes do not stream data. The extra steps required to get data on to a seed node, are one of the main reasons that it's not a good idea to designate all nodes as seed nodes.
You could just run a nodetool repair or rebuild on the new seed node, and that would stream data to it. The problem with this approach, is that it will still be accepting requests (and probably failing) while it is streaming data.
The other approach, would be to add the new node while specifying other existing node(s) in that node's seeds list.
Once it is up and has streamed data, then you have a couple of options:
Leave everything as-is, and any future nodes can use your new node in its seed list.
If your other existing nodes have a node(s) in their seed list that don't exist anymore, you can update those to point to your new node as a seed.
The nice part about option #2, is that you can change that in the cassandra.yaml and not have to restart them. This is because the only time you'll ever need that change is when a restart happens anyway. The seed node designation don't come into play during normal operations.
Hope that helps!
I added a new node to Cassandra cluster by making it seed node and than started rebuilding it (nodetool rebuild ) command. Although node joined the cluster quickly, the rebuild process that started streaming from all nodes in selected caused the whole dc nodes to slow down. The impact on application is severe. I'll have to stop rebuild process in order to keep normal operation ON!.
Here, I'm seeking advice if you guys can share ways/tricks so to minimize the impact of (node rebuild ) operation on rest of dc nodes and application.
I'll much appreciate your suggestions - thanks for reading my message and your help in advance.
While adding a new node you shouldn't make it a seed node. The seed node is used to bootstrap other nodes and join them in cluster. Making the new node as a seed node will not allow to join the new node in the cluster. Follow the steps provided in the Cassandra docs provided in the link below.
https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
This is the best way to add a new node in the cluster.
Note: Make sure the new node is not listed in the -seeds list. Do not make all nodes seed nodes. Please read Internode communications (gossip).
As I understand, you added a node as a seed node just so it will not bootstrap and join the cluster instantly. While is approach is right for it joins the cluster quickly, the downside is, it will not bootstrap and hence will not copy all the data from other nodes that it is responsible for. When you run a rebuild on that node, data is blindly copied (without doing any validation) from other nodes, which can choke up existing nodes throughput and your network pipeline. This approach is very safe and is recommended when using adding a new DC but not when adding nodes to an existing DC.
When you are adding a node, the simplest way is to add a node using the procedure described here. https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
When the node bootstrap, it will copy data needed from other nodes but will not start taking client connections until it fully bootstraps and validates the data. So, add one node at a time and let it bootstrap so all the necessary data is copied and validated. After you are done adding the number of nodes you desire, run a cleanup on all the previously joined nodes to cleanup all the keys for which the old nodes are not responsible.
Is it possible to add a new node to an existing cluster in cassandra 1.2 without running nodetool cleanup on each individual node once data has been added?
It probably isn't but I need to ask because I'm trying to create an application where each user's machine is a server allowing for endless scaling.
Any advice would be appreciated.
Yes, it is possible. But you should be aware of the side-effects of not doing so.
nodetool cleanup purges keys that are no longer allocated to that node. According to the Apache docs, these keys count against the allocated data for that node, which can cause the auto bootstrap process for the next node to not properly balance the ring. So depending on how you are bringing new user machines into the ring, this may or may not be a problem.
Also keep in mind that nodetool cleanup only needs to be run on nodes that lost keyspace to the new node - i.e. adjacent nodes, not all nodes, in the cluster.