I am looking to add 3 nodes to an existing 6 nodes cassandra cluster, but I'm a bit confused on how best to do this due to the token assignments.
Currently, the existing 6 node cluster is not using vNodes (this can't be changed) and is using RandomPartitioner and so the current tokens were added as per the token generator. The issue is that adding 3 nodes to a 6 node cluster means that the recalculated tokens would put the new node 7 with the same token as the current node 5.
What is the best practice here? Should I do a nodetool move on the existing nodes to add in the recalculated tokens, THEN bootstrap the new nodes with the correct config and tokens. Or do I add the new nodes with no token, and once bootstrapped, then nodetool move across all the nodes adding in the newly calculated tokens, starting from the second node (as first node is always 0 with RandomPartitioner).
I've done a lot of reading, but can't seem to find a scenario that covers this eventuality. And I can't add more than 3 nodes, long story...
Any help greatly received!
You need to recalculate the tokens for you whole cluster and assign the new tokens to the existing nodes. Detailed instructions can be found here. Since you are recalculating for the whole cluster, you should not have the issues that you are talking about.
I think that the best solution would be to add a new DC where you would use vnodes and Murmur3. After the data is replicated and you have moved all your clients to the new DC, you would decommission the old DC.
Related
Can someone explain the working of vnodes allocation in Cassandra?
If we have a cluster of N nodes and a new node is added how are token ranges allocated to this new node?
Rebalancing a cluster is automatically accomplished when adding or removing nodes. When a node joins the cluster, it assumes responsibility for an even portion of data from the other nodes in the cluster. If a node fails, the load is spread evenly across other nodes in the cluster.
Here is some reading that might help you better understand how vnodes work and how ranges are being allocated - Virtual nodes in Cassandra 1.2
As I said above, Cassandra automatically handles the calculation of token ranges for each node in the cluster in proportion to their num_tokens value. Token assignments for vnodes are calculated by the org.apache.cassandra.dht.tokenallocator.ReplicationAwareTokenAllocator class.
When a new node joins the cluster, it will inject it's own ranges and steal some rages from the existing nodes. Also this video might help
We are in the process of adding new nodes to our existing cassandra 3.x cluster. Basically following the steps outlined in this article (https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html).
Currently, out 3.x cluster does incremental repairs. What I'm not 100 percent sure about is if we need to do anything after we do the node cleanup. Specifically, are our new added nodes setup to do incremental repairs after following the procedure listed above?
Thanks
Marshall
Adding a node involves rebalancing the token distribution among nodes in the cluster and bootstrapping the new node. Repair is a regular maintenance process and it's not necessarily needed in the adding node process.
For the token redistribution part, simplified example is if your have 20 tokens and 4 nodes initially. If random enough, each node would be the primary node for 5 tokens. When add a node and change the configuration, 20 tokens would be distributed to 5 nodes and each node would be the primary node for 4 tokens. When you run bootstrap to add the new node, the new node knows what tokens belongs to it and it will stream missing tokens from other existing nodes. After bootstrapping is done, nodetool cleanup on the existing nodes would remove tokens which don't belong to them anymore. To sum up, bootstrapping new node would redistribute tokens and stream objects to new nodes based on the distribution. you don't need repair in this process to stream data. cleanup would remove objects which the ownership is changed.
However, repair is a regular process to guarantee data consistency and incremental or full options are out of scope in terms of talking about adding node.
Good reference to read under the hood on what happens when you change the topology of a cluster
I added a new node to Cassandra cluster by making it seed node and than started rebuilding it (nodetool rebuild ) command. Although node joined the cluster quickly, the rebuild process that started streaming from all nodes in selected caused the whole dc nodes to slow down. The impact on application is severe. I'll have to stop rebuild process in order to keep normal operation ON!.
Here, I'm seeking advice if you guys can share ways/tricks so to minimize the impact of (node rebuild ) operation on rest of dc nodes and application.
I'll much appreciate your suggestions - thanks for reading my message and your help in advance.
While adding a new node you shouldn't make it a seed node. The seed node is used to bootstrap other nodes and join them in cluster. Making the new node as a seed node will not allow to join the new node in the cluster. Follow the steps provided in the Cassandra docs provided in the link below.
https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
This is the best way to add a new node in the cluster.
Note: Make sure the new node is not listed in the -seeds list. Do not make all nodes seed nodes. Please read Internode communications (gossip).
As I understand, you added a node as a seed node just so it will not bootstrap and join the cluster instantly. While is approach is right for it joins the cluster quickly, the downside is, it will not bootstrap and hence will not copy all the data from other nodes that it is responsible for. When you run a rebuild on that node, data is blindly copied (without doing any validation) from other nodes, which can choke up existing nodes throughput and your network pipeline. This approach is very safe and is recommended when using adding a new DC but not when adding nodes to an existing DC.
When you are adding a node, the simplest way is to add a node using the procedure described here. https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
When the node bootstrap, it will copy data needed from other nodes but will not start taking client connections until it fully bootstraps and validates the data. So, add one node at a time and let it bootstrap so all the necessary data is copied and validated. After you are done adding the number of nodes you desire, run a cleanup on all the previously joined nodes to cleanup all the keys for which the old nodes are not responsible.
I have a 4 node cluster and I have upgraded all the nodes from an older version to Cassandra 1.2.8. Total data present in the cluster is of size 8 GB. Now I need to enable vNodes on all the 4 nodes of cluster without any downtime. How can I do that?
As Nikhil said, you need to increase num_tokens and restart each node. This can be done one at once with no down time.
However, increasing num_tokens doesn't cause any data to redistribute so you're not really using vnodes. You have to redistribute it manually via shuffle (explained in the link Lyuben posted, which often leads to problems), by decommissioning each node and bootstrapping back (which will temporarily leave your cluster extremely unbalanced with one node owning all the data), or by duplicating your hardware temporarily just like creating a new data center. The latter is the only reliable method I know of but it does require extra hardware.
In the conf/cassandra.yaml you will need to comment out the initial_token parameter, and enable the num_tokens parameter (by default 256 I believe). Do this for each node. Then you will have to restart the cassandra service on each node. And wait for the data to get redistributed throughout the cluster. 8 GB should not take too much time (provided your nodes are all in the same cluster), and read requests will still be functional, though you might see degraded performance until the redistribution of data is complete.
EDIT: Here is a potential strategy to migrate your data:
Decommission two nodes of the cluster. The token-space should get distributed 50-50 between the other two nodes.
On the two decommissioned nodes, remove the existing data, and restart the Cassandra daemon with a different cluster name and with the num_token parameters enabled.
Migrate the 8 GB of data from the old cluster to the new cluster. You could write a quick script in python to achieve this. Since the volume of data is small enough, this should not take too much time.
Once the data is migrated in the new cluster, decommission the two old nodes from the old cluster. Remove the data and restart Cassandra on them, with the new cluster name and the num_tokens parameter. They will bootstrap and data will be streamed from the two existing nodes in the new cluster. Preferably, only bootstrap one node at a time.
With these steps, you should never face a situation where your service is completely down. You will be running with reduced capacity for some time, but again since 8GB is not a large volume of data you might be able to achieve this quickly enough.
TL;DR;
No you need to restart servers once the config has been edited
The problem is that enabling vnodes means a lot of the data is redistributed randomly (the docs say in a vein similar to the classic ‘nodetool move’
Is it possible to add a new node to an existing cluster in cassandra 1.2 without running nodetool cleanup on each individual node once data has been added?
It probably isn't but I need to ask because I'm trying to create an application where each user's machine is a server allowing for endless scaling.
Any advice would be appreciated.
Yes, it is possible. But you should be aware of the side-effects of not doing so.
nodetool cleanup purges keys that are no longer allocated to that node. According to the Apache docs, these keys count against the allocated data for that node, which can cause the auto bootstrap process for the next node to not properly balance the ring. So depending on how you are bringing new user machines into the ring, this may or may not be a problem.
Also keep in mind that nodetool cleanup only needs to be run on nodes that lost keyspace to the new node - i.e. adjacent nodes, not all nodes, in the cluster.