I added nodes to a cluster which initialy used the wrong network interface as listen_adress. I fixed it by changeing the listen_address to the correct IP. The cluster is running well with that configuration but clients trying to connect to that cluster still receive the wrong IPs as Metadata from cluster. Is there any way to refresh metadata of a cluster whithout decommissioning the nodes and setting up new ones again?
First of all, you may try to follow this advice: http://www.datastax.com/documentation/cassandra/2.1/cassandra/operations/ops_gossip_purge.html
You will need to restart the entire cluster on a rolling basis - one node at a time
If this does not work, try this on each node:
USE system;
SELECT * FROM peers;
Then delete bad records from the peers and restart the node, then go to the next node and do it again.
Related
I'm configuring a 3 node Cassandra cluster (multi datacenter) and everything works well until I set up the authentication process, setting from AllowAllAuthenticator to PasswordAuthenticator, as defined in Cassandra's doc.
The problem is, once I changed and restart nodes, I cannot access anymore the database, in this case with cassandra superuser, displaying this message:
Connection error: ('Unable to connect to any servers', {'10.0.0.10': AuthenticationFailed('Failed to authenticate to 10.0.0.10: Error from server: code=0100 [Bad credentials] message="Unable to perform authentication: Cannot achieve consistency level QUORUM"',)})
It's important to mention that before to set up the authenticator, I already updated the system_auth to NetworkTopologyStratety, setting up each node.
Also, without authentication all replications work fine, which means the cluster is running fine.
Does anyone have some idea about it? This is really driving me crazy, once I didn't find any reference about it.
All the best!
My guess is that you need to run repair on all of the nodes for the "system_auth", and if you're running DSE, ensure any keyspace that starts with "dse" that has "simple strategy" is updated to Network Topology Strategy with appropriate DC and RF settings - and run repair on each node for those as well.
That should solve your problem. My guess is that you created your users and then updated the keyspaces to use Network Topology. Once done, any new records will be be propagated correctly, but the existing records need repair to "fan them out" as it won't happen on its own.
-Jim
I'm a beginner in Cassandra. I want to understand the two nodes(the streaming node and joining node) how to work when a new node joins an existing cluster. Can they provide normal services to the outside?
If the service is provided normally. I assumed the joining node is nodeA, and the node where the fetching data is nodeB. That means nodeA fetch data from nodeB. Assume that the data range C is transmitted from the nodeB to the nodeA, at which time new data falling into the range C is inserted into the cluster. Is the new data written to nodeA or nodeB?
I'm using datastax community edition of cassandra, version 3.11.3.
thanks!
Sun your question is bit confusing .. but what I make of it is , You want to understand the process to adding new node to existing cluster.
Adding a new node to existing cluster requires cassandra.yaml properties for new node identification and communications.
Set the following properties in the cassandra.yaml and, depending on the snitch, the cassandra-topology.properties or cassandra-rackdc.properties configuration files:
auto_bootstrap - This property is not listed in the default cassandra.yaml configuration file, but it might have been added and set to false by other operations. If it is not defined in cassandra.yaml, Cassandra uses true as a default value. For this operation, search for this property in the cassandra.yaml file. If it is present, set it to true or delete it..
cluster_name - The name of the cluster the new node is joining.
listen_address/broadcast_address - Can usually be left blank. Otherwise, use IP address or host name that other Cassandra nodes use to connect to the new node.
endpoint_snitch - The snitch Cassandra uses for locating nodes and routing requests.
num_tokens - The number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
seeds - Determines which nodes the new node contacts to learn about the cluster and establish the gossip process. Make sure that the -seeds list includes the address of at least one node in the existing cluster.
When a new node joins a cluster using topology defined, Seed nodes starts the gossip with the new node by the time it do not communicate with the client directly. Once the gossip completes the new node is ready to take the actual data load.
Hope this helps in understanding the process.
Background
6 node Kafka Cluster
3 node Zookeeper Cluster
3 node Nimbus Cluster
Apache Storm Worker hosts dynamically adjusted using amazon spot fleet
Scenario
For a particular topology for a given partition it subscribes to, the Zookeeper entry looks as follows
{"topology":{"id":"Topology_Name-25-1520374231","name":"Topology_Name"},"offset":217233,"partition":0,"broker":{"host":"Zk_host_name","port":9092},"topic":"topic1"}
Now for worker hosts to access Zk_host_name, a mapping is added on each worker host in /etc/hosts file as ip ZK_host_name
Now we decided to move to something called Route 53 DNS management service provided by AWS. That way a fixed name such as QA-ZK-Host1 can be set and be mapped to corresponding ip. So that ip can be changed in future giving a flexibility.
Now the original entry as above needed to be changed for the sake of consistency. So corresponding topology was stopped, so as to avoid ongoing changes to offset and using set command the value of the hostname is changed.
set /node_path {"topology":{"id":"Topology_Name-25-1520374231","name":"Topology_Name"},"offset":217233,"partition":0,"broker":{"host":"QA-ZK-Host1","port":9092},"topic":"topic1"}
Problem
The above command works fine and get command on the path gives the changed value. But the moment topology is restarted, old name is restored.
So how to make it persist even after topology restart.
The object you are referencing is being written to Storm's Zookeeper here https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/PartitionManager.java#L341.
The "broker" property is created at https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L186. As you can see, the host property is not your Zookeeper host, but the host running Kafka. The value is being read from Kafka's Zookeeper (see point 3 at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper).
If you want to change the value, you'll likely need to do it in Kafka. Take a look at http://kafka.apache.org/090/documentation.html (or whatever version you're using) and search for "advertised.host.name", I think that's the setting you want to change.
My Understanding on auto_bootstrap is
Below are my understanding about auto_bootstrap property. At first, please correct me if I am wrong at any point.
Initially the property ‘auto_bootstrap’ will not be available in the cassandra.yaml file. This means that the default value was ‘true’.
true - this means that bootstrap/stream the data to the respective node from all the other nodes while starting/restarting
false - do not stream the data while starting/restarting
Where do we need ‘auto_bootstrap: true’
1) When a new node needs to be added in the existing cluster, this needs to set to ‘true’ to bootstrap the data automatically from all the other nodes in the cluster. This will take some considerable amount of time (based on the current load of the cluster) to get the new node added in the cluster. But this will make the load balance automatically in the cluster.
Where do we need ‘auto_bootstrap: false’
1) When a new node needs to be added quickly in the existing cluster without bootstrapping the data, this needs to set to ‘false’. The new node will be added quickly irrespective of the current load of the cluster. Later we need to manually stream the data to the new node to make cluster load balanced.
2) When initializing the fresh cluster with no data, this needs to set to ‘false’. At least the first seed node to be started/added in the fresh cluster should have the value as ‘false’.
My Question is
We are using Cassandra 2.0.3 of six nodes with two data centers (each has 3 nodes). Our Cassandra is a stand-alone process (not service). I am going to change few properties in cassandra.yaml file for one node. It is apparent that node should be restarted after updating the cassandra.ymal file to take the changes effect. Our cluster is loaded with huge data.
How to restart the node
After killing the node, I can simply restart the node as below
$ cd install_location
$ bin/cassandra
This means that restart the node with no auto_bootstrap property (default is true).
with 'true'
1) The node to be restarted currently has its own huge data. Does the node bootstrap again all its own data and replace the existing data.
2) Will it take more time the node to join the cluster again.
with 'false'
I do not want to bootstrap the data. So
3) Can I add the property as auto_bootstrap: false and restart the node as mentioned above.
4) After successful restart I will go and delete the auto_bootstrap property. Is that okay?
Else
5) As I am restarting the node with the same ip address, Will the cluster automatically identify that this is an existing node through gossip info and hence restart the node without streaming the data despite auto_bootstrap is set to true or not present in cassandra.yaml file?
As I am restarting an existing node with the same ip address, restart will happen without streaming any data despite the value of auto_bootstrap. So we can merely restart the existing node without touching any parameters. So option 5 fits here.
First of all, you should always run
nodetool drain
on the node before killing Cassandra so that client connections/ongoing operations have a chance to gracefully complete.
Assuming that the node was fully bootstrapped & had status "Up" and "Joined": when you start Cassandra up again, the node will not need to bootstrap again since it's already joined the cluster & taken ownership of certain sets of tokens. However, it will need to catch up with the data that has been mutated since it was down. Therefore, the commitlogs that occurred during that time will be streamed to the node and the changes will be applied. So, it will take much less time to start up after it has bootstrapped once. Just don't leave it down for too long.
You should not set auto_bootstrap to false unless you're creating the first seed node for a new cluster.
The node will be identified as a pre-existing node which has tokens assigned to it by virtue of the host id that is assigned to it when it joins the cluster. The IP address does not matter unless it is a seed node.
I might be misunderstanding something here, as it's not clear to me how I should connect to a Cassandra cluster. I have a Cassandra 1.2.1 cluster of 5 nodes managed by Priam, on AWS. i would like to use Astyanax to connect to this cluster by using a code similar to the code bellow:
conPool = new ConnectionPoolConfigurationImpl(getConecPoolName()) .setMaxConnsPerHost(CONNECTION_POOL_SIZE_PER_HOST).setSeeds(MY_IP_SEEDS)
.setMaxOperationsPerConnection(100) // 10000
What should I use as MY_IP_SEEDS? Should I use the IPs of all my nodes split by comma? Or should I use the IP of just 1 machine (the seed machine)? If I use the ip of just one machine, I am worried about overloading this machine with too many requests.
I know Priam has the "get_seeds" REST api (https://github.com/Netflix/Priam/wiki/REST-API) that for each node returns a list of IPs and I also know there is one seed per RAC. However, I am not sure what would happen if the seed node gets down... I would need to connect to others when trying to make new connections, right?
Seed nodes are only for finding the way into the cluster on node startup - no overload problems.
Of course one of the nodes must be reachable and up in the cluster to get the new one up and running.
So the best way is to update the seed list from Priam before starting the node. Priam should be behind an automatically updated DNS entry.
If you're highest availability you should regularly store the current list of seeds from Priam and store them in a mirrored fashion just as you store your puppet or chef config to be able to get nodes up even when Priam isn't reachable.