There is 3 node cassandra cluster running and which is serving Production Traffic And in cassandra.yaml file "endpoint_snitch: GossipingPropertyFileSnitch" is configured but somehow we have forgot to remove file cassandra-topology.properties from cassandra conf directory. As per Cassandra documentation if you are using GossipingPropertyFileSnitch you should remove cassandra-topology.properties file.
Now As all three nodes are running and serving Production traffic So can I remove this file all three nodes or I have to remove this file after shutdown the nodes one by one.
Apache Cassandra Version is "3.11.2"
./bin/nodetool status
Datacenter: dc1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN x.x.x.x1 409.39 GiB 256 62.9% cshdkd-6065-4813-ae53-sdh89hs98so RAC1
UN x.x.x.x2 546.33 GiB 256 67.8% jfdsdk-f18f-4d46-af95-33jw9yhfcsd RAC2
UN x.x.x.x3 594.73 GiB 256 69.3% 7s9skk-a27f-4875-a410-sdsiudw9eww RAC3
If the cluster is already migrated to GossippingPropertyFileSnitch, then you can safely remove that file without stopping the cluster nodes. See the item 7 in DSE 5.1 documentation (compatible with Cassandra 3.11)
Related
Are there any known issues with initial_token collision when adding nodes to a cluster in a VM environment?
I'm working on a 4 node cluster set up on a VM. We're running into issues when we attempt to add nodes to the cluster.
In the cassandra.yaml file, initial_token is left blank.
Since we're running > 1.0 cassandra, auto_bootstrap should be true by default.
It's my understanding that each of the nodes in the cluster should be assigned an initial token at startup.
This is not what we're currently seeing.
We do not want to manually set the value for initial_token for each node (kind of defeats the goal of being dynamic..)
We also have set the partitioner to random: partitioner: org.apache.cassandra.dht.RandomPartitioner
I've outlined the steps we follow and results we are seeing below.
Can someone please asdvise as to what we're missing here?
Here are the detailed steps we are taking:
1) Kill all cassandra instances and delete data & commit log files on each node.
2) Startup Seed Node (S.S.S.S)
Starts up fine.
3) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
4) X.X.X.X Startup
INFO [GossipStage:1] 2012-11-29 21:16:02,194 Gossiper.java (line 850) Node /X.X.X.X is now part of the cluster
INFO [GossipStage:1] 2012-11-29 21:16:02,194 Gossiper.java (line 816) InetAddress /X.X.X.X is now UP
INFO [GossipStage:1] 2012-11-29 21:16:02,195 StorageService.java (line 1138) Nodes /X.X.X.X and /Y.Y.Y.Y have the same token 113436792799830839333714191906879955254. /X.X.X.X is the new owner
WARN [GossipStage:1] 2012-11-29 21:16:02,195 TokenMetadata.java (line 160) Token 113436792799830839333714191906879955254 changing ownership from /Y.Y.Y.Y to /X.X.X.X
5) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
113436792799830839333714191906879955254
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
W.W.W.W datacenter1 rack1 Up Normal 123.87 KB 100.00% 113436792799830839333714191906879955254
6) Y.Y.Y.Y Startup
INFO [GossipStage:1] 2012-11-29 21:17:36,458 Gossiper.java (line 850) Node /Y.Y.Y.Y is now part of the cluster
INFO [GossipStage:1] 2012-11-29 21:17:36,459 Gossiper.java (line 816) InetAddress /Y.Y.Y.Y is now UP
INFO [GossipStage:1] 2012-11-29 21:17:36,459 StorageService.java (line 1138) Nodes /Y.Y.Y.Y and /X.X.X.X have the same token 113436792799830839333714191906879955254. /Y.Y.Y.Y is the new owner
WARN [GossipStage:1] 2012-11-29 21:17:36,459 TokenMetadata.java (line 160) Token 113436792799830839333714191906879955254 changing ownership from /X.X.X.X to /Y.Y.Y.Y
7) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
113436792799830839333714191906879955254
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
Y.Y.Y.Y datacenter1 rack1 Up Normal 123.87 KB 100.00% 113436792799830839333714191906879955254
8) Z.Z.Z.Z Startup
INFO [GossipStage:1] 2012-11-30 04:52:28,590 Gossiper.java (line 850) Node /Z.Z.Z.Z is now part of the cluster
INFO [GossipStage:1] 2012-11-30 04:52:28,591 Gossiper.java (line 816) InetAddress /Z.Z.Z.Z is now UP
INFO [GossipStage:1] 2012-11-30 04:52:28,591 StorageService.java (line 1138) Nodes /Z.Z.Z.Z and /Y.Y.Y.Y have the same token 113436792799830839333714191906879955254. /Z.Z.Z.Z is the new owner
WARN [GossipStage:1] 2012-11-30 04:52:28,592 TokenMetadata.java (line 160) Token 113436792799830839333714191906879955254 changing ownership from /Y.Y.Y.Y to /Z.Z.Z.Z
9) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
113436792799830839333714191906879955254
W.W.W.W datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
Z.Z.Z.Z datacenter1 rack1 Up Normal 123.87 KB 100.00% 113436792799830839333714191906879955254
Thanks in advance.
This is what I did to fix this problem:
Stop the Cassandra service
Set auto_bootstrap: false on the seed node.
Empty data and commitlog directories:
sudo rm -rf /var/lib/cassandra/data/*
sudo rm -rf /var/lib/cassandra/commitlog/*
And then restart the service
I tested this with Cassandra 3.7.
Clearly your nodes are holding onto some past cluster information that is being used at startup. Make sure to delete the LocationInfo directories, which contain the data about the cluster. You have a very strange token layout (where's the 0 token, for example?), so you're certainly going to need to reassign them if you want the proper ownership.
It may help to explain how token assignment works, so let me also address this. In a brand new cluster, the first node will get assigned token 0 by default and will have 100% ownership. If you do not specify a token for your next node, Cassandra will calculate a token such that the original node owns the lower 50% and the new node the higher 50%.
When you add node 3, it will insert the token between the first and second, so you'll actually end up with ownership that looks like 25%, 25%, 50%. This is really important, because the lesson to learn here is that Cassandra will NEVER reassign a token by itself to balance the ring. If you want your ownership balanced properly, you must assign your own tokens. This is not hard to do, and there's actually a utility provided to do this.
So Cassandra's initial bootstrap process, while dynamic, may not yield the desired ring balance. You can't simply allow new nodes to join willy nilly without some intervention to make sure you get the desired result. Otherwise you will end up with the scenario you have laid out in your question.
My Error is as below
OpsCenter was not able to add the cluster: OpsCenter was unable to resolve the ip for Test_Cluster, please specify seed nodes using the rpc_address
My OS is CentOS 7
I install DSE 6
i found that datastax forbidden my ip
I'm using cassandra 3.11.0 on Ubuntu 16.04 (VM).
So with 2 nodes in same cluster using 0.8s for select * from emp.
But when i add new node in same cluster it using 2.0s
Keyspace
class: SimpleStrategy
replication_factor: 2
Table
CREATE TABLE emp (key int PRIMARY KEY,empname text);
Cassandra.yaml
Node1 and Node2 have a same config
autobootstrap: false
seed provider: "node1,node2"
num_tokens: 256
endpoint_snitch: GossipingPropertyFileSnitch
Node3 (new node) config
autobootstrap: true
seed provider: "node1,node2"
num_tokens: 256
endpoint_snitch: GossipingPropertyFileSnitch
dsetool status
DC: dc1 Workload: Cassandra Graph: no
======================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns VNodes Rack Health [0,1]
UN 192.168.1.130 810.47 MiB ? 256 2a 0.90
UN 192.168.1.131 683.53 MiB ? 256 2a 0.90
UN 192.168.1.132 821.33 MiB ? 256 2a 0.90
DC: dc2 Workload: Analytics Graph: no Analytics Master: 192.168.2.131
=========================================================================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns VNodes Rack Health [0,1]
UN 192.168.2.130 667.05 MiB ? 256 2a 0.90
UN 192.168.2.131 845.48 MiB ? 256 2a 0.90
UN 192.168.2.132 887.92 MiB ? 256 2a 0.90
when I try to launch the spark-submit job
dse -u user -p password spark-submit --class com.sparkLauncher test.jar prf
i am getting the following error (edited)
ERROR 2017-09-14 20:14:14,174 org.apache.spark.deploy.rm.DseAppClient$ClientEndpoint: Failed to connect to DSE resource manager
java.io.IOException: Failed to register with master: dse://?
....
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: The method DseResourceManager.registerApplication does not exist. Make sure that the required component for that method is active/enabled
....
ERROR 2017-09-14 20:14:14,179 org.apache.spark.deploy.rm.DseSchedulerBackend: Application has been killed. Reason: Failed to connect to DSE resource manager: Failed to register with master: dse://?
org.apache.spark.SparkException: Exiting due to error from cluster scheduler: Failed to connect to DSE resource manager: Failed to register with master: dse://?
....
WARN 2017-09-14 20:14:14,179 org.apache.spark.deploy.rm.DseSchedulerBackend: Application ID is not initialized yet.
ERROR 2017-09-14 20:14:14,384 org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
ERROR 2017-09-14 20:14:14,387 org.apache.spark.deploy.DseSparkSubmitBootstrapper: Failed to start or submit Spark application
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
I can confirm that I have granted privileges as mentioned in this documentation, https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/security/secAuthSpark.html
I am trying this on AWS if that makes a difference and I can confirm that the routes between the nodes are all open.
I am able to start spark shell from any of the spark nodes, can bring up the Spark UI, can get spark master from cqlsh commands
Any pointers will be helpful, thanks in advance!
The master address must point to one or more nodes in a valid Analytics enabled datacenter.
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException:
The method DseResourceManager.registerApplication does not exist.
Make sure that the required component for that method is active/enabled```
Indicates that the connected node was not analytics enabled.
If you run from a non analytics node you must still point at one of the analytics nodes in the master ui.
dse://[Spark node address[:port number]]?[parameter name=parameter value;]...
By default the dse://? url connects to localhost for it's initial cluster connection.
See the documentation for more information.
For some reason I am unable to pin point, I can run it as mentioned in cluster mode but not in client mode
Are there any known issues with initial_token collision when adding nodes to a cluster in a VM environment?
I'm working on a 4 node cluster set up on a VM. We're running into issues when we attempt to add nodes to the cluster.
In the cassandra.yaml file, initial_token is left blank.
Since we're running > 1.0 cassandra, auto_bootstrap should be true by default.
It's my understanding that each of the nodes in the cluster should be assigned an initial token at startup.
This is not what we're currently seeing.
We do not want to manually set the value for initial_token for each node (kind of defeats the goal of being dynamic..)
We also have set the partitioner to random: partitioner: org.apache.cassandra.dht.RandomPartitioner
I've outlined the steps we follow and results we are seeing below.
Can someone please asdvise as to what we're missing here?
Here are the detailed steps we are taking:
1) Kill all cassandra instances and delete data & commit log files on each node.
2) Startup Seed Node (S.S.S.S)
Starts up fine.
3) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
4) X.X.X.X Startup
INFO [GossipStage:1] 2012-11-29 21:16:02,194 Gossiper.java (line 850) Node /X.X.X.X is now part of the cluster
INFO [GossipStage:1] 2012-11-29 21:16:02,194 Gossiper.java (line 816) InetAddress /X.X.X.X is now UP
INFO [GossipStage:1] 2012-11-29 21:16:02,195 StorageService.java (line 1138) Nodes /X.X.X.X and /Y.Y.Y.Y have the same token 113436792799830839333714191906879955254. /X.X.X.X is the new owner
WARN [GossipStage:1] 2012-11-29 21:16:02,195 TokenMetadata.java (line 160) Token 113436792799830839333714191906879955254 changing ownership from /Y.Y.Y.Y to /X.X.X.X
5) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
113436792799830839333714191906879955254
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
W.W.W.W datacenter1 rack1 Up Normal 123.87 KB 100.00% 113436792799830839333714191906879955254
6) Y.Y.Y.Y Startup
INFO [GossipStage:1] 2012-11-29 21:17:36,458 Gossiper.java (line 850) Node /Y.Y.Y.Y is now part of the cluster
INFO [GossipStage:1] 2012-11-29 21:17:36,459 Gossiper.java (line 816) InetAddress /Y.Y.Y.Y is now UP
INFO [GossipStage:1] 2012-11-29 21:17:36,459 StorageService.java (line 1138) Nodes /Y.Y.Y.Y and /X.X.X.X have the same token 113436792799830839333714191906879955254. /Y.Y.Y.Y is the new owner
WARN [GossipStage:1] 2012-11-29 21:17:36,459 TokenMetadata.java (line 160) Token 113436792799830839333714191906879955254 changing ownership from /X.X.X.X to /Y.Y.Y.Y
7) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
113436792799830839333714191906879955254
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
Y.Y.Y.Y datacenter1 rack1 Up Normal 123.87 KB 100.00% 113436792799830839333714191906879955254
8) Z.Z.Z.Z Startup
INFO [GossipStage:1] 2012-11-30 04:52:28,590 Gossiper.java (line 850) Node /Z.Z.Z.Z is now part of the cluster
INFO [GossipStage:1] 2012-11-30 04:52:28,591 Gossiper.java (line 816) InetAddress /Z.Z.Z.Z is now UP
INFO [GossipStage:1] 2012-11-30 04:52:28,591 StorageService.java (line 1138) Nodes /Z.Z.Z.Z and /Y.Y.Y.Y have the same token 113436792799830839333714191906879955254. /Z.Z.Z.Z is the new owner
WARN [GossipStage:1] 2012-11-30 04:52:28,592 TokenMetadata.java (line 160) Token 113436792799830839333714191906879955254 changing ownership from /Y.Y.Y.Y to /Z.Z.Z.Z
9) Run nodetool -h W.W.W.W ring and see:
Address DC Rack Status State Load Effective-Ownership Token
113436792799830839333714191906879955254
W.W.W.W datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
S.S.S.S datacenter1 rack1 Up Normal 28.37 GB 100.00% 24360745721352799263907128727168388463
Z.Z.Z.Z datacenter1 rack1 Up Normal 123.87 KB 100.00% 113436792799830839333714191906879955254
Thanks in advance.
This is what I did to fix this problem:
Stop the Cassandra service
Set auto_bootstrap: false on the seed node.
Empty data and commitlog directories:
sudo rm -rf /var/lib/cassandra/data/*
sudo rm -rf /var/lib/cassandra/commitlog/*
And then restart the service
I tested this with Cassandra 3.7.
Clearly your nodes are holding onto some past cluster information that is being used at startup. Make sure to delete the LocationInfo directories, which contain the data about the cluster. You have a very strange token layout (where's the 0 token, for example?), so you're certainly going to need to reassign them if you want the proper ownership.
It may help to explain how token assignment works, so let me also address this. In a brand new cluster, the first node will get assigned token 0 by default and will have 100% ownership. If you do not specify a token for your next node, Cassandra will calculate a token such that the original node owns the lower 50% and the new node the higher 50%.
When you add node 3, it will insert the token between the first and second, so you'll actually end up with ownership that looks like 25%, 25%, 50%. This is really important, because the lesson to learn here is that Cassandra will NEVER reassign a token by itself to balance the ring. If you want your ownership balanced properly, you must assign your own tokens. This is not hard to do, and there's actually a utility provided to do this.
So Cassandra's initial bootstrap process, while dynamic, may not yield the desired ring balance. You can't simply allow new nodes to join willy nilly without some intervention to make sure you get the desired result. Otherwise you will end up with the scenario you have laid out in your question.