I am keep getting below error when I tried to run single node or multinode Cassandra cluster.
Single node cluster with default config works fine, however nodetool staus shows 127.0.0.1 as IP Address.
After changing listen_address: 192.168.1.143 (this is my ip address) on cassandra.yaml file I am getting below error.
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers
java.lang.RuntimeException: Unable to gossip with any peers
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1443)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:547)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:804)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:664)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:379)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
Well, after trying different approaches finally I was able to resolve it and able to run single and 3 node cluster.
Below are the configuration changes you need to make on cassandra.yaml file
First Node
--------------
listen_address: 192.168.1.143 (This should be your server/node IP)
seeds: "192.168.1.143" (For your first node please mention your node IP address)
Second Node
---------------
listen_address: 192.168.1.144 (This should be your server/node IP)
seeds: "192.168.1.143" (specify your first node IP, additionally, you can also mention current IP address ,192.168.1.144)
Third Node
---------------
listen_address: 192.168.1.145 (This should be your server/node IP)
seeds: "192.168.1.143" (specify your first/second node IP, additionally, you can also mention current IP address ,192.168.1.145)
After starting cassandra on all 3 servers, nodetool status returned the following
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.143 258.83 KiB 256 100.0% 7b3a0644-c8dd-4a47-9186-0237f3725941 rack1
UN 192.168.1.144 309.71 KiB 256 100.0% e7a11a60-d795-47ee-8d21-7cc21b4cbdca rack1
UN 192.168.1.145 309.71 KiB 256 100.0% b2a4545a-f279-r5h7-2fy6-23dk8fg5c8kq rack1
Hope this helps!!
Yes, for joining cassandra cluster first time. you should start seed node first then other nodes.
Related
I installed Data Stax 3.7 on my Windows machine(IP:10.175.12.249) and made following changes in my cassandra.yaml file:
cluster_name: 'Test_cluster'
listen_address: "10.175.12.249"
start_rpc: true
rpc_address: "0.0.0.0"
broadcast_rpc_address: "10.175.12.249"
seeds: "10.175.12.249"
endpoint_snitch: SimpleSnitch
Now, I started the service and cassandra is running fine on seed node.
I tried adding another node to my cluster. So I installed Data Stax 3.7 on another Windows machine(IP:192.168.158.78) and made following changes in cassandra.yaml file:
cluster_name: 'Test_cluster'
listen_address: "192.168.158.78"
start_rpc: true
rpc_address: "0.0.0.0"
broadcast_rpc_address: "192.168.158.78"
seeds: "10.175.12.249"
endpoint_snitch: SimpleSnitch
Now when I started the cassandra service on my 2nd machine, I am getting the following error:
INFO 09:41:27 Cassandra version: 3.7.0
INFO 09:41:27 Thrift API version: 20.1.0
INFO 09:41:27 CQL supported versions: 3.4.2 (default: 3.4.2)
INFO 09:41:27 Initializing index summary manager with a memory pool size of 100 MB and a resize interval of 60 minutes
INFO 09:41:27 Starting Messaging Service on /192.168.158.78:7000 (Intel(R) Centrino(R) Advanced-N 6235)
INFO 09:41:27 Scheduling approximate time-check task with a precision of 10 milliseconds
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any seeds
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1386)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:561)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:855)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714)
ERROR 09:41:58 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1386) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:561) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:855) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370) [apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.0.jar:3.7.0]
WARN 09:41:58 No local state or state is in silent shutdown, not announcing shutdown
INFO 09:41:58 Waiting for messaging service to quiesce
Below is the output of nodetool status on seed node(IP:10.175.12.249):
C:\Program Files\DataStax-DDC\apache-cassandra\bin>nodetool status
Datacenter: datacenter1
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
DN 192.168.158.78 ? 256 68.1% 6bc4e927-3def-4dfc-b5e7-31f5882ce475 rack1
UN 10.175.12.249 257.76 KiB 256 65.7% 300d731e-a27c-4922-aacc-6d42e8e49151 rack1
Thanks!!!
The - seeds: in conf/cassandra.yaml should have the same value (same IP or the hostname) as listen_address: in the same conf file.
I came across this error when the IP addresses were not matching. Try keeping the same and restart the cluster. Hope this helps...
I am using mac osx. i created 3 virtual box by virtualbox. I've installed centos7 minimal version on each of the virtual box.
Then i installed cassandra on each of the box. After installation it was starting by cqlsh and nodetool status command.
But after then when i was trying to link each other and edit cassandra.yaml file its started showing
('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
i've edited the cassandra.yaml file as follows:
cluster_name: 'Home Cluster'
num_tokens: 256
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
- seeds: "192.168.56.102,192.168.56.103"
storage_port: 7000
listen_address: 192.168.56.102
rpc_address: 192.168.56.102
rpc_port: 9160
endpoint_snitch: SimpleSnitch
my /etc/hosts file contains:
192.168.56.102 node01
192.168.56.103 node02
192.168.56.104 node03
Please tell me whats wrong i'm doing? My cassandra cluster not working.
solution: I got the solution from AKKI. The problem was enpoint_snitch. I made the endpoint_snitch=GossipingPropertyFileSnitch and it fixed. My now output is as follows:
[root#dbnode2 ~]# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.56.101 107.38 KB 256 62.5% 0526a2e1-e6ce-4bb4-abeb-b9e33f72510a rack1
UN 192.168.56.102 106.85 KB 256 73.0% 0b7b76c2-27e8-490f-8274-571d00e60c20 rack1
UN 192.168.56.103 83.1 KB 256 64.5% 6c8d80ec-adbb-4be1-b255-f7a0b63e95c2 rack1
I had faced similar problem,
I tried the following solution:
In Cassandra.yaml file check if you have,
start_rpc = true
Changed my endpoint snitch to
endpoint_snitch: GossipingFilePropertySnitch
Opened all ports Cassandra uses on my CentOS
Cassandra inter-node ports
Port number Description
7000 Cassandra inter-node cluster communication.
7001 Cassandra SSL inter-node cluster communication.
7199 Cassandra JMX monitoring port.
Cassandra client port
Port number Description
9042 Cassandra client port.
9160 Cassandra client port (Thrift).
Command to open ports on CentOs 7(Find it according to your OS):
>sudo firewall-cmd --zone=public --add-port=9042/tcp --permanent
>sudo firewall-cmd –reload
Then Restart your systems
Also it seems that you are changing the Cassandra.Yaml file after starting cassandra.
Make sure you edit your Cassandra.yaml file on all nodes before starting Cassandra
Also remember to start the seed node first.
I am having trouble with three nodes in Cassandra, each of them in an individual computer, as I am trying to set up my first Cassandra structure. I have set up everything as in the Datastax documentation, and I have the same configuration in the different cassandra.yaml of each machine (changing the relative ips). The thing is that after configuring everything, each computer sees each other as DN, and each machine (localhost) appears as UN, with the difference that in the .101 computer I can see two different datacenters, while in the other computers only one datacenter appears.
So in my 192.168.1.101 machine when I type
sudo nodetool status
I get this output:
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
DN 192.168.1.200 ? 256 ? 968d5d1e-a113-40ce-9521-e392a927ea5e r1
DN 192.168.1.102 ? 256 ? fc5c2dbe-8834-4040-9e77-c3d8199b6767 r1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 446.13 KB 256 ? 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
While when I type "nodetool status" in one of the other two machines, I get this output:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
DN 192.168.1.200 ? 256 ? 968d5d1e-a113-40ce-9521-e392a927ea5e rack1
UN 127.0.0.1 506,04 KB 256 ? fc5c2dbe-8834-4040-9e77-c3d8199b6767 rack1
DN 192.168.1.101 ? 256 ? 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
In OpsCenter I can only see my 192.168.1.101 machine:
... Which makes me think that something's odd in the yaml file of this machine and the others, but I have checked several times and it seems that the configuration is the same in the other computers. Enpoint_snitch is set to "GossipingPropertyFileSnitch".
Any tips on how to solve the reason why all the other nodes appear as Down Normal and why I am getting two datacenters would be highly appreaciated. It's driving me crazy!
Thanks for reading.
Any tips on how to solve the reason why all the other nodes appear as Down Normal and why I am getting two datacenters would be highly appreaciated. It's driving me crazy!
On each machine, edit the $CASSANDRA_HOME/conf/cassandra-rackdc.properties file to set:
dc=dc1
rack=rack1
nodetool status shows that you set the wrong DC name for 2 nodes (DC1 instead of dc1). It's case sensitive
It looks like some of the installed nodes were dead, so I deleted the nodes that were not the local machine in each of the nodes, ie:
nodetool removenode 968d5d1e-a113-40ce-9521-e392a927ea5e
nodetool removenode fc5c2dbe-8834-4040-9e77-c3d8199b6767
and after that I got the right output when I executed nodetools status
[machine1]~$ sudo nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 286.93 KB 256 ? 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
[machine2]~$ sudo nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 268.45 KB 256 ? fc5c2dbe-8834-4040-9e77-c3d8199b6767 rack1
And made sure that the parameters cluster_name, seeds, listen_address and rpc_address were right.
cluster_name: 'Test Cluster'
seeds: "192.168.1.101, 192.168.1.102"
listen_address: 192.168.1.101
rpc_address: 192.168.1.101
Changing listen_address and rpc_address to the corresponding ip of each machine in their corresponding cassandra.yaml file.
After that I got the right output (I am using only 2 machines for the nodes now):
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.101 309.13 KB 256 51.6% 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
UN 192.168.1.102 257.15 KB 256 48.4% fc5c2dbe-8834-4040-9e77-c3d8199b6767 rack1
I'm trying to set up a Cassandra multi node cluster in my computer just to test, but it seems not work... The Cassandra version is 1.1 and It runs on Ubuntu.
Fist of all, I've modified the cassandra.yaml file for each node as follows:
node0
initial_token: 0
seeds: "127.0.0.1"
listen_address: 127.0.0.1
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
node1
same as node0 exept for:
initial_token: 28356863910078205288614550619314017621 (get using
cassandra token generator)
listen_address: 127.0.0.2
After that, I've started first the seed node 127.0.0.1 and, once the node is up, I've started the other node 127.0.0.2. I've got the following:
[...]
INFO 06:09:27,146 Listening for thrift clients...
INFO 06:09:27,909 Node /127.0.0.1 is now part of the cluster
INFO 06:09:27,911 InetAddress /127.0.0.1 is now UP
INFO 06:09:27,913 Nodes /127.0.0.1 and /127.0.0.2 have the same token 0. Ignoring /127.0.0.1
Running nodetool -h localhost ring it shows:
Address: 127.0.0.2
DC: datacenter1
Rack: rack1
Status: Up
State: Normal
Load: 11,21 KB
Owns: 100,00%
Token: 0
As you can see, only the information of the second node is showed owning 100% of the ring. Indeed, the token is initialized to 0 instead of to the value I defined at its cassandra.yaml file.
The gossip Info is:
/127.0.0.2
LOAD:25559.0
STATUS:NORMAL,0
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RELEASE_VERSION:1.1.6-SNAPSHOT
RPC_ADDRESS:0.0.0.0
/127.0.0.1
LOAD:29859.0
STATUS:NORMAL,0
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RELEASE_VERSION:1.1.6-SNAPSHOT
RPC_ADDRESS:0.0.0.0
Does anyone know what is happening and how can I fix it?
Thank you so much in advance!!
initial_token is only checked at first startup, when it is written to a system table. Delete the system table files and restart.
Running Cassandra 1.0, I am shrinking a ring from 5 nodes down to 4. In order to do that I ran nodetool decommission on the node I want to remove, then stopped cassandra on that host and used nodetool move and nodetool cleanup to update the tokens on the remaining 4 nodes to rebalance the cluster.
My seed nodes are A and B. The node I removed is C.
That seemed to work fine for 6-7 days, but now one of my four nodes thinks the decommissioned node is still part of the ring.
Why did this happen, and what's the proper way to fully remove the decommissioned node from the ring?
Here's the output of nodetool ring on the one node that still thinks the decommissioned node is part of the ring:
Address DC Rack Status State Load Owns Token
127605887595351923798765477786913079296
xx.x.xxx.xx datacenter1 rack1 Up Normal 616.17 MB 25.00% 0
xx.xxx.xxx.xxx datacenter1 rack1 Up Normal 1.17 GB 25.00% 42535295865117307932921825928971026432
xx.xxx.xx.xxx datacenter1 rack1 Down Normal ? 9.08% 57981914123659253974350789668785134662
xx.xx.xx.xxx datacenter1 rack1 Up Normal 531.99 MB 15.92% 85070591730234615865843651857942052864
xx.xxx.xxx.xx datacenter1 rack1 Up Normal 659.92 MB 25.00% 127605887595351923798765477786913079296
Here's the output of nodetool ring on the other 3 nodes:
Address DC Rack Status State Load Owns Token
127605887595351923798765477786913079296
xx.x.xxx.xx datacenter1 rack1 Up Normal 616.17 MB 25.00% 0
xx.xxx.xxx.xxx datacenter1 rack1 Up Normal 1.17 GB 25.00% 42535295865117307932921825928971026432
xx.xx.xx.xxx datacenter1 rack1 Up Normal 531.99 MB 25.00% 85070591730234615865843651857942052864
xx.xxx.xxx.xx datacenter1 rack1 Up Normal 659.92 MB 25.00% 127605887595351923798765477786913079296
UPDATE:
I tried to remove the node using nodetool removetoken on node B, which is the one that still claims node C is in the ring. That command ran for 5 hours and didn't seem to do anything. The only change is that the state of Node C is "Leaving" now when I run nodetool ring on Node B.
I was able to remove the decommissioned node using nodetool removetoken, but I had to use the force option.
Here's the output of my commands:
iowalker:~$ nodetool -h `hostname` removetoken 57981914123659253974350789668785134662
<waited 5 hours, the node was still there>
iowalker:~$ nodetool -h `hostname` removetoken status
RemovalStatus: Removing token (57981914123659253974350789668785134662). Waiting for replication confirmation from [/xx.xxx.xxx.xx,/xx.x.xxx.xx,/xx.xx.xx.xxx].
iowalker:~$ nodetool -h `hostname` removetoken force
RemovalStatus: Removing token (57981914123659253974350789668785134662). Waiting for replication confirmation from [/xx.xxx.xxx.xx,/xx.x.xxx.xx,/xx.xx.xx.xxx].
iowalker:~$ nodetool -h `hostname` removetoken status
RemovalStatus: No token removals in process.
With Cassandra 2.0, you need to use sh nodetool decommission on node to be removed.
In your case, check if you have removed entries in cassandra-topology.properties