I am having trouble with a 3-node Cassandra cluster on AWS.
There is one seed node and two data nodes. The nodes are crashing
when they are launched and when I am trying to start them manually.
The error message appears in all three nodes.
Cassandra's version is 2.0.9
I have tried the following settings:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "<seed.node.public.IP>"
rpc_address: <node.public.IP>
rpc_port: 9160
listen_address: (or with the node's public IP)
storage_port: 7000
endpoint_snitch: SimpleSnitch (and RackInferringSnitch as well).
The error message is
ERROR [main] 2014-09-29 08:59:45,241 CassandraDaemon.java (line 513) Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision (StorageService.java:446)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:657)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:504)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
Ports 7000, 7001, 7199, 8080, 9042, 9160, 61620 and 61621 are open within the Cluster's security group.
I have also read and tried the solutions given on the following links:
Cassandra Not Starting Up
Starting cassandra as a service does not work for 2.0.5, sudo cassandra -f works
Apache Cassandra: Unable to gossip with any seeds
Datastax Enterprise is crashing with Unable to gossip with any seeds error
https://github.com/Netflix/Priam/issues/313
Cassandra can not bind to the public IP address in EC2.
Replacing it with the public DNS or the private IP address
in listen_address, rpc_address and seeds.
The public DNS is resolving to the private IP address which is
the eth0 intherface on EC2 instances, where Cassandra is binding.
The working configuration is:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "<seed.node.public.DNS>"
rpc_address: <node.public.DNS>
rpc_port: 9160
listen_address: (or with the node's public DNS)
storage_port: 7000
endpoint_snitch: SimpleSnitch (and RackInferringSnitch as well).
Related
I installed Cassandra DB on my Ubuntu VM1 and also installed on other VM Ubuntu VM2.
MY question is:
How to initialize multi node cluster (single data-center) on my VM's?
I also changed my Cassandra.yaml like below:
IP default to IP system seed , listen_address.
cluster_name: 'MyDigitalOceanCluster'
initial_token: 0
seed_provider:
seeds: "198.211.50.0"
listen_address: 198.211.50.0
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
After the changes I received this message:
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
Connection refused")})
Aaron has pretty much given you the answers but for what it's worth set the following in cassandra.yaml:
num_tokens: 16 or if you prefer to specify tokens, make sure you pick tokens which are balanced
listen_address: private_ip
rpc_address: public_ip (or same IP if your VM only has 1 NIC)
set seeds to the IP of both nodes (1 of the nodes is sufficient but 2 is preferred)
endpoint_snitch: GossipingPropertyFileSnitch is recommended and almost always the right choice (see my reasons here https://community.datastax.com/questions/8887/)
When connecting with cqlsh, you need to specify the node's client IP (rpc_address) otherwise it will default to localhost (127.0.0.1). For example:
$ cqlsh 198.211.50.0
Cheers!
I have brand new install of Cassandra 3.0.9 on CentOS 7.4.1708.
I am trying to change from the default SimpleSnitch to GosspingPropertyFileSnitch.
When I try and follow the steps on the DatStax Website, it appears as though I should only have to change the endpoint_snitch setting in the cassandra.yaml file. When I do that and restart the cassandra service, I am no longer able to connect to cassandra with nodetool or cqlsh (Connection Refused). Changing the settign back to the SimpleSnitch yields no reversal and ability to connect. I am confused as to what settings I am missing that are causing this to happen. I'd like to know
What I am missing in trying to get it moved to the GossipingPropertyFileSnitch?
Why changing the setting back does not revert things to its previous state where I can connect using nodetool or cqlsh?
I have two nodes in the cluster, both of which I want to act as seeds.
cassandra.yaml: - Node 1
cluster_name: '<My Cluster Name>'
- seeds: "<IP Add1>, <IP Add2>"
listen_address: <IP Add1>
rpc_address: <IP Add1>
endpoint_snitch: GossipingPropertyFileSnitch
#broadcast_address: 1.2.3.4
cassandra.yaml: - Node 2
cluster_name: '<My Cluster Name>'
- seeds: "<IP Add1>, <IP Add2>"
listen_address: <IP Add2>
rpc_address: <IP Add2>
endpoint_snitch: GossipingPropertyFileSnitch
#broadcast_address: 1.2.3.4
cassandra-rackdc.properties - Node 1:
dc=<DC1 Name>
rack=rack1
prefer_local=true
cassandra-rackdc.properties - Node 2:
dc=<DC1 Name>
rack=rack2
prefer_local=true
cassandra-topology.properties - Both Nodes:
<IP Add1>=<DC1 Name>:RAC1
<IP Add2>=<DC1 Name>:RAC2
In CQLSH - Both Nodes
UPDATE system.local SET cluster_name = '<My Cluster Name>' where key='local';
I installed Data Stax 3.7 on my Windows machine(IP:10.175.12.249) and made following changes in my cassandra.yaml file:
cluster_name: 'Test_cluster'
listen_address: "10.175.12.249"
start_rpc: true
rpc_address: "0.0.0.0"
broadcast_rpc_address: "10.175.12.249"
seeds: "10.175.12.249"
endpoint_snitch: SimpleSnitch
Now, I started the service and cassandra is running fine on seed node.
I tried adding another node to my cluster. So I installed Data Stax 3.7 on another Windows machine(IP:192.168.158.78) and made following changes in cassandra.yaml file:
cluster_name: 'Test_cluster'
listen_address: "192.168.158.78"
start_rpc: true
rpc_address: "0.0.0.0"
broadcast_rpc_address: "192.168.158.78"
seeds: "10.175.12.249"
endpoint_snitch: SimpleSnitch
Now when I started the cassandra service on my 2nd machine, I am getting the following error:
INFO 09:41:27 Cassandra version: 3.7.0
INFO 09:41:27 Thrift API version: 20.1.0
INFO 09:41:27 CQL supported versions: 3.4.2 (default: 3.4.2)
INFO 09:41:27 Initializing index summary manager with a memory pool size of 100 MB and a resize interval of 60 minutes
INFO 09:41:27 Starting Messaging Service on /192.168.158.78:7000 (Intel(R) Centrino(R) Advanced-N 6235)
INFO 09:41:27 Scheduling approximate time-check task with a precision of 10 milliseconds
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any seeds
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1386)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:561)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:855)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714)
ERROR 09:41:58 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1386) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:561) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:855) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370) [apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.0.jar:3.7.0]
WARN 09:41:58 No local state or state is in silent shutdown, not announcing shutdown
INFO 09:41:58 Waiting for messaging service to quiesce
Below is the output of nodetool status on seed node(IP:10.175.12.249):
C:\Program Files\DataStax-DDC\apache-cassandra\bin>nodetool status
Datacenter: datacenter1
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
DN 192.168.158.78 ? 256 68.1% 6bc4e927-3def-4dfc-b5e7-31f5882ce475 rack1
UN 10.175.12.249 257.76 KiB 256 65.7% 300d731e-a27c-4922-aacc-6d42e8e49151 rack1
Thanks!!!
The - seeds: in conf/cassandra.yaml should have the same value (same IP or the hostname) as listen_address: in the same conf file.
I came across this error when the IP addresses were not matching. Try keeping the same and restart the cluster. Hope this helps...
I am using mac osx. i created 3 virtual box by virtualbox. I've installed centos7 minimal version on each of the virtual box.
Then i installed cassandra on each of the box. After installation it was starting by cqlsh and nodetool status command.
But after then when i was trying to link each other and edit cassandra.yaml file its started showing
('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
i've edited the cassandra.yaml file as follows:
cluster_name: 'Home Cluster'
num_tokens: 256
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
- seeds: "192.168.56.102,192.168.56.103"
storage_port: 7000
listen_address: 192.168.56.102
rpc_address: 192.168.56.102
rpc_port: 9160
endpoint_snitch: SimpleSnitch
my /etc/hosts file contains:
192.168.56.102 node01
192.168.56.103 node02
192.168.56.104 node03
Please tell me whats wrong i'm doing? My cassandra cluster not working.
solution: I got the solution from AKKI. The problem was enpoint_snitch. I made the endpoint_snitch=GossipingPropertyFileSnitch and it fixed. My now output is as follows:
[root#dbnode2 ~]# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.56.101 107.38 KB 256 62.5% 0526a2e1-e6ce-4bb4-abeb-b9e33f72510a rack1
UN 192.168.56.102 106.85 KB 256 73.0% 0b7b76c2-27e8-490f-8274-571d00e60c20 rack1
UN 192.168.56.103 83.1 KB 256 64.5% 6c8d80ec-adbb-4be1-b255-f7a0b63e95c2 rack1
I had faced similar problem,
I tried the following solution:
In Cassandra.yaml file check if you have,
start_rpc = true
Changed my endpoint snitch to
endpoint_snitch: GossipingFilePropertySnitch
Opened all ports Cassandra uses on my CentOS
Cassandra inter-node ports
Port number Description
7000 Cassandra inter-node cluster communication.
7001 Cassandra SSL inter-node cluster communication.
7199 Cassandra JMX monitoring port.
Cassandra client port
Port number Description
9042 Cassandra client port.
9160 Cassandra client port (Thrift).
Command to open ports on CentOs 7(Find it according to your OS):
>sudo firewall-cmd --zone=public --add-port=9042/tcp --permanent
>sudo firewall-cmd –reload
Then Restart your systems
Also it seems that you are changing the Cassandra.Yaml file after starting cassandra.
Make sure you edit your Cassandra.yaml file on all nodes before starting Cassandra
Also remember to start the seed node first.
I have a scenario to run cassandra of two different versions in the same machine but at different ports.
I started one cluster with following cassandra config at port 9161,
# TCP port, for commands and data
storage_port: 7000
# SSL port, for encrypted communication. Unused unless enabled in
# encryption_options
ssl_storage_port: 7004
port for the CQL native transport to listen for clients on
native_transport_port: 9043
port for Thrift to listen for clients on
rpc_port: 9161
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "127.0.0.1"
It runs well,
$ /usr/local/apache-cassandra-2.1.1/bin/cassandra -f
...
INFO 05:08:42 Loading settings from file:/usr/local/apache-cassandra-2.1.1/conf/cassandra.yaml
...
INFO 05:09:29 Starting listening for CQL clients on localhost/127.0.0.1:9043...
INFO 05:09:29 Binding thrift service to localhost/127.0.0.1:9161
INFO 05:09:29 Listening for thrift clients...
INFO 05:19:25 No files to compact for user defined compaction
$ jps
5866 CassandraDaemon
8848 Jps
However while starting another cassandra cluster configured to run at port 9160 with config,
# TCP port, for commands and data
storage_port: 7000
# SSL port, for encrypted communication. Unused unless enabled in
# encryption_options
ssl_storage_port: 7004
port for the CQL native transport to listen for clients on
native_transport_port: 9042
port for Thrift to listen for clients on
rpc_port: 9160
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "127.0.0.1"
it fails with message
$ /usr/local/apache-cassandra-2.0.11/bin/cassandra -f
Unable to bind JMX, is Cassandra already running?
How can I make it run two different version of cassandra in the same machine?
The problem is that I have no permission to stop the previous version. Neither can I use https://github.com/pcmanus/ccm
The problem is your new version cassandra is also trying to use port 7199 for the JMX monitoring. Change the JMX port to some other unused port, then it will start. The JMX port can be change in the file called cassandraFolder/bin/cassandra.bat. There will be a line
-Dcom.sun.management.jmxremote.port=7199^
Change the above port to some other unused port.
If you are using cassandra in linux environment the JMX configuration will be located in the file called cassandraFolder/conf/cassandra-env.sh. There will be a line
JMX_PORT="7199"
Change this to some other unused port.
But I was unclear with your question.
Are you trying to run new cassandra to join in the existing cluster?
If yes, changing the JMX port will be sufficient.
Are you trying to run new cassandra in a stand alone mode?
If yes, change following configuration in yaml file.
seeds: "127.0.0.2"
listen_address: 127.0.0.2
rpc_address: 127.0.0.2
add the following entry,
127.0.0.1 127.0.0.2
in the /etc/hosts file if your you are running in linux. If you are running in windows add the above entry in C:\Windows\System32\drivers\etc\hosts file. If your intension is to run in stand alone mode then be careful in your configuration. If you made anything wrong then your new cassandra will join into the existing cluster.
This link helps you to run cassandra cluster in a single windows machine
Well I fixed it with changing few more configurations which are storage_port/ssl_storage_port in conf/cassandra.yaml and JMX_PORT in conf/cassandra-env.sh,
conf/cassandra.yaml
# TCP port, for commands and data
storage_port: 7004
# SSL port, for encrypted communication. Unused unless enabled in
# encryption_options
ssl_storage_port: 7005
port for the CQL native transport to listen for clients on
native_transport_port: 9043
port for Thrift to listen for clients on
rpc_port: 9161
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "127.0.0.1"
conf/cassandra-env.sh
# Specifies the default port over which Cassandra will be available for
# JMX connections.
JMX_PORT="7200"