I have a single node Cassandra cluster and I added a new node to that cluster. I made sure that the cluster name is same as well as the rack-dc properties are same. I also made sure that the nodes can communicate through all necessary ports. I can cqlsh from both nodes to each other as well as telnet all required ports 7000,7199,9042
cassandra.yaml for new node
1) seed: - "<old_node_ip>,<new_node_ip>"
2) auto_bootstrap: - true
3) listen_address: <new_node_ip>
4) rpc_address: <new_node_ip>
nodetool status command shows only the host node(itself) on both nodes
Running cassandra v 3.11.4 on both with open jdk 8
Am I missing anything?
Update
Here's the log:
java.lang.RuntimeException: Unable to gossip with any peers
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO [StorageServiceShutdownHook] 2019-10-21 13:08:14,218 HintsService.java:209 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2019-10-21 13:08:14,219 Gossiper.java:1559 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO [StorageServiceShutdownHook] 2019-10-21 13:08:14,219 MessagingService.java:981 - Waiting for messaging service to quiesce
INFO [ACCEPT-/10.128.1.3] 2019-10-21 13:08:14,220 MessagingService.java:1336 - MessagingService has terminated the accept() thread
INFO [StorageServiceShutdownHook] 2019-10-21 13:08:14,764 HintsService.java:209 - Paused hints dispatch
it says unable to gossip with peers but I can telnet on JMX port and the seed values are mentioned correct as well
There are three main things required for getting a new node to join/gossip with an existing node:
cluster_name - The cluster names of the two nodes must match exactly (case-sensitive).
seeds - The new node must use the existing node as a seed node. If you're working in a cloud environment, the "external" (aka "floating") IP addresses must be used.
node-to-node SSL (if used) - The nodes must have matching truststores to connect.
One thing is on my old_node the cassandra is using localhost (127.0.0.1) as shown in nodetool status.
2a. Neither of the nodes can use the home IP of 127.0.0.1.
I was getting this error when trying to add a new node to a single DC cluster. The reason behind my error was that my broadcast_address parameter in cassandra.yaml had a different IP than the listen_address. I set it to blank and restarted the server which fixed the issue.
# Address to broadcast to other Cassandra nodes
# Leaving this blank will set it to the same value as listen_address
broadcast_address:
Related
I have a 16-node Cassandra cluster (3.11.9) with 3 seed nodes (.54, .115 and 164), replication factor 3 and gc_grace_seconds 10 days(default). Some nodes show DN on some other nodes but on other nodes they show up as UN. For example below is the nodetool status from the .54 and the .115 nodes:
.54
and .115
while for example on .87 every node is UN. This is happening for at least a couple of weeks now, and it started from two nodes that were showing each other down, the .54 and .147. However, it seems it expanded right now more and more nodes show DN on some nodes(but not on all). Just to also add that there were no writes these weeks.
I have tried enabling, disabling the gossip and restarting cassandra on all nodes. Generation stamp is up to date in system_auth table. I can connect to these nodes with cqlsh but, as expected, in some cases I get NoHostAvailable because some data are located on the "dead" nodes.
nodetool describecluster shows the DN nodes to be Unreachable, depending on which node I am executing it. So i.e. the .54 shows the .164,.115,.147 and .19 as Unreachable.
Also in nodetool gossipinfo everything looks ok with status: normal and up-to-date generation.
In the debug.log file I only get:
DEBUG [MessagingService-Outgoing-/192.168.100.147-Gossip] 2022-05-30 03:58:43,478 OutboundTcpConnection.java:546 - Unable to connect to /192.168.100.147
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_312]
at sun.nio.ch.Net.connect(Net.java:482) ~[na:1.8.0_312]
at sun.nio.ch.Net.connect(Net.java:474) ~[na:1.8.0_312]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) ~[na:1.8.0_312]
at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.11.9.jar:3.11.9]
at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.11.9.jar:3.11.9]
at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:434) [apache-cassandra-3.11.9.jar:3.11.9]
at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-cassandra-3.11.9.jar:3.11.9]
In the system.log it actually has logged for all the nodes as "Node...has restarted, now UP" and "Node...state jump to Normal". However I also noticed this, which may has nothing to do:
WARN [GossipStage:1] 2022-05-30 07:22:06,164 Gossiper.java:1693 - \
Received an ack from /192.168.100.127, who isn't a seed.
Ensure your seed list includes a live node. Exiting shadow round
Is there any way to understand what is happening and why is this happening? Do I miss something?
Please let me know if you need any more information.
This looks like a classic networking issue to me where the nodes are unable to gossip with each other because there's no connectivity on the internode port (default is 7000). The debug message you posted clearly states the cause:
java.net.NoRouteToHostException: No route to host
You need to check that there are no firewalls like iptables or firewalld blocking the traffic on port 7000, otherwise the nodes can't talk to each other.
It is simple enough to test it using Linux tools such as telnet or nc. For example, run this command on node .54:
$ telnet 192.168.100.115 7000
If you get a "connection refused" error, it means that one of the following is true:
there's no network route to the node,
the traffic to the default gossip port 7000 is blocked, or
gossip is configured on another port (check storage_port in cassandra.yaml
But in my experience, the most likely cause is that traffic is blocked by a firewall. Cheers!
I try to build Cassandra cluster with 3 nodes
On node 1 i've got a trouble when i try use cqlsh:
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
In cassandra.yaml i dont't see strings with ip 127.0.0.1, i wrote there correct parametres (in my mind)
cluster_name: 'cassandra_cluster'
seeds: "node_public_ip_1,node_public_ip_2,node_public_ip_3"
listen_address: node_public_ip_1
rpc_adress: node_public_ip_1
in cassandra-topology.properties:
`# Cassandra Node IP=Data Center:Rack`
node_public_ip_1=dc1:rac1
node_public_ip_2=dc1:rac1
node_public_ip_3=dc1:rac1
What i do wrong?
Thanks!
When you run cqlsh without specifying the host, it defaults to localhost (127.0.0.1). You need to specify the client IP address (rpc_address) of the node you want to connect to when running cqlsh.
Also, the standard recommendation for multi-homed servers is to use the private IP for internal cluster comms and the public IP for client connections. This means that in cassandra.yaml you would configure:
listen_address: private_ip
rpc_address: public_ip
Since the seeds are used for seeding cluster communication, you would also configure the seeds list with the nodes' private IP addresses.
Finally, if you're using the old PropertyFileSnitch, you should also configure cassandra-topology.properties with the nodes' private IP addresses.
However, PFS is really old and in almost all cases our recommendation is to use GossipingPropertyFileSnitch since it has all the benefits of being able to expand the cluster in the future without any downsides as I've explained in this post -- https://community.datastax.com/questions/8887/.
It is also a lot simpler to manage GPFS since you only need to set a single node's DC and rack configuration in cassandra-rackdc.properties without having to reconfigure all the nodes whenever you add/remove nodes from the cluster.
When you do switch to GPFS, you should delete cassandra-topology.properties on the nodes to prevent any gossip issues in the future as I've explained in this post -- https://community.datastax.com/questions/4621/. Cheers!
Try run the "nodetool status" and "nodetool describecluster" on both nodes of this cluster to identify the IP used by nodes.
and try connect passing the host/ip. like:
cqlsh <host> <port> -u <user> -p <password>
If I change the port one of the cassandra node will it communicate to other node?
I am using cassandra 3.6 and 3 node of cassandra .
node1 and 2 having default port like storage_port=7000, rpc_port=9160 etc.
but I change the port for node3 given below.
storage_port: 7005
ssl_storage_port: 7006
native_transport_port: 9043
rpc_port: 9161
but when I connect to cassandra with node3, this error I am facing
Exception (java.lang.RuntimeException) encountered during startup:
Unable to gossip with any seeds
how to fixed this issue please help me out.
The ports on all nodes need to be the same.
How to enable cassandra port to connect with BI application. Here my setup with cassandra is of multiple nodes (192.xxx.xx.01,192.xxx.xx.02,192.xxx.xx.03). In this scenario which node will be acting like master / coordinator with my application.
Although i worked with listen_address, rpc_address, broadcast_rpc_address and seeds, I opened both tcp ports 9042 and 9160.
version: 3.10
Kindly, lead me to the right direction.
Cassandra uses master-less architecture.All nodes are equal in cassandra.
When you connect to one of the node that node act as co-ordinator node, any of the node can be co-ordinator.
The coordinator is selected by the driver based on the policy you have set. Common policies are DCAwareRoundRobinPolicy and TokenAware Policy.
For DCAwareRoundRobinPolicy, the driver selects the coordinator node based on its round robin policy. See more here: http://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/policies/DCAwareRoundRobinPolicy.html
For TokenAwarePolicy, it selects a coordinator node that has the data being queried - to reduce "hops" and latency. More info: http://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/policies/TokenAwarePolicy.html
native_transport_port is 9042 by default and clients use native transport by default.
Hence you should have connection from your BI to Cassandra host on port 9042.
I have created an IBM container with Cassandra v2.2.8 on Bluemix. When starting it, I get the following error:
TINFO 09:20:56 DiskAccessMode 'auto' determined to be
mmap, indexAccessMode is mmap
FINFO 09:20:56 Global memtable on-heap threshold is enabled at 1996MB
GINFO 09:20:56 Global memtable off-heap threshold is enabled at 1996MB
ŽException (org.apache.cassandra.exceptions.ConfigurationException)
encountered during startup: Unknown listen_address '172.32.0.4 172.32.0.4'
/Unknown listen_address '172.32.0.4 172.32.0.4'
dERROR 09:20:56 Exception encountered during startup:
Unknown listen_address '172.32.0.4 172.32.0.4'
I see that a similar issue is present in Cassandra v2.2:
https://github.com/docker-library/cassandra/pull/77
How do I implement the solution mentioned in the above link in a Bluemix environment?
Thanks for your time.
It looks like you will need to set an environment variable for CASSANDRA_LISTEN_ADDRESS and possibly CASSANDRA_BROADCAST_ADDRESS.
I'm not sure if you can get away with just setting 127.0.0.1 (I haven't used cassandra before), but if you can't, the best thing to do would be to request an IP address before deploying the container and then pass that IP address as the value for the listen address when you start the container. Then bind that container to the IP address.
Hope this helps.