I am trying to setup a multi-node multi-datacenter cluster in Cassandra 3.11
For data-center 1 I have Cassandra running on 3 nodes(eg. 10.90.22.11, 10.90.22.12 and 10.90.22.13) and for data-center 2 I have Cassandra running on 2 nodes(eg. 10.90.22.21 and 10.90.22.22).
The ring is up but they are working separately. To make them work together I update the endpoint_snitch to be GossipingPropertyFileSnitch and also the dc and rac in cassandra-rackdc.properties to be DC1 and DC2 for respective nodes following the steps mentioned in this link.
After these changes when I restart Cassandra, the status of Cassandra is running however when I check for the ring with nodetool status I receive a error:
nodetool: Failed to connect to '127.0.0.1:7199'
ConnectException: 'Connection refused (Connection refused)'
What am I missing?
This error you posted indicates that nodetool couldn't connect to JMX that is supposed to be listening on port 7199:
Failed to connect to '127.0.0.1:7199'
Verify that Cassandra is running and check that the process is bound to various ports including 7199, 9042 and 7000. You can try running one of these commands:
$ netstat -tnlp
$ sudo lsof -nPi | grep LISTEN | grep java
Cheers!
You should try nodetool command with host/IP what you have put in your cassandra.yaml. Also, you should check your port 7199 or custom port if you set is open/allow from firewall.
nodetool -h hostname/ip status.
you can mention username.password if you enabled. please refer below link for more details and understanding:-
http://cassandra.apache.org/doc/latest/tools/nodetool/status.html
Related
I am trying to run Cassandra on my machine. Tying cqlsh gives me this error,
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': error(61, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
Running cassandra -f hangs.
I installed cassandra using brew install cassandra.
Also I have done all of the followings:
Killed docker instance
Uninstall and reinstall Cassandra (deleted all the Cassandra data and filles from my machine)
Restarted my machine
Purged docker data
I don't know why still I am running into this. Running brew services list shows:
Name Status User Plist
cassandra started aanish /Users/aanish/Library/LaunchAgents/homebrew.mxcl.cassandra.plist
mysql stopped
mysql#5.7 started aanish /Users/aanish/Library/LaunchAgents/homebrew.mxcl.mysql#5.7.plist
postgresql stopped
The cqlsh connection error indicates that localhost is not listening on port 9042 (Cassandra's CQL client port). The most likely reason is that the Cassandra process is not running.
You can confirm whether the Java process is listening on port 9042 if you run the following command in a terminal:
$ sudo lsof -nPi -sTCP:LISTEN
You will need to review the system.log (typically in /var/log/cassandra) to figure out what is going on with Cassandra.
Note that if you set rpc_address in cassandra.yaml to the IP of your mac, you need to specify that address when you connect with cqlsh. For example:
$ cqlsh 10.1.2.3
When ever you get a Cassandra cqlsh Connection error as follows:
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
You can also connect without using the IP address - just use the hostname:
cqlsh ‘hostname -I’
Another solution, type cqlsh <listen_address> [<port>] if it is not set to 127.0.0.1 or localhost
I was having the same issue with Cassandra 3.11.0, anytime I changed the address of rpc or listen address cqlsh wouldn't work. I had to add the same local ip to seeds
So after trial and error my cassandra.yml looked like:
class-name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
-seeds: "192.168.0.30"
listen_adress: 192.168.0.30
rpc_address: 192.168.0.30
then running
cqlsh 192.168.0.30 9042
When I installed Cassandra 3.11.1, I came across this problem.
I also found if I ran
service cassandra status
, there is a
cassandra dead but pid file exists
problem. It indicates that the Cassandra service does not start
I checked the
/var/log/cassandra/cassandra.log
and found this error:
Exception encountered during startup...
.It is a bug and already reported. The original post link https://issues.apache.org/jira/browse/CASSANDRA-14173.
The solution is to downgrade Cassandra to 3.0
download Cassandra rpm
curl -O
https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm
or
wget
https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm
rpm -ivh cassandra-3.0.15-1.noarch.rpm
service cassandra start
service cassandra status # check cassandra status, it should say
cassandra (pid 1234) is running...
cqlsh #start cassandra
Hope this helps you
I am trying to configure Cassandra Datastax Community Edition for remote connection on windows,
Cassandra Server is installed on a Windows 7 PC, With the local CQLSH it connects perfectly to the local server.
But when i try to connect with CQLSH from another PC in the same Network, i get this error message:
Connection error: ('Unable to connect to any servers', {'MYHOST':
error(10061, "Tried connecting to [('HOST_IP', 9042)]. Last error: No
connection could be made because the target machine actively refused
it")})
So i am wondering how to configure correctly (what changes should i make on cassandra.yaml config file) the Cassandra server to allow remote connections.
Thank you in advance!
How about this:
Make these changes in the cassandra.yaml config file:
start_rpc: true
rpc_address: 0.0.0.0
broadcast_rpc_address: [node-ip]
listen_address: [node-ip]
seed_provider:
- class_name: ...
- seeds: "[node-ip]"
reference: https://gist.github.com/andykuszyk/7644f334586e8ce29eaf8b93ec6418c4
Remote access to Cassandra is via its thrift port for Cassandra 2.0. In Cassandra 2.0.x, the default cqlsh listen port is 9160 which is defined in cassandra.yaml by the rpc_port parameter. By default, Cassandra 2.0.x and earlier enables Thrift by configuring start_rpc to true in the cassandra.yaml file.
In Cassandra 2.1, the cqlsh utility uses the native protocol. In Cassandra 2.1, which uses the Datastax python driver, the default cqlsh listen port is 9042.
The cassandra node should be bound to the IP address of your server's network card - it shouldn't be 127.0.0.1 or localhost which is the loopback interface's IP, binding to this will prevent direct remote access. To configure the bound address, use the rpc_address parameter in cassandra.yaml. Setting this to 0.0.0.0 will listen on all network interfaces.
Have you checked that the remote machine can connect to the Cassandra node? Is there a firewall between the machines? You can try these steps to test this out:
1) Ensure you can connect to that IP from the server you are on:
$ ssh user#xxx.xxx.xx.xx
2) Check the node's status and also confirm it shows the same IP:
$nodetool status
3) Run the command to connect with the IP (only specify the port if you are not using the default):
$ cqlsh xxx.xxx.xx.xx
Alternate solution to Kat. Worked with Ubuntu 16.04
ssh into server server_user#**.**.**.**
Stop cassandra if running:
Check if running with ps aux | grep cassandra
If running, will output a large block of commands / flags, e.g.
ubuntu 14018 4.6 70.1 2335692 712080 pts/2 Sl+ 04:15 0:11 java -Xloggc:./../logs/gc.log ........
Note 14018 in the example is the process id
Stop with kill <process_id> (in this case 14018)
edit cassandra.yaml file to be the following
rpc_address: 0.0.0.0
broadcast_rpc_address: **.**.**.** <- your server's IP (cannot be set to 0.0.0.0)
Restart cassandra ./bin/cassandra -f (from within cassandra root)
Open another terminal on local machine & connect via cqlsh **.**.**.** (your server's IP) to test.
The ./bin/nodetool status address reported my localhost IP (127.0.0.1), but cqlsh remotely still worked despite that.
I am running a 6 node cluster of cassandra 1.2 on an Amazon Web Service VPC with Oracle's 64-bit JVM version 1.7.0_10.
When I'm logged on to one of the nodes (ex. 10.0.12.200) I can run nodetool -h 10.0.12.200 status just fine.
However, if I try to use another ip address in the cluster (10.0.32.153) from that same terminal I get Failed to connect to '10.0.32.153:7199: Connection refused'.
On the 10.0.32.153 node I am trying to connect to I've made the following checks.
From 10.0.12.200 I can run telnet 10.0.32.153 7199 and I get a connection, so it doesn't appear to be a security group/firewall issue to port 7199.
On 10.0.32.153 if I run netstat -ant|grep 7199 I see
tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN
so cassandra does appear to be listening on the port
The cassandra-env.sh file on 10.0.32.153 has all of the JVM_OPTS for jmx active
-Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
The only shot in the dark I've seen while trying to solve this problem while searching the interwebs is to set the following:
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.0.32.153"
But when I do this I don't even get a response. It just hangs.
Any guidance would be greatly appreciated.
The issue did end up being a firewall/security group issue. While it is true that the jmx port 7199 is used, apparently other ports are used randomly for rmi. Cassandra port usage - how are the ports used?
So the solution is to open up the firewalls then configure the cassandra-env.sh to include
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<ip>
Im trying to use nodetool to check the status of my cluster, but its unable to connect.
My cassandra.yaml is configured with listen_address and rpc_address set as the server IP (e.g. 10.10.10.266).
Im able to connect through cqlsh and cassandra-cli using the same IP, but when I connect to nodetool it doesnt work.
/bin$ nodetool -h 10.10.10.266 ring
Failed to connect to '10.10.10.266:7199': Connection has timed out
I dont think I have a firewall enabled on the server (Ubuntu). Im running this directly on the server in question, so I wouldnt have thought it would be a firewall issue anyway.
You probably need to uncomment the following parameter in cassandra-env.sh:
-Djava.rmi.server.hostname=<public name>
Replace with the address of the interface you want the jmx interface to listen on.
nodetool connects through JMX interface. By default it's listening on port 7199 (other tools use RPC interface listening on port 9160 by default). Check JMX settings in cassandra-env.sh file. Most likely JMX server is listening on wrong interface (or probably loopback interface).
Default JMX configuration section (cassandra ver. 1.1.5) contains link to troubleshooting guide:
# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
# JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"
#
# see
# https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole
# for more on configuring JMX through firewalls, etc. (Short version:
# get it working with no firewall first.)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
It also worths to list all network interfaces using ifconfig and try telnet'ing port 7199 on all interfaces.
I was facing the same timeout issue. However I found that my cluster was not getting started properly because of token issue and I was getting "Host ID collision between active endpoint". Once i deleted data directory and restarted cluster then nodetool started working fine.
I also saw this same issue but it turned out to be some weirdness in my hosts file that was preventing JMX from binding to the interfaces.
Specifically, the host file had an entry for the external IP address with the hostname. Our servers had two interfaces, one external and one for an internal network. Removing that hosts entry did the trick.
As someone mentioned, it connects to the JMX port.
You can find the JMX port:
In /etc/cassandra/cassandra-env.sh. This won't work for ccm based local clusters OR
(my fav) by looking at the command-line of Cassandra node process running on the node.
My case was a cluster created locally using ccm so all my nodes were running on same host with different JMX port.
vagrant#triforce:~$ ps -eaf | grep cassandra | grepi -o " [^ ]*jmx.local.port[^ ]* "
-Dcassandra.jmx.local.port=7100
-Dcassandra.jmx.local.port=7300
-Dcassandra.jmx.local.port=7200
vagrant#triforce:~$
This is because I have 3 nodes running on the localhost.
vagrant#triforce:~$ nodetool -p 7100 ring
Datacenter: datacenter1
==========
Address Rack Status State Load Owns Token
3074457345618258602
127.0.0.1 rack1 Up Normal 64.65 MB 33.33% -9223372036854775808
127.0.0.2 rack1 Up Normal 65.26 MB 33.33% -3074457345618258603
127.0.0.3 rack1 Up Normal 65.92 MB 33.33% 3074457345618258602
vagrant#triforce:~$