I've just started working with Cassandra (datastax), version 2.1.3 and cqlsh version 5.0.1.
Cassandra starts up fine and the cluster is operational instantly.
Cqlsh is not working (on any of the nodes) and emits the following error:
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111,
"Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
I have tried starting cqlsh up with the hosts own ip, other hosts ip, different ports yet the result remains the same - always connection refused.
You need to edit cassandra.yaml on the node you are trying to connect to and set the node ip address for rpc_address and listen_address and restart Cassandra.
rpc_address is the address on which Cassandra listens to the client calls.
listen_address is the address on which Cassandra listens to the other Cassandra nodes.
Try to change the rpc_address to point to the node's IP instead of 0.0.0.0 and specify the IP while connecting to the cqlsh, as if the IP is 10.0.1.34 and the rpc_port left to the default value 9160 then the following should work:
cqlsh 10.0.1.34 9160
Or:
cqlsh 10.0.1.34
Also make sure that start_rpc is set to true in /etc/cassandra/cassandra.yaml configuration file.
If you check the system.log file for cassandra in /var/log/cassandra, you will see that this problem occurs because the rpc server has not started.
By default, the start_rpc is set to false in the cassandra.yaml file. Set it to start_rpc: true and then try again.
From Cassandra 3.0 onwards, start_rpc is set to true by default. https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html
This is because of some pre-configurations in the cassandra. And to fix this, there are following commented lines in the cassandra-env.sh file:
#add this if you’re having trouble connecting:
#JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>
where you have to start fixing the error.
Open the terminal and start fixing using the following easy steps:
Step-1
use
sudo nano /etc/cassandra/cassandra-env.sh
this command and search for JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname= uncomment the line by simply removing # attached in the start of the line.
NOTE: If you have never opened this file to fix this error then you will find JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>
Step-2
Now replace <public name> with 127.0.0.1 or the server IP.
Step-3 Save the file and restart the cassandra using systemctl restart cassandra.service (if you haven't the server is alredy running). OR start the cassandra using systemctl start cassandra.service (if the server is not running).
Step-4 Check the status either by using sudo service cassandra status or systemctl status cassandra.service.
Note: Once check the system monitor whether cassandra is running there or not.
Try cqlsh now. It'll work.
Note: nano is editor you can use other editors you are comfortable with.
Look for native_transport_port in /etc/cassandra/cassandra.yaml
The default is 9842.
native_transport_port: 9842
For connecting to localhost with cqlsh, this port worked for me.
cqlsh 127.0.0.1 9842
It is a good idea to check the cassandra log if even the server is running. I was getting exactly the same message and unable to do anything with that and then I found out that there are errors in the log and the system is actually not working.
Silly, I know, but could happen...
Had same problem recently after downgrade from Cassandra 3.0 to Cassandra 2.2 on ArchLinux.
Unlike above solutions my problem wasn't in .cassandra, but version 3.0 left its configuration in /var/lib/cassandra directory.
Following commands solved my problem:
sudo rm -R /var/lib/cassandra
sudo rm -R /var/log/cassandra
sudo rm -R /usr/share/cassandra
Then i installed cassandra and everything worked again :)
Try to change the rpc_address to point to the node's IP instead of 0.0.0.0 and specify the IP while connecting to the cqlsh, as if the IP is 10.0.2.64 and the rpc_port left to the default value 9160 then the following should work:
cqlsh 10.0.2.64 9160
OR
cqlsh 10.0.2.64
Also make sure that start_rpc is set to true in /etc/cassandra/cassandra.yaml configuration file.
The first and foremost step to diagnose is to check the Cassandra logs.
Find out the process and there will be hints in the command line parameters to tell you where the logs files are.
ps aux | grep cassandra
In my case, it was in default location /etc/cassandra/conf/cassandra.yaml (if you installed the yum package).
Make sure three things are configured and configured the same host/ip.
listen_address - you may leave it blank it will default to hostname/first IP binding with eth0 network card
rpc_address - this is optional, leave blank if you want it to be same as listen_address
seeds - this is a double quoted string of comma-separated list of IP addresses or hostnames. This was default to a hardcoded "127.0.0.1", changing it to the same as list_address and restart of Cassandra service made it work for me.
References:
https://stackoverflow.com/a/42507530/416394
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__seed_provider
I was experiencing the same issue with Cassandra 3.11.0, anytime I changed the address of rpc or listen address cqlsh wouldn't work. I had to add the same local ip to seeds
So after much trial and error my working cassandra.yml ended up like this:
class-name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
-seeds: "192.168.0.30"
listen_adress: 192.168.0.30
rpc_address: 192.168.0.30
For me it turned out that the service wasn't running at all.
Check with
service cassandra status
If you got the same error as I got, or another type, then messing around with IP addresses won't solve your problem at all.
The error I got:
cassandra dead but pid file exists
Edit: This was the solution for my problem: https://stackoverflow.com/a/46743119/3881406
try changing the native_transport_protocol to port 9160 (if it is set to anything other than 9160; it might be pointing to 9042). Check your logs and see on which port cassandra is listening for CQL clients?
Got into this issue for [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4] had to set start_native_transport: true in cassandra.yaml file.
For verification,
try opening tailf /var/log/cassandra/system.log file in one-tab
update cassandra.yaml
restart cassandra sudo service cassandra restart
In logfile is shows.
INFO [main] 2019-03-15 19:53:06,156 Server.java:156 - Starting listening for CQL clients on /10.139.45.34:9042 (unencrypted)...
I was trying to run it on a network drive. So I increased the client_timeout in cqlsh and it worked.
Make sure that you also set "seeds" to the address which you provided at "listen_address"
Try to telnet to the given address. In my case, there was a firewall in place blocking me.
When I installed Cassandra 3.11.1, I came across this problem. I checked the /var/log/cassandra/cassandra.log and found this error
Exception encountered during startup....It is a bug and already reported. The original post link https://issues.apache.org/jira/browse/CASSANDRA-14173.
The solution is to downgrade Cassandra to 3.0
download Cassandra rpm
curl -O
https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm
or
wget
https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm
rpm -ivh cassandra-3.0.15-1.noarch.rpm
service cassandra start
service cassandra status # check cassandra status
cassandra (pid 2322) is running...
cqlsh # start cassandra
cqlsh --cqlversion="3.4.0"
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.9 | CQL spec 3.4.0 | Native protocol v4]
try the above command. It works for me.
First, you need to check the port 9042 whether this open or not if open then check below things like rpc_address and listen_address. you should set the server own IP address instead 0.0.0.0 or 127.0.0.1.
If all okay then please run below command to connect cqlsh.
cqlsh IP address 9042
If you enable authentication and authorization you have to put user/pass with this command.
Check for correct IP address in the cassandra.yaml file. Majority of times the error is due to incorrect IP address of your system also the username and password too.
After doing so initiate cqlsh by the command :-
cqlsh 10.31.79.1 -u cassandra -p cassandra
check what the CASSANDRA_HOME environment variable points to. I had a old version of cassandra to which it was set. I had reset to the 3.11.6 installation that I upgraded and it started working.
Another case that wasn't mentioned for receiving the Unable to connect to any servers error is were the Client-to-Node Encryption is enabled in the client_encryption_options block in the cassandra.yaml file.
In that case you need to add the --ssl flag in order to connect:
cqlsh --ssl.
Related
I am facing some issues trying to connect to Cassandra 3.3.0 on Centos 7 remotely from another machine in the network. It is returning a "timed out" error while trying to connect via
$ cqlsh 192.168.1.56 9042
Same with:
$ cqlsh 192.168.1.56
In my point of view, I have configured the cassandra.yaml file correctly as it worked fine when I tried to connect to a locally running VM (Ubuntu) with an active Cassandra instance.
When I am checking the IPs and ports which have been bound via netstat -lnt, it is returning
tcp6 0 0 192.168.1.56:9042 :::* Listen
Actually the only one which is bound to tcp6. 192.168.1.56:9160 is bound to tcp.
Running on Ubuntu, it returns the same binding to tcp. So the fault is related to that point, I guess.
Some background facts:
I am running Cassandra as root. Both server and clients are running cqlsh 5.0.1, CQL spec 3.4.0 and Cassandra 3.3.0. Python 2.7.5.
The VM is accessible via 192.168.1.56 without problems. Firewall is deactivated and I even have tried to de- and reactivate IPv6. I have tried different combinations configuring the cassandra.yaml -- such as taking the interface instead of its IP directly, but without changes in the result.
All my research has not shown success so far. Has anybody an idea?!
Problem has been solved due to network misconfiguration.
I've changed the below configuration in /etc/ssh/sshd_config
Port 22
AddressFamily inet
ListenAddress 0.0.0.0
ListenAddress ::
I've changed AddressFamily from "any" to "inet".
i am trying to run kairosdb and cassandra, but kairosdb shutsdown after i get the following error, i believe it is because kairosdb is not able to establish connection with cassandra. Cassandra seems to be running fine and i cannot understand why this error is popping:
18:33:08.463 [main] ERROR [HConnectionManager.java:71] - Could not start connection pool for host localhost(127.0.0.1):9160
Error injecting constructor, org.kairosdb.core.exception.DatastoreException: me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client. ...
Also, i noticed that kairos_cache ix not created which is mentioned in this link. I changed the permissions of the /tmp/ folder to user from root, still it is not working.
Open your cassandra.yaml file and do the following:
Check if you have the apache thrift rpc server enabled and if it is listening on the port Kairos is listening.
start_rpc: true
rpc_address: localhost
rpc_port: 9160
The message is because KairosDB cannot reach Cassandra.
Probably your cassandra DB is not listening on 127.0.0.1 (loopback).
Check your cassandra.yaml file, probably it is using the IP Adress of your network interface as listen_adress and not 127.0.0.1.
Cassandra only listens on one address, by default it is the local host name of IP.
Otrherwise you may check your port just in case, but the ListenAddress is often the source of this problem.
I had the same issue with a docker deployment of cassandra with KairosDB.
As #JVasques said in his answer, the parameter "start_rpc" is disabled (set to false) in the newest cassandra.yaml file as default.
If anyone needs a default/standard YAML configuration file, it is recommended to download the latest release or the version you are using. You can download it from the official cassandra package on the Apache Website: http://cassandra.apache.org/download/
It is located under conf/cassandra.yaml
Beware: Configuration files of older cassandra versions might not be compatible!
It worked with the following settings in Docker for me:
start_rpc: true
rpc_address: 0.0.0.0
rpc_port: 9160
I have a cassandra node at a machine. When I access cqlsh from the same machne it works properly.
But when I tried to connect to it's cqlsh using "192.x.x.x" from another machine, I'm getting an error saying
Connection error: ('Unable to connect to any servers', {'192.x.x.x': error(111, "Tried connecting to [('192.x.x.x', 9042)]. Last error: Connection refused")})
What is the reason for this? How can I fix it?
Probably the remote Cassandra node is not bound to the external network interface but to the loopback one (this is the default configuration). You can ensure this by using "telnet thecassandrahost 9042" from the remote machine, it should not work.
In order to bind Cassandra to the external network interface you need to edit the cassandra.yaml configuration file and set the properties "listen_address" and "rpc_address" to your remote IP or "0.0.0.0" (not all versions of Cassandra support wildcard addresses).
Check also that the firewall is properly configured or disabled (sudo service iptables stop).
Set the config parameter where this file is located. Possibly /etc/cassandra/cassandra.yaml.
cassandra.yaml
listen_address: 192.x.x.x
rpc_address: 192.x.x.x
Then, restart the service.
1.Update：./conf/cassandra.yaml
rpc_address: 0.0.0.0 ("0.0.0.0" allow anywhere IP,but you can appoint an IP)
\# broadcast_rpc_address: 1.2.3.4 (Delete comment if rpc_address=0.0.0.0)
2.restart
./bin/cassandra
Case: I met a problem that I can't remote access cassandra When I using java access cassandra
I got the same issue. And I am following answer in this post. Unfortunately, I have no luck to make it work. I did some research. And it works now. Here is my change.
Environment
Ubuntu Server 16.04.3 LTS on VirtualBox,
DSE version 5.1
Install DSE
I am installing DSE follow this page
https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/install/installGUIdse.html
Go to /etc/dse/cassandra/cassandra.yaml
Change 'seeds' from 127.0.0.1 to {seriver ip} exp. mine is 172.20.10.9
listen_address
In DSE 5.1 official document says 'Never specify 0.0.0.0; it is always wrong.'
What I did is comment out 'listen_address' setting
'rpc_address' change from localhost to 0.0.0.0
'broadcast_address' change to server IP. Mine is 172.20.10.9
'broadcast_rpc_address' change to server ip
restart dse wait for couple minute,it is changing. If still not work, restart machine.
This is my log in 15 seconds.
`ubuntu08#ubuntu08:~$ nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
ubuntu08#ubuntu08:~$ nodetool status
Error: The node does not have system_traces yet, probably still bootstrapping
ubuntu08#ubuntu08:~$ nodetool status
Datacenter: SearchGraphAnalytics
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.0.44 278.51 KiB 32 ? 19db0016-df63-4470-9921-f3b5fe4e9341 rack1
`
I can access by run 'cqlsh 172.20.10.9' from local or another machine.
This is another document for Cassandra https://docs.datastax.com/en/developer/java-driver/3.3/manual/address_resolution/
I had the same issue, I was not allowed to listen to 0.0.0.0 and Cassandra was running on a VM with Bridged network. The solution I found was to let the VM SSH to itself, port forwarding the port on the bridged network interface to localhost:
ssh -L 192.168.x.x:9042:127.0.0.1:9042 myvmuser#localhost
Since the IP of the bridged network card would change (depending on which developers machine it was run) the ssh-command had to first get the IP, this snipped worked for that:
ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'
Also, as this should happen on boot you have to create a SSH key for the vm-user and trust it via .ssh/authorized_keys.
Even after I set the RPC_ADDRESS, it didn't work for me , until I set the -e CASSANDRA_START_RPC=true option.
It was always set to false in my case. I have tried this with Ubuntu, Docker and Cassandra.
Set the following config parameter in cassandra.yaml file (For CentOS it is located in /etc/cassandra/default.conf)
rpc_address: 0.0.0.0
Verify that following values are same as below(usually they are default)
start_native_transport:true
native_transport_port:9042
Last step for CentOS , Update the firewall configuration and allow port 9042 through for incoming connections
Access the firewall from “System / Administration / Firewall” in the CentOS menu
Add the port under “Other Ports”
I edit cassandra.yaml set listen_address and roc_address as ip address resolve the problem.
I set rpc_port to the public IP address, and now I can connect to Cassandra just fine from an outside server.
However, I cannot connect from the Cassandra server itself, using cqlsh
I am getting an error.Thar are:
Connection error: Could not connect to localhost:9160
Is there a configuration, I can change to be able to connect from the server itself ?
Check the cassandra.yaml file on the line with rpc_port : normally by default = 9160
Login to the machine hosting cassandra
Try cqlsh 127.0.0.1 9160 : should be ok in any case
try cqlsh [IP of host] 9160 : if ok then cassandra reachable from lan, if ok then cassandra is not reachable from lan but only localhost 127.0.0.1
You need to connect to cassandra through the rpc_address defined in cassandra.yaml. For example, I use cqlsh 10.0.80.49 9160.
Consider changing /etc/cassandra.yaml:
# Whether to start the thrift rpc server.
start_rpc: false
to
start_rpc: true
if you see this in cassandra logs:
INFO [main] 2015-07-21 12:06:27,426 CassandraDaemon.java:406 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
then just open a terminal and
$ nodetool enablethrift
as written in the INFO message. Should now work. Got this when my system upgraded to cassandra 2.2.0
If you are on OSX
brew install cassandra
First start the Cassandra
cassandra
Connect via CQL shell
cqlsh 127.0.0.1
Maybe start cassandra on your local machine by bin/cassandra -f first?
It's not listening on 127.0.0.1 since you told it to only listen on <public IP>. Make the listen address 0.0.0.0 to listen on all addresses (or just omit it if possible as this is usually the default). See Listening Sockets .NET tutorial or any other socket tutorial to get a basic understanding of socket binding.
Update (#c45556037):
Note that listen_address is the one for other nodes to use to connect to this one (a misleading name). rpc_address is the actual address to locally bind to. It's unclear from the 2.0 docs and is explained better in the 1.0 docs.
I faced same situation while starting cqlsh . I got following error while starting cassandra
Enter only ----cassandra----- in terminal.
it will show all jars and log files . if terminal hang , just escape from it and then enter cqlsh. then it will enter to cassandra cli.
This worked for me
for 2.0.5 the following works for me ..
$CASSANDRA_HOME/bin/cqlsh xx.xx.xxx.xxx 9160
Exception connecting to localhost/9160. reason: connection refused
Connection refused to cassandra cli mode .
goto the root directory of cassandra :
bin/cassandra -- host {host-ip} --port {9160}
if you are having trouble with this , check your {cassandra-root-directory}/conf/cassandra.yaml
the thrift ip or rpc_address is the address used as the host-ip for connecting to cli .
make it your local IP and if you are having trouble connecting using the port 9160 , try changing the rpc_port to 8070 and now try connecting to cassandra-cli mode using the command
bin/cassandra --host {local-IP} --port 8070
This worked for me, hope it works for you too .
Please execute the below query to resolve the issue
#!/bin/bash
export CASSANDRA_HOME=/opt/apache-cassandra-2.1.8
export CQLSH_HOST=192.168.1.200
export CQLSH_PORT=9042
echo $#
$CASSANDRA_HOME/bin/cqlsh $#'
Make sure to change the IP and location of cassandra Home Directory
Use this command
sudo service cassandra start
to start cassandra and then use
cqlsh command
Consider changing /etc/cassandra-env.sh:
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"
not "true" but "false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
sudo service cassandra restart
How to change default port 9042 in Cassandra ?
I resolved issue using below steps :
1) Stop cassandara services
sudo su -
systemctl stop datastax-agent
systemctl stop opscenterd
systemctl stop app-dse
2) Take backup and Change port from 9042 to 9035
cp /opt/dse/resources/cassandra/conf/cassandra.yaml /opt/dse/resources/cassandra/conf/bkp_cassandra.yaml
vi /opt/dse/resources/cassandra/conf/cassandra.yaml
native_transport_port: 9035
3) Start Cassandra services
systemctl start datastax-agent
systemctl start opscenterd
systemctl start app-dse
4) create cqlshrc file.
vi /root/.cassandra/cqlshrc
[connection]
hostname = 198.168.1.100
port = 9035
OS: CentOS 6.4 (server)
I have successfully installed (yum install dsc20) the cassandra database layer in my server and can connect to it using the CQL SHel (cqlsh). But I need to run queries remotely using the DataStax's DevCenter software. So I installed it (DevCenter) in a separate workstation. (CentOS 6.4 - desktop) and tried to add a new connection in order to connect to the cassandra db.
So I gave the IP of the "CentOS 6.4 - server" (in which cassandra database is running) for the host and the port as 9160. But when testing the connection it fails.
I also tried to turn off the firewall in the server, (/etc/init.d/iptables stop). But was no luck.
I'm sure this may be due to some misconfiguration which I cannot figure out. I'll be grateful, if someone can give me a solution for this as I was researching for this and found no answers. Thank you very much in advance.
DataStax DevCenter being built on top of the DataStax Java driver has the same connectivity requirements as the driver. Namely:
start_native_transport: true
rpc_address: IP or hostname reachable from the client
Machines in the cluster accept connections on port 9042 (the native_transport_port config option)
If rpc_address is set to either a private IP or to 0.0.0.0, DevCenter will not know what node to connect to.
If your cluster has multiple nodes and these are using rpc_address: 0.0.0.0, even if you configure DevCenter with the IP(s) of a couple of nodes, it will still have issues discovering the other nodes in the cluster.
In cassandra.yaml there's a comment/warning about using rpc_address: 0.0.0.0:
Note that unlike ListenAddress above, it is allowed to specify 0.0.0.0
here if you want to listen on all interfaces, but that will break clients
that rely on node auto-discovery.
Note: it is possible that in the future DevCenter might be able to ignore/filter out nodes in the cluster configured with rpc_address: 0.0.0.0.
Try this.. It worked for me!
Run the following command on your machine where you have DevCenter installed
$> ssh -L 9042:localhost:9042 <username>#<ip-address-cassandra-db> -N
After running this command, it will prompt for password
Now in your DevCenter > Contact Hosts, provide "127.0.0.1" and test connection. Bingo, you see the magic...
In the cassandra.yaml, I set
listen_address: localhost
rpc_address: 1.2.3.4
broadcast_rpc_address: 1.2.3.4
After doing a
sudo service cassandra restart
DevCenter was able to successfully connect.
After this change, cqlsh would not start without the address:
cqlsh 1.2.3.4
You can set the environment variable $CQLSH_HOST=1.2.3.4. Then simply type cqlsh.
If there's somebody working with docker locally, it can run this command.
docker run --name cassandra -p 127.0.0.1:9042:9042 -p 127.0.0.1:9160:9160
-e CASSANDRA_LISTEN_ADDRESS=localhost
-e CASSANDRA_BROADCAST_ADDRESS=localhost
-e CASSANDRA_RPC_ADDRESS=localhost -d cassandra