I have installed cassandra and worked on it. It worked properly. Now, it is showing as-
localhost/<> is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
Fatal configuration error; unable to start server. See log for stacktrace.
INFO 09:17:02 Announcing shutdown
INFO 09:17:02 Compacted 4 sstables to [./../data/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-33,]. 6,485 bytes to 5,751 (~88% of original) in 223ms = 0.024595MB/s. 4 total partitions merged to 1. Partition merge counts were {4:1, }
INFO 09:17:04 Waiting for messaging service to quiesce
user#inblrlt-user:~/dev/Cassandra/apache-cassandra-2.1.7/bin$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': OperationTimedOut('errors=None, last_host=None',)})
How to change my server address so that the issue is cleared?
Your localhost is already in use. Follow the following steps-
$ jps
You see some processes running. For example:
9107 Jps
1112 CassandraDaemon
Then kill the CassandraDaemon process by the process id you see after executing jps. In my example, here process id 1112 for CassandraDaemon.
$ kill -9 1112
Then check processes again after a while-
$ jps
You will see CassandraDaemon will no longer be available.
9170 Jps
Then remove your saved_caches and commilog and start cassandra again.
If you want to change the listen_address from localhost to any private ip or public ip, you need to make the following changes:
change seeds: at cassandra.yaml
change listen_address: at cassandra.yaml
change rpc_address: at cassandra.yaml
set JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<place_your_ip_here>" at cassandra-env.sh
Related
I am trying to setup a multi-node multi-datacenter cluster in Cassandra 3.11
For data-center 1 I have Cassandra running on 3 nodes(eg. 10.90.22.11, 10.90.22.12 and 10.90.22.13) and for data-center 2 I have Cassandra running on 2 nodes(eg. 10.90.22.21 and 10.90.22.22).
The ring is up but they are working separately. To make them work together I update the endpoint_snitch to be GossipingPropertyFileSnitch and also the dc and rac in cassandra-rackdc.properties to be DC1 and DC2 for respective nodes following the steps mentioned in this link.
After these changes when I restart Cassandra, the status of Cassandra is running however when I check for the ring with nodetool status I receive a error:
nodetool: Failed to connect to '127.0.0.1:7199'
ConnectException: 'Connection refused (Connection refused)'
What am I missing?
This error you posted indicates that nodetool couldn't connect to JMX that is supposed to be listening on port 7199:
Failed to connect to '127.0.0.1:7199'
Verify that Cassandra is running and check that the process is bound to various ports including 7199, 9042 and 7000. You can try running one of these commands:
$ netstat -tnlp
$ sudo lsof -nPi | grep LISTEN | grep java
Cheers!
You should try nodetool command with host/IP what you have put in your cassandra.yaml. Also, you should check your port 7199 or custom port if you set is open/allow from firewall.
nodetool -h hostname/ip status.
you can mention username.password if you enabled. please refer below link for more details and understanding:-
http://cassandra.apache.org/doc/latest/tools/nodetool/status.html
I've running cassandra on my local machine.
I've starting it sudo service cassandra start. And then check logs under var/log/cassandra/system-log and it says:
INFO [main] 2019-07-28 13:13:17,226 Server.java:162 - Starting listening for CQL clients on localhost/127.0.0.1:9042 (unencrypted)...
INFO [main] 2019-07-28 13:13:17,270 CassandraDaemon.java:501 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO [SharedPool-Worker-1] 2019-07-28 13:13:27,133 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO [OptionalTasks:1] 2019-07-28 13:13:27,298 CassandraRoleManager.java:339 - Created default superuser role 'cassandra'
Then I try to connect with cqlsh in terminal and it says:
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
What is wrong? Also I couldn't see 9042 port with netstat -tulpn command.
Go to /etc/cassandra/cassandra-env.sh
Uncomment
# JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"
and change it to
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname==localhost"
Set listen_address and broadcast_rpc_address to local ip (get ip address from ifconfig).
Restart Cassandra.
cqlsh localhost 9042
This would work if you didn't change the cassandra.yml file.
I have two Ubuntu 16.04 nodes on which I installed Cassandra 3.11.3 with java version "1.8.0_181". I want to merge these two nodes into a Cassandra cluster. Their intern ips are 172.16.10.20 and 172.16.10.30.
on each /etc/cassandra/cassandra.yaml file I modified the following lines:
cluster_name: 'my_cluster'
- seeds: "172.16.10.20,172.16.10.30"
listen_address: XXXX
rpc_address: XXXX
where XXXX is respectively the intern ip of the current node.
I then restart Cassandra on each node
sudo service cassandra restart
and check that Cassandra runs :
sudo service cassandra status
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/init.d/cassandra; generated)
Active: active (running) since Wed 2018-08-08 00:31:42 UTC; 3s ago
Docs: man:systemd-sysv-generator(8)
and the cluster
nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.16.10.20 190.11 KiB 256 100.0% 84dded4c-c74e-45f4-9481-ff837fec229d rack1
UN 172.16.10.30 265.06 KiB 256 100.0% 4695fef4-70c7-46b2-a0bd-8b752fe5beb6 rack1
Everything is up and normal.
I want to connect now to Cassandra:
cqlsh
and get:
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
Some googling later, I want to start cassandra by hand
cassandra
and get (among a huge message):
ERROR [main] 2018-08-07 23:02:51,365 CassandraDaemon.java:708 - Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use (Bind failed)
It looks like the port 7199 is already in use. I kill the corresponding pid, change in /etc/cassandra/cassandra-env.sh the JMX_PORT to 7200... same issue, the port is said to be in use, plus the error
00:33:06,236 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[SYSTEMLOG] - openFile(/var/log/cassandra/system.log,true) call failed. java.io.FileNotFoundException: /var/log/cassandra/system.log (Permission denied)
I have changed the permission, but the error remains. At this point of the story I am running out of ideas. What I am trying to achieve seem pretty straighforward so I guess others must have run into a similar issue.
The nodetool status output says it all here. You had everything running just fine. So revert any changes in terms of port usage.
As your nodetool status reveals that your node IPs are 172.16.10.20 and 172.16.10.30, try running cqlsh and providing one of those IPs. cqlsh tries to connect to 127.0.0.1 by default, which will not work in a plural node cluster.
cqlsh 172.16.10.20 -u yourusername -p yourpassword
Note: You can omit -u and -p if you don't have auth enabled. But if that's true, then you should really change your cluster to enable auth.
I am running a 6 node cluster of cassandra 1.2 on an Amazon Web Service VPC with Oracle's 64-bit JVM version 1.7.0_10.
When I'm logged on to one of the nodes (ex. 10.0.12.200) I can run nodetool -h 10.0.12.200 status just fine.
However, if I try to use another ip address in the cluster (10.0.32.153) from that same terminal I get Failed to connect to '10.0.32.153:7199: Connection refused'.
On the 10.0.32.153 node I am trying to connect to I've made the following checks.
From 10.0.12.200 I can run telnet 10.0.32.153 7199 and I get a connection, so it doesn't appear to be a security group/firewall issue to port 7199.
On 10.0.32.153 if I run netstat -ant|grep 7199 I see
tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN
so cassandra does appear to be listening on the port
The cassandra-env.sh file on 10.0.32.153 has all of the JVM_OPTS for jmx active
-Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
The only shot in the dark I've seen while trying to solve this problem while searching the interwebs is to set the following:
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.0.32.153"
But when I do this I don't even get a response. It just hangs.
Any guidance would be greatly appreciated.
The issue did end up being a firewall/security group issue. While it is true that the jmx port 7199 is used, apparently other ports are used randomly for rmi. Cassandra port usage - how are the ports used?
So the solution is to open up the firewalls then configure the cassandra-env.sh to include
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<ip>
Im trying to use nodetool to check the status of my cluster, but its unable to connect.
My cassandra.yaml is configured with listen_address and rpc_address set as the server IP (e.g. 10.10.10.266).
Im able to connect through cqlsh and cassandra-cli using the same IP, but when I connect to nodetool it doesnt work.
/bin$ nodetool -h 10.10.10.266 ring
Failed to connect to '10.10.10.266:7199': Connection has timed out
I dont think I have a firewall enabled on the server (Ubuntu). Im running this directly on the server in question, so I wouldnt have thought it would be a firewall issue anyway.
You probably need to uncomment the following parameter in cassandra-env.sh:
-Djava.rmi.server.hostname=<public name>
Replace with the address of the interface you want the jmx interface to listen on.
nodetool connects through JMX interface. By default it's listening on port 7199 (other tools use RPC interface listening on port 9160 by default). Check JMX settings in cassandra-env.sh file. Most likely JMX server is listening on wrong interface (or probably loopback interface).
Default JMX configuration section (cassandra ver. 1.1.5) contains link to troubleshooting guide:
# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
# JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"
#
# see
# https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole
# for more on configuring JMX through firewalls, etc. (Short version:
# get it working with no firewall first.)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
It also worths to list all network interfaces using ifconfig and try telnet'ing port 7199 on all interfaces.
I was facing the same timeout issue. However I found that my cluster was not getting started properly because of token issue and I was getting "Host ID collision between active endpoint". Once i deleted data directory and restarted cluster then nodetool started working fine.
I also saw this same issue but it turned out to be some weirdness in my hosts file that was preventing JMX from binding to the interfaces.
Specifically, the host file had an entry for the external IP address with the hostname. Our servers had two interfaces, one external and one for an internal network. Removing that hosts entry did the trick.
As someone mentioned, it connects to the JMX port.
You can find the JMX port:
In /etc/cassandra/cassandra-env.sh. This won't work for ccm based local clusters OR
(my fav) by looking at the command-line of Cassandra node process running on the node.
My case was a cluster created locally using ccm so all my nodes were running on same host with different JMX port.
vagrant#triforce:~$ ps -eaf | grep cassandra | grepi -o " [^ ]*jmx.local.port[^ ]* "
-Dcassandra.jmx.local.port=7100
-Dcassandra.jmx.local.port=7300
-Dcassandra.jmx.local.port=7200
vagrant#triforce:~$
This is because I have 3 nodes running on the localhost.
vagrant#triforce:~$ nodetool -p 7100 ring
Datacenter: datacenter1
==========
Address Rack Status State Load Owns Token
3074457345618258602
127.0.0.1 rack1 Up Normal 64.65 MB 33.33% -9223372036854775808
127.0.0.2 rack1 Up Normal 65.26 MB 33.33% -3074457345618258603
127.0.0.3 rack1 Up Normal 65.92 MB 33.33% 3074457345618258602
vagrant#triforce:~$