Cannot start Cassandra : Port already in use - cassandra

I have two Ubuntu 16.04 nodes on which I installed Cassandra 3.11.3 with java version "1.8.0_181". I want to merge these two nodes into a Cassandra cluster. Their intern ips are 172.16.10.20 and 172.16.10.30.
on each /etc/cassandra/cassandra.yaml file I modified the following lines:
cluster_name: 'my_cluster'
- seeds: "172.16.10.20,172.16.10.30"
listen_address: XXXX
rpc_address: XXXX
where XXXX is respectively the intern ip of the current node.
I then restart Cassandra on each node
sudo service cassandra restart
and check that Cassandra runs :
sudo service cassandra status
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/init.d/cassandra; generated)
Active: active (running) since Wed 2018-08-08 00:31:42 UTC; 3s ago
Docs: man:systemd-sysv-generator(8)
and the cluster
nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.16.10.20 190.11 KiB 256 100.0% 84dded4c-c74e-45f4-9481-ff837fec229d rack1
UN 172.16.10.30 265.06 KiB 256 100.0% 4695fef4-70c7-46b2-a0bd-8b752fe5beb6 rack1
Everything is up and normal.
I want to connect now to Cassandra:
cqlsh
and get:
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
Some googling later, I want to start cassandra by hand
cassandra
and get (among a huge message):
ERROR [main] 2018-08-07 23:02:51,365 CassandraDaemon.java:708 - Port already in use: 7199; nested exception is:
java.net.BindException: Address already in use (Bind failed)
It looks like the port 7199 is already in use. I kill the corresponding pid, change in /etc/cassandra/cassandra-env.sh the JMX_PORT to 7200... same issue, the port is said to be in use, plus the error
00:33:06,236 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[SYSTEMLOG] - openFile(/var/log/cassandra/system.log,true) call failed. java.io.FileNotFoundException: /var/log/cassandra/system.log (Permission denied)
I have changed the permission, but the error remains. At this point of the story I am running out of ideas. What I am trying to achieve seem pretty straighforward so I guess others must have run into a similar issue.

The nodetool status output says it all here. You had everything running just fine. So revert any changes in terms of port usage.
As your nodetool status reveals that your node IPs are 172.16.10.20 and 172.16.10.30, try running cqlsh and providing one of those IPs. cqlsh tries to connect to 127.0.0.1 by default, which will not work in a plural node cluster.
cqlsh 172.16.10.20 -u yourusername -p yourpassword
Note: You can omit -u and -p if you don't have auth enabled. But if that's true, then you should really change your cluster to enable auth.

Related

Enable cassandra access from other hosts

I am trying to enable cassandra access from other hosts.
I updated the cassandra.yaml listen_address and rpc_address to the host IP v4 address.
as specified in the doc
From
listen_address: localhost
rpc_address: localhost
To
listen_address: cassandra_host_ipv4_address
rpc_address: cassandra_host_ipv4_address
I restarted cassandra to pick the configs.
The nodetool status command shows correctly
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN cassandra_host_ipv4_address 204.8 KiB 8 ? null r1
I tried connecting from the same cassandra host before trying with other hosts using
cqlsh
cqlsh localhost
cqlsh cassandra_host_ipv4_address
All of them failed with
Connection error: ('Unable to connect to any servers', {'ip-v4-address': error(111, "Tried connecting to [('ip-v4-address', 9042)]. Last error: Connection refused")})
I tried checking the logs but only gc.log.0.current exist?
Does anyone know how to expose cassandra such that it can be connected from different host.

Multi-node multi-datacenter CASSANDRA

I am trying to setup a multi-node multi-datacenter cluster in Cassandra 3.11
For data-center 1 I have Cassandra running on 3 nodes(eg. 10.90.22.11, 10.90.22.12 and 10.90.22.13) and for data-center 2 I have Cassandra running on 2 nodes(eg. 10.90.22.21 and 10.90.22.22).
The ring is up but they are working separately. To make them work together I update the endpoint_snitch to be GossipingPropertyFileSnitch and also the dc and rac in cassandra-rackdc.properties to be DC1 and DC2 for respective nodes following the steps mentioned in this link.
After these changes when I restart Cassandra, the status of Cassandra is running however when I check for the ring with nodetool status I receive a error:
nodetool: Failed to connect to '127.0.0.1:7199'
ConnectException: 'Connection refused (Connection refused)'
What am I missing?
This error you posted indicates that nodetool couldn't connect to JMX that is supposed to be listening on port 7199:
Failed to connect to '127.0.0.1:7199'
Verify that Cassandra is running and check that the process is bound to various ports including 7199, 9042 and 7000. You can try running one of these commands:
$ netstat -tnlp
$ sudo lsof -nPi | grep LISTEN | grep java
Cheers!
You should try nodetool command with host/IP what you have put in your cassandra.yaml. Also, you should check your port 7199 or custom port if you set is open/allow from firewall.
nodetool -h hostname/ip status.
you can mention username.password if you enabled. please refer below link for more details and understanding:-
http://cassandra.apache.org/doc/latest/tools/nodetool/status.html

why is cassandra not getting connected to server?

I have installed cassandra and worked on it. It worked properly. Now, it is showing as-
localhost/<> is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
Fatal configuration error; unable to start server. See log for stacktrace.
INFO 09:17:02 Announcing shutdown
INFO 09:17:02 Compacted 4 sstables to [./../data/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-33,]. 6,485 bytes to 5,751 (~88% of original) in 223ms = 0.024595MB/s. 4 total partitions merged to 1. Partition merge counts were {4:1, }
INFO 09:17:04 Waiting for messaging service to quiesce
user#inblrlt-user:~/dev/Cassandra/apache-cassandra-2.1.7/bin$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': OperationTimedOut('errors=None, last_host=None',)})
How to change my server address so that the issue is cleared?
Your localhost is already in use. Follow the following steps-
$ jps
You see some processes running. For example:
9107 Jps
1112 CassandraDaemon
Then kill the CassandraDaemon process by the process id you see after executing jps. In my example, here process id 1112 for CassandraDaemon.
$ kill -9 1112
Then check processes again after a while-
$ jps
You will see CassandraDaemon will no longer be available.
9170 Jps
Then remove your saved_caches and commilog and start cassandra again.
If you want to change the listen_address from localhost to any private ip or public ip, you need to make the following changes:
change seeds: at cassandra.yaml
change listen_address: at cassandra.yaml
change rpc_address: at cassandra.yaml
set JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<place_your_ip_here>" at cassandra-env.sh

cassandra is not running as service

The system is Linux 14.04.1-Ubuntu x86_64, 200GB space, 8GB memory. Everything is done in both root and user. We installed the Cassandra version 3.6.0 from datastax using the following command (followed the instruction from website: http://docs.datastax.com/en/cassandra/3.x/cassandra/install/installDeb.html):
$ apt-get update
$ apt-get install datastax-ddc
However, the cassandra is not started as service.
root#e7:~# nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
root#e7:~# service cassandra start
root#e7:~# service cassandra status
* Cassandra is not running
We can start Cassandra manually using the command:
$ cassandra -R -f
...
INFO 18:45:02 Starting listening for CQL clients on /127.0.0.1:9042 (unencrypted)...
INFO 18:45:02 Binding thrift service to /127.0.0.1:9160
INFO 18:45:02 Listening for thrift clients...
INFO 18:45:12 Scheduling approximate time-check task with a precision of 10 milliseconds
root#e7:~# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 153.45 KiB 256 100.0% 28ba16df-1e4c-4a40-a786-ebee140364bf rack1
However, we have to start cassandra as a service. Any suggestions how to fix the problem?
Try using http://docs.datastax.com/en/cassandra/3.0/cassandra/install/installDeb.html
This is more stable and I have tried it.
I think the ports are not opened.
Try opening the following ports:
Cassandra inter-node ports
Port number Description
7000 Cassandra inter-node cluster communication.
7001 Cassandra SSL inter-node cluster communication.
7199 Cassandra JMX monitoring port.
Cassandra client port
Port number Description
9042 Cassandra client port.
9160 Cassandra client port (Thrift).
Also what type of Snitch is defined in the Cassandra.yaml file ?

cassandra 3.4 on virtual box not starting

I am using mac osx. i created 3 virtual box by virtualbox. I've installed centos7 minimal version on each of the virtual box.
Then i installed cassandra on each of the box. After installation it was starting by cqlsh and nodetool status command.
But after then when i was trying to link each other and edit cassandra.yaml file its started showing
('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
i've edited the cassandra.yaml file as follows:
cluster_name: 'Home Cluster'
num_tokens: 256
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
- seeds: "192.168.56.102,192.168.56.103"
storage_port: 7000
listen_address: 192.168.56.102
rpc_address: 192.168.56.102
rpc_port: 9160
endpoint_snitch: SimpleSnitch
my /etc/hosts file contains:
192.168.56.102 node01
192.168.56.103 node02
192.168.56.104 node03
Please tell me whats wrong i'm doing? My cassandra cluster not working.
solution: I got the solution from AKKI. The problem was enpoint_snitch. I made the endpoint_snitch=GossipingPropertyFileSnitch and it fixed. My now output is as follows:
[root#dbnode2 ~]# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.56.101 107.38 KB 256 62.5% 0526a2e1-e6ce-4bb4-abeb-b9e33f72510a rack1
UN 192.168.56.102 106.85 KB 256 73.0% 0b7b76c2-27e8-490f-8274-571d00e60c20 rack1
UN 192.168.56.103 83.1 KB 256 64.5% 6c8d80ec-adbb-4be1-b255-f7a0b63e95c2 rack1
I had faced similar problem,
I tried the following solution:
In Cassandra.yaml file check if you have,
start_rpc = true
Changed my endpoint snitch to
endpoint_snitch: GossipingFilePropertySnitch
Opened all ports Cassandra uses on my CentOS
Cassandra inter-node ports
Port number Description
7000 Cassandra inter-node cluster communication.
7001 Cassandra SSL inter-node cluster communication.
7199 Cassandra JMX monitoring port.
Cassandra client port
Port number Description
9042 Cassandra client port.
9160 Cassandra client port (Thrift).
Command to open ports on CentOs 7(Find it according to your OS):
>sudo firewall-cmd --zone=public --add-port=9042/tcp --permanent
>sudo firewall-cmd –reload
Then Restart your systems
Also it seems that you are changing the Cassandra.Yaml file after starting cassandra.
Make sure you edit your Cassandra.yaml file on all nodes before starting Cassandra
Also remember to start the seed node first.

Resources