Cassandra nodetool throws an error after updating OpenJDK
nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - URISyntaxException: 'Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199'.
This also affects the current official Docker-Hub Image https://hub.docker.com/_/cassandra version 3.11.12
How can I fix this error?
There seems to be an issue with "improved" IPv6 address parsing in the latest jdk update.
The workaround would be to use the IPv6 notation of localhost
nodetool -h ::FFFF:127.0.0.1 status
You can upgrade to Apache Cassandra 3.11.13 or use this command:
nodetool -Dcom.sun.jndi.rmiURLParsing=legacy status
Another way is to add this -Dcom.sun.jndi.rmiURLParsing=legacy to JAVA_TOOL_OPTIONS environment variable.
Related
I am trying to setup a multi-node multi-datacenter cluster in Cassandra 3.11
For data-center 1 I have Cassandra running on 3 nodes(eg. 10.90.22.11, 10.90.22.12 and 10.90.22.13) and for data-center 2 I have Cassandra running on 2 nodes(eg. 10.90.22.21 and 10.90.22.22).
The ring is up but they are working separately. To make them work together I update the endpoint_snitch to be GossipingPropertyFileSnitch and also the dc and rac in cassandra-rackdc.properties to be DC1 and DC2 for respective nodes following the steps mentioned in this link.
After these changes when I restart Cassandra, the status of Cassandra is running however when I check for the ring with nodetool status I receive a error:
nodetool: Failed to connect to '127.0.0.1:7199'
ConnectException: 'Connection refused (Connection refused)'
What am I missing?
This error you posted indicates that nodetool couldn't connect to JMX that is supposed to be listening on port 7199:
Failed to connect to '127.0.0.1:7199'
Verify that Cassandra is running and check that the process is bound to various ports including 7199, 9042 and 7000. You can try running one of these commands:
$ netstat -tnlp
$ sudo lsof -nPi | grep LISTEN | grep java
Cheers!
You should try nodetool command with host/IP what you have put in your cassandra.yaml. Also, you should check your port 7199 or custom port if you set is open/allow from firewall.
nodetool -h hostname/ip status.
you can mention username.password if you enabled. please refer below link for more details and understanding:-
http://cassandra.apache.org/doc/latest/tools/nodetool/status.html
When ever you get a Cassandra cqlsh Connection error as follows:
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
You can also connect without using the IP address - just use the hostname:
cqlsh ‘hostname -I’
Another solution, type cqlsh <listen_address> [<port>] if it is not set to 127.0.0.1 or localhost
I was having the same issue with Cassandra 3.11.0, anytime I changed the address of rpc or listen address cqlsh wouldn't work. I had to add the same local ip to seeds
So after trial and error my cassandra.yml looked like:
class-name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
-seeds: "192.168.0.30"
listen_adress: 192.168.0.30
rpc_address: 192.168.0.30
then running
cqlsh 192.168.0.30 9042
When I installed Cassandra 3.11.1, I came across this problem.
I also found if I ran
service cassandra status
, there is a
cassandra dead but pid file exists
problem. It indicates that the Cassandra service does not start
I checked the
/var/log/cassandra/cassandra.log
and found this error:
Exception encountered during startup...
.It is a bug and already reported. The original post link https://issues.apache.org/jira/browse/CASSANDRA-14173.
The solution is to downgrade Cassandra to 3.0
download Cassandra rpm
curl -O
https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm
or
wget
https://www.apache.org/dist/cassandra/redhat/30x/cassandra-3.0.15-1.noarch.rpm
rpm -ivh cassandra-3.0.15-1.noarch.rpm
service cassandra start
service cassandra status # check cassandra status, it should say
cassandra (pid 1234) is running...
cqlsh #start cassandra
Hope this helps you
I am using Ubuntu 15.10 64 bits with OpenJDK 1.8.0_66. Cassandra 3.3 and OpsCenter 5.2.4 are installed from apt packages and running locally. I am using the following sources.
deb http://www.apache.org/dist/cassandra/debian 33x main
deb http://debian.datastax.com/community stable main
The OpsCenter doc says: "In Add Cluster, enter the Hostnames or IP addresses of two or three nodes in the cluster, set the JMX and Native Transport ports, and click Save Cluster".
I have a single-node cluster, so I don't have two or three IP addresses to enter. When I enter the address of the node, I get the following message: "Error creating cluster: Unable to connect to cluster. Error is: Unable to connect to any seed nodes, tried ['127.0.0.1']".
I am using the default ports. netstat -an gives among others the following lines.
tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN
tcp6 0 0 127.0.0.1:9042 :::* LISTEN
nodetool status is working fine.
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 201.17 KB 256 ? 04da2c59-1a88-4ed3-9af9-8f64ae27e9ac rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
So is cqlsh.
$ cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.3 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cqlsh>
After looking at other questions I tried uncommenting the following in /etc/cassandra/cassandra-env.sh. I tried both localhost and 127.0.0.1. It did not help.
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<hostname>"
There is nothing in /var/log/cassandra/debug.log or /var/log/cassandra/system.log when the connection attempts fail. The only log messages are from Cassandra starting/stopping.
Is it possible at all to use OpsCenter with a single-node cluster? If so, what might I be missing?
Cassandra 3.3 and OpsCenter 5.2.4
You're out of luck, OpsCenter 5.x only work for Cassandra 2.1 branch and the future OpsCenter 6.x that will be compatible with Cassandra 3.x branch will only be available wit the Datastax Enterprise version, read this: http://docs.datastax.com/en/opscenter/5.2/opsc/opscPolicyChanges.html
Im trying to use nodetool to check the status of my cluster, but its unable to connect.
My cassandra.yaml is configured with listen_address and rpc_address set as the server IP (e.g. 10.10.10.266).
Im able to connect through cqlsh and cassandra-cli using the same IP, but when I connect to nodetool it doesnt work.
/bin$ nodetool -h 10.10.10.266 ring
Failed to connect to '10.10.10.266:7199': Connection has timed out
I dont think I have a firewall enabled on the server (Ubuntu). Im running this directly on the server in question, so I wouldnt have thought it would be a firewall issue anyway.
You probably need to uncomment the following parameter in cassandra-env.sh:
-Djava.rmi.server.hostname=<public name>
Replace with the address of the interface you want the jmx interface to listen on.
nodetool connects through JMX interface. By default it's listening on port 7199 (other tools use RPC interface listening on port 9160 by default). Check JMX settings in cassandra-env.sh file. Most likely JMX server is listening on wrong interface (or probably loopback interface).
Default JMX configuration section (cassandra ver. 1.1.5) contains link to troubleshooting guide:
# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
# JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"
#
# see
# https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole
# for more on configuring JMX through firewalls, etc. (Short version:
# get it working with no firewall first.)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
It also worths to list all network interfaces using ifconfig and try telnet'ing port 7199 on all interfaces.
I was facing the same timeout issue. However I found that my cluster was not getting started properly because of token issue and I was getting "Host ID collision between active endpoint". Once i deleted data directory and restarted cluster then nodetool started working fine.
I also saw this same issue but it turned out to be some weirdness in my hosts file that was preventing JMX from binding to the interfaces.
Specifically, the host file had an entry for the external IP address with the hostname. Our servers had two interfaces, one external and one for an internal network. Removing that hosts entry did the trick.
As someone mentioned, it connects to the JMX port.
You can find the JMX port:
In /etc/cassandra/cassandra-env.sh. This won't work for ccm based local clusters OR
(my fav) by looking at the command-line of Cassandra node process running on the node.
My case was a cluster created locally using ccm so all my nodes were running on same host with different JMX port.
vagrant#triforce:~$ ps -eaf | grep cassandra | grepi -o " [^ ]*jmx.local.port[^ ]* "
-Dcassandra.jmx.local.port=7100
-Dcassandra.jmx.local.port=7300
-Dcassandra.jmx.local.port=7200
vagrant#triforce:~$
This is because I have 3 nodes running on the localhost.
vagrant#triforce:~$ nodetool -p 7100 ring
Datacenter: datacenter1
==========
Address Rack Status State Load Owns Token
3074457345618258602
127.0.0.1 rack1 Up Normal 64.65 MB 33.33% -9223372036854775808
127.0.0.2 rack1 Up Normal 65.26 MB 33.33% -3074457345618258603
127.0.0.3 rack1 Up Normal 65.92 MB 33.33% 3074457345618258602
vagrant#triforce:~$
I've got Cassandra 0.7 running in standalone mode and I'm tryin to run nodetool but I'm getting JMX exceptions. Isn't the JMX configuration required on accessing a remote server? I'm accessing my local machine.
Also why is nodetool looking for 63.251.179.13?
[rav#ubix bin]$ ./nodetool -h 127.0.0.1 flush
Error connection to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: 63.251.179.13; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2343)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:296)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:144)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:621)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at java.net.Socket.<init>(Socket.java:392)
at java.net.Socket.<init>(Socket.java:206)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 10 more
Thanks,
Try nodetool with -h or --host and -p or --port as per the instructions:
-h,--host <arg> node hostname or ip address
-p,--port <arg> remote jmx agent port number
When Cassandra is offline, check the ports in use to see if another process is using the default port that Cassandra binds to. You can find the default in conf/cassandra-env.sh
Once you know the port, you can see if another process is bound to it with netstat -an
If nothing is running on the port, and you start up cassandra, verify that it is running on the correct port and try to connect again with the -p or --port arguments. More information can be found here: http://wiki.apache.org/cassandra/GettingStarted
Is the machine unix or windows? do you have a bad entry in /etc/hosts indicating that 127.0.0.1 maps to another hostname or IP address, namely 63.251.179.13
I had a similar issue running nodetool on an instance of Cassandra running locally on my machine. When trying to run nodetool -h 127.0.0.1 nodetool was issuing an exception relating to JMX that looked like this (where there was an unknown - to me - IP Address).
Error connecting to remote JMX agent!
java.rmi.ConnectIOException: Exception creating connection to: ; nested exception is:
java.net.SocketException: Host is down
Douglas Muth posted a similar issue here, and from this, I found out that Cassandra seems to be recording the hostname at startup. Unfortunately, by the time I ran nodetool the hostname had become stale (my IP address is allocated dynamically).
My solution then, was to restart cassandra, which updated the IP and rerun nodetool. No more JMX errors, no more strange IP address. This worked a treat for me as I'm running a local instance of Cassandra on localhost and don't mind the restart but it's not a very satisfactory solution.