Cassandra nodetool throws an error after updating OpenJDK
nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - URISyntaxException: 'Malformed IPv6 address at index 7: rmi://[127.0.0.1]:7199'.
This also affects the current official Docker-Hub Image https://hub.docker.com/_/cassandra version 3.11.12
How can I fix this error?
There seems to be an issue with "improved" IPv6 address parsing in the latest jdk update.
The workaround would be to use the IPv6 notation of localhost
nodetool -h ::FFFF:127.0.0.1 status
You can upgrade to Apache Cassandra 3.11.13 or use this command:
nodetool -Dcom.sun.jndi.rmiURLParsing=legacy status
Another way is to add this -Dcom.sun.jndi.rmiURLParsing=legacy to JAVA_TOOL_OPTIONS environment variable.
Related
Hey guys i tried many of the solutions posted on different sites and stackoverflow too, but none of them worked for me.
I have eximented with the following cassandra.yaml paramters-
rpc_address
rpc_broadcast_address
listen_address
Most of the solutions are out dated, little help is highly appreciated
As discussed in chat, there was a typo while adding private ip address for listen_address in the cassandra.yaml. Due to this error, Cassandra never got started and hence no output for the following commands.
nodetool status
- Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)
netstat -an | grep 9042
- returns nothing
As always, the first place to look at the errors is cassandra system log. And it clearly indicated the following problem.
ERROR [main] 2018-05-08 02:01:26,541 CassandraDaemon.java:708 - Exception encountered during startup: Invalid yaml: file:/etc/cassandra/cassan$
Error: while scanning a simple key; could not found expected ':'; in 'reader', line 678, column 1:
# Set rpc_address OR rpc_interfa ...
After fixing the ":" error in cassandra.yaml file, Cassandra came up smoothly.
For my 6 node cluster, I am unable to cqlsh <ip> to the cluster. It gives me :
Connection error: ('Unable to connect to any servers', {'192.168.19.2': ProtocolError("cql_version '3.3.1' is not supported by remote (w/ native protocol). Supported versions: [u'3.4.4']",)})
But, I checked nodetool status, all my nodes are up and running. Also, I am able to contact the cluster via client.
For the cassandra.yaml on each node, I set the following parameters:
listen_address: node's ip
broadcast_address: node's ip
rpc_address: node's ip
broadcast_address: left blank
seed node: same two ips in each nodes
What seems to be the issue?
Update:
I am able to cqlsh from other nodes but not from the first nodes. I can observe that while working on the first node, I somehow messed up with its cqlsh version. Is it possible to change it?
Connection error: ('Unable to connect to any servers',
{'192.168.19.2': ProtocolError("cql_version '3.3.1' is not supported
by remote (w/ native protocol). Supported versions: [u'3.4.4']",)})
You do have different versions of cassandra installed or using the wrong driver. You want to upgrade to your drivers for connecting to your cluster as 3.4 seems to be a 3.10.x cluster while your driver only supports 3.0.x.
I am trying to start cqlsh and this is what I get:
/bin$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
OperationTimedOut('errors=None, last_host=None',)})
I tried removing ~/.cassandra, did not work. I also compared cassandra.yaml with a version that worked.
Any ideas?
posting on this old thread as a memo for others, as i couln't managed to find any info to resolve these symptoms without debugging up to now:
got the same issue with a slow testing cluster, and i resolved it by setting a missing control_connection_timeout kwargs in Cluster() init, in cqlsh.py file.
issue opened and patch proposal provided at https://issues.apache.org/jira/browse/CASSANDRA-10959
Depending on your version and configuration, check the values specified for listen_address and/or rpc_address in your cassandra.yaml. If they are defined to anything other than localhost, you will need to provide that address when connecting with cqlsh.
$ grep listen_address: /etc/cassandra/cassandra.yaml
listen_address: 210.156.89.15
$ cqlsh 210.156.89.15 -u aploetz -p aploetz
Connected to PermanentWaves at 210.156.89.15.
[cqlsh 5.0.1 | Cassandra 2.1.4 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
aploetz#cqlsh>
nodetool -h <ipaddress> -p 7199 status
Error connecting to remote Jmx agent!
java.rmi.NoSuchObectException: no such object in the table
Am getting the above error when I tried to run the nodetool status or any other nodetool command. Cassandra is running fine and nodetool status on other nodes in the cluster shows it is UN state. I tried to add the below entry in cassandra-env.sh file but still I got the same error
JVM_OPTS = "$JVM_OPTS -Djava.rmi.server.hostname="
You have to use your listen_address for nodetool host ip.
nodetool -h <listen_address> -p 7199 status
or if it is not working, try with sudo.
In cassandra.yaml file it is written that jmx by default will only work from localhost. To run it from a remote host you need to uncomment and provide values of the parameters written in that file.
also try
cat /var/lob/cassandra/cassandra.log | grep Error
see if it gives you any error regarding JMX connectivity
I have a problem with a single node Cassandra installation.
I can start it without any errors in the log.
I can create a keyspace, create tables, insert and delete data.
However truncate is not working
cqlsh> CREATE KEYSPACE mykeyspace WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor': 1};
cqlsh> use mykeyspace;
cqlsh:mykeyspace> create table test1 (num int, primary key (num));
cqlsh:mykeyspace> insert into test1 (num) values (12);
cqlsh:mykeyspace> select * from test1;
num
-----
12
(1 rows)
cqlsh:mykeyspace> truncate test1;
Unable to complete request: one or more nodes were unavailable.
Also if I try to run nodetool describecluster it doesn't return complete response
[XXXX#XXXX dsc-cassandra-2.0.6]$ ./bin/nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
UNREACHABLE: [127.0.0.1]
I'm using
Cassandra DSC 2.0.6.
Red Hat 5.8.
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
I get responses for ping 127.0.0.1 and ping localhost
I checked all the ports that I am aware of cassandra may need (7000, 9160, 7199, 9042) using telnet - for example
telnet 127.0.0.1 7199
telnet localhost 7199
I can connect to these ports.
I'm using the default cassandra.yaml. These are the lines where either IP or hostname shows up
listen_address: localhost
rpc_address: localhost
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "127.0.0.1"
I also looked into the source code. I believe the problem can be close to the method org.apache.cassandra.service.StorageProxyMBean.describeSchemaVersions(). Most likely I get no response to the SCHEMA_CHECK message.
I tried to enable TRACE log in log4j for nodetool (conf/log4j-tools.properties) to get more information about the issue, but somehow log4j didn't start logging (it did create the file that I set in the appender, but the file was empty.)
There must be something specific to this environment because I can't repeat this problem in any other environments. So I can't figure out what's causing it.
The problem was that Cassandra couldn't load snappy.
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:359)
at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:150)
I turned off compression in cassanda.yaml
internode_compression: none
Now both nodetool describecluster and I truncate work.
I also found a similar post here Cassandra Startup Error 1.2.6 on Linux x86_64
Since I can't install another glibc on this machine for the sake of testing I downloaded snappy-java-1.0.4.1.jar and replaced libsnappyjava.so in my snappy-java-1.0.5.jar
With this jar I was able to run cassandra with
internode_compression: all
(I have glibc 2.5 installed)