Node is unreachable in single node Cassandra installation - cassandra

I have a problem with a single node Cassandra installation.
I can start it without any errors in the log.
I can create a keyspace, create tables, insert and delete data.
However truncate is not working
cqlsh> CREATE KEYSPACE mykeyspace WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor': 1};
cqlsh> use mykeyspace;
cqlsh:mykeyspace> create table test1 (num int, primary key (num));
cqlsh:mykeyspace> insert into test1 (num) values (12);
cqlsh:mykeyspace> select * from test1;
num
-----
12
(1 rows)
cqlsh:mykeyspace> truncate test1;
Unable to complete request: one or more nodes were unavailable.
Also if I try to run nodetool describecluster it doesn't return complete response
[XXXX#XXXX dsc-cassandra-2.0.6]$ ./bin/nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
UNREACHABLE: [127.0.0.1]
I'm using
Cassandra DSC 2.0.6.
Red Hat 5.8.
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
I get responses for ping 127.0.0.1 and ping localhost
I checked all the ports that I am aware of cassandra may need (7000, 9160, 7199, 9042) using telnet - for example
telnet 127.0.0.1 7199
telnet localhost 7199
I can connect to these ports.
I'm using the default cassandra.yaml. These are the lines where either IP or hostname shows up
listen_address: localhost
rpc_address: localhost
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "127.0.0.1"
I also looked into the source code. I believe the problem can be close to the method org.apache.cassandra.service.StorageProxyMBean.describeSchemaVersions(). Most likely I get no response to the SCHEMA_CHECK message.
I tried to enable TRACE log in log4j for nodetool (conf/log4j-tools.properties) to get more information about the issue, but somehow log4j didn't start logging (it did create the file that I set in the appender, but the file was empty.)
There must be something specific to this environment because I can't repeat this problem in any other environments. So I can't figure out what's causing it.

The problem was that Cassandra couldn't load snappy.
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:359)
at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:150)
I turned off compression in cassanda.yaml
internode_compression: none
Now both nodetool describecluster and I truncate work.
I also found a similar post here Cassandra Startup Error 1.2.6 on Linux x86_64
Since I can't install another glibc on this machine for the sake of testing I downloaded snappy-java-1.0.4.1.jar and replaced libsnappyjava.so in my snappy-java-1.0.5.jar
With this jar I was able to run cassandra with
internode_compression: all
(I have glibc 2.5 installed)

Related

Apache Cassandra 3.7 snitch issue cannot start data center

I am using ubuntu 14.04 with apache cassandra 3.7. I am trying to start it but get the following error message:
ERROR [main] 2016-07-15 15:22:10,627 CassandraDaemon.java:731 - Cannot start node if snitch's data center (dc1) differs from previous data center (datacenter1). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
I know I can set -Dcassandra.ignore_dc=true, BUT that is not a fix, its a band-aid and for development use only, this is suppose to be in production. I tried to clear out all the files and folders in /var/lib/cassandra, I MEAN every SINGLE FILE AND FOLDER, started apache cassandra again, AND STILL THE SAME ERROR MESSAGE... any other idea??
change in file:
/etc/cassandra/cassandra-rackdc.properties
entry from dc1 to datacenter1
on all nodes
and then do a rolling restart of nodes.
If have just switched to GossipingPropertyFileSnitch, start Cassandra with the option
-Dcassandra.ignore_dc=true
If it starts successfully, execute:
nodetool repair
nodetool cleanup
Afterwards, Cassandra should be able to start normally without the ignore option.
I faced the issue while upgrading my Apache cassandra from 3.11.1 to 3.11.4 .
cassandra.yaml
old_Config : endpoint_snitch: GossipingPropertyFileSnitch
New_Config: endpoint_snitch: SimpleSnitch
{changed it to GossipingPropertyFileSnitch}
cassandra-rackdc.properties
old_version_config: dc:Dc1 rack:Rack1
New_version_config: dc:dc rack:rack (changed this to Dc1 and Rack1)
this resolves my issue

Datastax OpsCenter - Can't connect to Cassandra All host(s) tried for query failed

I have a two node and I installed DataStax OpsCenter on 10.5.0.201 but I give error ( Unable connect to any seed nodes ) when I create new cluster on OpsCenter. Can you check my config ?
address.yaml ( 10.5.0.201 )
address.yaml ( 10.5.0.203 )
nodetool status
DataStax OpsCenter
startup.log (10.5.0.201)
agent.log ( 10.5.0.201 )
Versions
Python 2.6.6
java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)
OpsCenter 5.2.4
Best Regards
In the comments of cassandra.yaml
# Note that unlike listen_address, you can specify 0.0.0.0, but you must also
# set broadcast_rpc_address to a value other than 0.0.0.0.
I would recommend to set rpc_address: 10.5.0.201 for the 1st node and to retry

Apache Cassandra - cqlsh operation timeout

I am trying to start cqlsh and this is what I get:
/bin$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
OperationTimedOut('errors=None, last_host=None',)})
I tried removing ~/.cassandra, did not work. I also compared cassandra.yaml with a version that worked.
Any ideas?
posting on this old thread as a memo for others, as i couln't managed to find any info to resolve these symptoms without debugging up to now:
got the same issue with a slow testing cluster, and i resolved it by setting a missing control_connection_timeout kwargs in Cluster() init, in cqlsh.py file.
issue opened and patch proposal provided at https://issues.apache.org/jira/browse/CASSANDRA-10959
Depending on your version and configuration, check the values specified for listen_address and/or rpc_address in your cassandra.yaml. If they are defined to anything other than localhost, you will need to provide that address when connecting with cqlsh.
$ grep listen_address: /etc/cassandra/cassandra.yaml
listen_address: 210.156.89.15
$ cqlsh 210.156.89.15 -u aploetz -p aploetz
Connected to PermanentWaves at 210.156.89.15.
[cqlsh 5.0.1 | Cassandra 2.1.4 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
aploetz#cqlsh>

Cassandra Cluster configuration

I am trying to configure two windows servers in my network as Cassandra cluster.
I did some reading in various sites and changed the below in Cassandra.yalm
after changing the default value of 127.0.0.1 to actual IP the Cassandra service is not starting.
I also added the map to actual IP to localhost in (windows) hosts file.
After doing the above change, the service is coming up when I start the service. it is stopping immediately.
The reason I am changing this IP is to make this a cluster with two node setup,
Please let me know if I miss some thing.
Version: Datastax community version of Cassandra
Server : windows.
Thx
Muthu
Message from Cassandra.txt in logs dir:
ERROR [main] 2014-09-18 11:43:12,155 DatabaseDescriptor.java (line 116) Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml Caused by: Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=seed_provider for JavaBean=org.apache.cassandra.config.Config#34e5190a; No suitable constructor with 2 arguments found for class org.apache.cassandra.config.SeedProviderDef in 'reader', line 8, column 1: cluster_name: 'Test Cluster'
If you want to create Cassandra cluster you must have at least two nodes and configure /etc/cassandra/cassandra.yaml
cassandra.yaml
cluster_name: 'Some Cluster Name'
listen_address: [Current IP]
rpc_address: [Current IP]
seed_provuder:
- seeds: "[Current IP], [Remote IP]"
Note: seeds must have at least two IPs which must be reachable for each other
Clean and start Cassandra instance
sudo rm -rf /var/lib/cassandra/* /var/log/cassandra/*
Note: Cassandra instance must be killed before cleaning those folders.

multi node cassandra installation ended with "UnknownHostException"

I am newbie to Cassandra. I wanted to install cassandra-0.8.4 on 3 nodes and to run Map/Reduce job that uploads data from HDFS to Cassandra.
I have installed Cassnadra on 3 nodes lab02(199.168.0.2),lab03(199.168.0.3) & lab04(199.168.0.4) respectively and can create a keyspace & column family and they got distributed across the cluster.
When I run my map/reduce program it ended up with "UnknownHostException". the same map/reduce program works well on single node cluster.
Here are the steps which I have followed.
cassandra.yaml details
lab02(199.168.0.2): (seed node)
auto_bootstrap: false seeds: "199.168.0.2" listen_address: 199.168.0.2
rpc_address: 199.168.0.2
lab03(199.168.0.3): auto_bootstrap: true seeds: "199.168.0.2"
listen_address: 199.168.0.3 rpc_address: 199.168.0.3
lab04(199.168.0.4): auto_bootstrap: true seeds: "199.168.0.2"
listen_address: 199.168.0.4 rpc_address: 199.168.0.4
2.
When I run my map/reduce program it ended up with "UnknownHostException"
Error: java.net.UnknownHostException: /199.168.0.2 at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849) at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200) at java.net.InetAddress.getAllByName0(InetAddress.java:1153) at java.net.InetAddress.getAllByName(InetAddress.java:1083) at java.net.InetAddress.getAllByName(InetAddress.java:1019) at java.net.InetAddress.getByName(InetAddress.java:969) at org.apache.cassandra.client.RingCache.refreshEndpointMap(RingCache.java:93) at org.apache.cassandra.client.RingCache.(RingCache.java:67) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter.(ColumnFamilyRecordWriter.java:98) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter.(ColumnFamilyRecordWriter.java:92) at org.apache.cassandra.hadoop.ColumnFamilyOutputFormat.getRecordWriter(ColumnFamilyOutputFormat.java:132) at org.apache.cassandra.hadoop.ColumnFamilyOutputFormat.getRecordWriter(ColumnFamilyOutputFormat.java:62) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:553) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) at org.apache.hadoop.mapred.Child.main(Child.java:170)
Here are the config line for map/reduce.
job4.setReducerClass(TblUploadReducer.class );
job4.setOutputKeyClass(ByteBuffer.class);
job4.setOutputValueClass(List.class);
job4.setOutputFormatClass(ColumnFamilyOutputFormat.class);
ConfigHelper.setOutputColumnFamily(job4.getConfiguration(), args[1],args[3] );
ConfigHelper.setRpcPort(job4.getConfiguration(), args[7]); // 9160
ConfigHelper.setInitialAddress(job4.getConfiguration(), args[9]); // 199.168.0.2
ConfigHelper.setPartitioner(job4.getConfiguration(), "org.apache.cassandra.dht.RandomPartitioner");
Steps which I have verified are
There is a passwordless ssh has been configured b/w lab02,lab03
&lab04. All the nodes can ping each other with out any issues.
When I ran "InetAddress.getLocalHost()" from java program on lab02 it
prints "lab02/199.168.0.2".
When I over looked "o/p" of bin/cassandra it prints couple of
messages and under InetAddress field "/199.168.0.3" etc. Here it does
not print "hostname/IP". Is that problem?
Kindly help me to resolve above issue.
Regards,
Thamizhannal
This has been resolved already in 0.8 tip: https://issues.apache.org/jira/browse/CASSANDRA-3044
It's trying to look up the literal string "/199.168.0.2" as an address, which is indeed invalid. If this is coming from your configuration, remove the slash. Otherwise, it might be coming from RingCache which would be a bug. If so, first upgrade to 0.7.8 to make sure it's not already fixed. If the problem persists, open a bug on https://issues.apache.org/jira/browse/CASSANDRA.
Edit: Nate is right, this is a known bug in 0.8.4 that is fixed for 0.8.5.

Resources