I am able to see 160 Native clients on a particular node from OpsCenter.
But none of the application is pointing to this DC or any of the nodes from this DC.
If above is the situtation what are those 160 Native clients?
Few of them are because I have connected to that node using DevCenter.
Rest can be because of Inter DC communication??
No keyspace has RF set also for this DC. I am about to decommission this DC.
But not sure what are those clients.
Any idea??
In latest versions you can use nodetool clientlist or select * from system_views.clients ; in cqlsh to view the actual connections. This will give you the host and port which you can then track on the system what application is bound to it.
In older DSE versions you can also use dsetool perf userlatencytracking [enable|disable] to enable the userlatency tracking to do something similar with select * from dse_perf.user_io;.
I have used below command to find connecting clients:
sudo lsof -i -n -P | grep 9042 | grep ESTABLISHED
By running above, I found processes & those processes were all the java applications connecting to Cassandra. But I have not mentioned any of these hosts in connection, still requests were coming on them.
I found all those requests were because of Consistency Level: QUORUM by client applications. Although applications are not referring to those DCs directly but to achieve QUORUM requests were going on all DCs.
1 more thing was there:
Java client aplications were using username(superuser) 'cassandra', this also need consistency level QUORUM.
Inter DC communication does not happen on 9042, so my assumption of some connections from Inter DC was also not correct.
Above was solution to my problem.
Related
I am about to use simple cassnadra cluster (3 nodes, x.x.x.104-106). I'm using CentOS7, so i used datastax repository, Cassandra 3.0.
I read on forum, it is better to install the cassandra-stress outside the cluster, otherwise it consumes CPU of the node.
Could you please help me, how to install it?
I tried to copied cassandra-stress.sh separately, but it is dependent on some cassandra files (probably created during installation).
So I decided to install whole Cassandra on separate server, in the same network space. Now, I'm struggling with the correct setup, how to run cassandra-stress tool against the cassandra cluster.
In cassandra.yaml I setup Cassandra name, listen_adress to public_ip, rpc_address to loopback address, I set seeds to cassandra cluster nodes (x.x.x.104-106)... but in general it does not make sense to set it up, since I dont wan't create another node in the Cassandra cluster.
Could you please help me?
Edit: Maybe using something like this might be the correct way?
cassandra-stress user profile=/usr/cassandra/stress-file.yaml ops(insert=1,books=1) n=10000 -node x.x.x.104,x.x.x.105,x.x.x.106 -port native= ?
Telnet [cassandra_node_ip_ddress] 7000 works fine
If you have your Cassandra cluster running with the proper ports open (by default 9042 for clients and 7199 for JMX), and Cassandra directory on a different machine, then you should be able to run cassandra-stress, from outside the cluster, against your cluster simply by passing the -node option with an IP of one of the nodes in your cluster (say x.x.x.104). For example,
$CASSANDRA_HOME/tools/bin/cassandra-stress write -node x.x.x.104
should work. You can see more options with
$CASSANDRA_HOME/tools/bin/cassandra-stress help
on every node:
in cassandra.yaml set rpc_address to IP address
in cassanda-env.sh set LOCAL_JMX=no and jmx options autenticate=false
open firewall port 7199
restart firewall and cassandra
on cassandra-stress server:
cassandra-stress user profile=/usr/cassandra/stress-books.yaml ops\
(insert=1,books=1\)
n=10000 -node 172.16.20.104,172.16.20.105,172.16.20.106 -port native=9042
thrift=9160 jmx=7199
Note! JMX communication is not secured
In other dbs, we connect to db cluster with load balance IP. How do we connect to cassandra cluster using command line? What socket is used? Is this always a single node and IP?
What if i connect o node1, and node1 goes down. Will this automatically connect to node2 or node3?
You have several options: the easiest one is to use the Cassandra Query Language Shell (CQLSH), which is a python based CQL interpreter to interact with Cassandra. It usually comes with every Cassandra installation, under the /bin folder of the installation directory. If you have ssh access to one of the nodes Cassandra is running onto, this can be an easy option (you will avoid any issues related to firewall blocking incoming connections to your cluster).
You can also use cqlsh to access remotely to the cluster:
cqlsh node_ip 9043
but this will require cqlsh to be present on your machine.
In general, Cassandra uses an initial set of contact nodes and a gossip protocol to contact and learn the cluster composition. You will be assigned a node as coordinator for your query. You may not worry about seed nodes being currently down, provided that at least one is up and running.
Another option to access remotely to the cluster is the Datastax DevCenter,which is a free-to-use grafical interface to execute CQL queries.
Hope this helps
I am new to Cassandra and learning it.
So question is how communication is done between nodes in Cassandra
Basic communication - failure detection and other
Data transmission from node to node and client
Any other type of communication
Answer of 1st one is Gossip protocol http://www.datastax.com/resources/faq
But I am little curious about protocol and methodology Cassandra uses to transfer data from one node to another or client.
Communication between nodes is through Gossip, as stated by you.
Failure detection is again through Gossip, each node checks for Gossip messages from other nodes. If it does not receive 'n'(configurable in cassandra.yaml file) number of gossip messages it considers the node as dead. Look for the tag phi-convict threshold.
I am not sure what cassandra uses for data transfer, mostly probably might be simple layers built over TCP. One of the major features of cassandra is that you don't have to worry about how Cassandra handles replication, you only have to think about the strategy
Cassandra inter node communication is separate to communication between nodes and clients.
Gossip - is used so that nodes are aware of failures (client not
involved)
This needs to be split: Nodes communicate/send data the storage_port (see cassandra.yaml - default port 7000), clients connect to port 9042 (or 9160 for old thrift clients) and communicate with a proprietary binary protocol specified here: https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v3.spec
Other communication you might care about is JMX, which node tool uses
More details here: http://www.datastax.com/documentation/cassandra/2.1/cassandra/security/secureFireWall_r.html
I've a four node apache cassandra community 1.2 cluster in single datacenter with a seed.
All configurations are similar in cassandra.yaml file.
The following issues are faced, please help.
1] Though fourth node isn't listed in nodetool ring or status command, system.log displayed only this node isn't communicating via gossip protoccol with other nodes.
However both jmx & telnet port is enabled with proper listen/seed address configured.
2] Though Opscenter is able to recognize all four nodes, the agents are not getting installed from opscenter.
However same JVM version is installed as well as JAVA_HOME is also set in all four nodes.
Further observed that problematic node has Ubuntu 64-Bit & other nodes are Ubuntu 32-Bit, can it be the reason?
What is the version of cassandra you are using. I had reported a similar kind of bug in cassandra 1.2.4 and it was told to move to subsequent versions.
Are you using gossiping property file snitch? If that's the case, your problem should have been solved by having updated cassandra-topology.properties files that are upto date.
If all these are fine, check your TCP level connection via netstat and TCP dump.If the connections are getting dropped at application layer, then consider a rolling restart.
You statement is actually very raw. Your server level configuration might be wrong in my assumption.
I would suggest you to check if cassandra-topology.properties and cassandra-racked.properties across all nodes are consistent.
I have a 5 node Cassandra cluster set up on EC2, all in the same region.
If I connect over cqlsh (9160), queries respond in under a second.
When I connect via Dev Center, or using the native Java Driver, both of which use port 9042, the queries take over 20 seconds to respond.
They consistently respond in the same 21 second region. Never fast and then slow.
I have set up a few Cassandra Clusters on EC2 and have seen this before but do not know how to fix the problem. The last time, I scrapped the cluster and built a new one and the response time on port 9042 was fine.
Any help in how to debug or fix this problem would be appreciated, thanks.
The current version of DevCenter was designed to support as main scenario running (longish) CQL scripts (vs an interactive console with queries executed one after another). DevCenter is using as an underlying connector the DataStax Java driver for Cassandra.
For the above mentioned scenario, in order to ensure there are no "conflicts", a new Session is created for each execution. When a Session is initialized, the driver performs an auto-node discovery, creates connection pools, etc. Basically it does a lot of preparation work. Depending on the latency from your client machine to the EC2 nodes, the size of the cluster and also the configuration of these nodes (see the connection requirements), this initialization phase can be quite expensive.
As you can imagine the time spent preparing wouldn't represent a large percentage of running a DDL script and a decent size of inserts/updates. But for an interactive scenario, it will result in a suboptimal behavior (the one you are describing)
The next version(s) of DevCenter will address the interactive scenario and optimize for it so the user experience would be what you'd expect. And supporting this scenario is pretty high on our list of priorities.
The underlying Java driver obtains the whole cluster topology when it initially connects. This enables it to automatically connect to any node in the cluster. On EC2 it only obtains the private addresses, tries each one, and then times out. It then sends the request over the initial connection