I am having a cassandra ring across two datacenters, below is the key space definition. when I trying to bring all the nodes of the local datacenter (aws), I am expecting the the datastax driver to query the remote nodes. But in this case I am getting HostNotAvaialble exception. Please help.
Keyspace definition as below
CREATE KEYSPACE IF NOT EXISTS mystore_stress WITH replication = {
'class':'NetworkTopologyStrategy',
'sol':2,
'aws':2
};
My session is created as follows:
public Session getSession() {
final Cluster cluster = Cluster.builder().addContactPoints(contactPoints)
.withRetryPolicy(DefaultRetryPolicy.INSTANCE)
.withLoadBalancingPolicy(new TokenAwarePolicy(new DCAwareRoundRobinPolicy("aws", 1)))
.withReconnectionPolicy(new ExponentialReconnectionPolicy(RECONNECTION_BASE_DELAY, RECONNECTION_MAX_DELAY))
.withSocketOptions(
new SocketOptions()
.setConnectTimeoutMillis(CONNECT_TIMEOUT_MILLIS)
.setReadTimeoutMillis(READ_TIMEOUT_MILLIS))
.withPort(PORT)
.build();
return cluster.connect();
}
I suspect you need to review the consistency level on your query. If it's quorum then you should ideally have 3 replicates. You can find some examples of read consistency here:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dmlClientRequestsRead.html
Related
I have a 3 node cluster with 1 seed and nodes in different zones. All running in GCE with GoogleCLoudSnitch.
I wanted to change hardware on each node so I started with adding a new seed in a different region which joined perfectly to the cluster. Then I started with "nodetool decommission" and when done I removed the the node when it is down and "nodetool status" states it's not in the cluster. I did this for all nodes and lastly I did it on the "extra" seed in the different region just to remove it to get back to a 3 node cluster.
We lost data! What can possibly be the problem? I saw a commando, "nodetool rebuild", which I ran and actually got some data back. "nodetool cleanup" didn't help either. Should I have run "nodetool flush" prior to "decommission"?
At the time of running "decommission" most keyspaces had ..
{'class' : 'NetworkTopologyStrategy', 'europe-west1' : 2}"
Should I first altered key spaces to include the new region/datacenter, which would be "'europe-west3' : 1" since only one node exist in that datacenter? I also noted that some keyspaces in the cluster had by mistake ..
{ 'class' : 'SimpleStrategy', 'replication_factor' : 1 }
Could this have caused the loss of data? It seems that it was in the "SimpleStrategy keyspaces" the data was lost.
(Disclaimer: I'm a ScyllaDB employee)
Did you 1st add new nodes to replace the ones you are decommissioning and configured the keyspace replication strategy accordingly? (you only mentioned the new seed node in your description, you did not mention if you did it for the other nodes).
Your data loss can very well be a result of the following:
Not altering the keyspaces to include the new region/zone with the proper replication strategy and replication factor.
Keyspaces that were configured with simple strategy (no netwrok aware) replication policy and replication factor 1. This means that the data was stored only in 1 node, and once that node went down and decommissioned, you basically lost the data.
Did you by any chance took snapshots and stored them outside your cluster? If you did you could try and restore them.
I would highly recommend reviewing these procedures for better understanding and the proper way to perform the procedure you intended to perform:
http://docs.scylladb.com/procedures/add_dc_to_exist_dc/
http://docs.scylladb.com/procedures/replace_running_node/
I have created table in Cassandra with below commands:
CREATE KEYSPACE test WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'dc1' : 3 } AND DURABLE_WRITES = false;
use test;
create table demo(id int primary key, name text);
Once the table got created successfully, I was running the below code to write the data into Cassandra from Spark.
But facing below error
Code Snippet of Spark
import com.datastax.spark.connector._
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector.cql._
val connectorToClusterOne = CassandraConnector(sc.getConf.set("spark.cassandra.connection.host","xx.xx.xx.xx").set("spark.cassandra.auth.username", "xxxxxxx").set("spark.cassandra.auth.password", "xxxxxxx"))
---K/V---
val data = sc.textFile("/home/ubuntu/test.txt").map(_.split(",")).map(p => demo(p(0).toInt,p(1)))
implicit val c = connectorToClusterOne
data.saveToCassandra("test","demo")
BELOW IS THE ERROR DESCRIPTION: .
Error while computing token map for keyspace test with datacenter dc1: could not achieve replication factor 3 (found 0 replicas only), check your keyspace replication settings.
Could any one suggest what could be the possible reason for this.
This error is usually because either the request is not being directed at the correct cluster or the datacenter does not exist or has an incorrect name.
To make sure you are connecting to the correct cluster double check the connection host used for your spark application.
To check the datacenter, use nodetool status to make sure that the datacenter which you requested exists and includes no extraneous whitespace.
Lastly, it could be possible that all the nodes in the datacenter are down, so double check this as well.
I have configured a cassandra clustter with 3 nodes
Node1(192.168.0.2) , Node2(192.168.0.3), Node3(192.168.0.4)
Created a keyspace 'test' with replication factor as 2.
Create KEYSPACE test WITH replication = {'class':'SimpleStrategy',
'replication_factor' : 2}
When I stop either Node2 or Node3 (one at a time and both at one time) , I am able to do the CRUD operations on the keyspace.table.
When I stop Node1 and try to update/create a row from Node4 or Node3, getting following error although Node3 and Node4 are up and running-:
All host(s) tried for query failed (tried: /192.168.0.4:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout while
trying to acquire available connection (you may want to increase the
driver number of per-host connections)))
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: /192.168.0.4:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout while
trying to acquire available connection (you may want to increase the
driver number of per-host connections)))
I am not sure how Cassandra elects a leader if a leader node dies.
So, you are using replication_factor 2, so only 2 nodes will have a replica of you keyspace (not all the 3 nodes).
My first advise is to change the RF to 3.
You have to pay attention to the consistency level you are using; If you have only 2 copies of you data (RF: 2), and you are using Consistency Level QUORUM, it will try to write the data on half of nodes + 1, in this case, all 2 nodes. So if 1 node is down, you will not be able to write/read data.
to verify where the data is replicated you could see how is the ring in you cluster. As you are using SimpleStrategy it will copy the data clockwise direction. And in your case its copied at nodes at 192.168.0.2 and 192.168.0.3.
Take a look at the concepts of replication factor: http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html
And Consistency Level: http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
Great answer about RF vs CL: https://stackoverflow.com/a/24590299/6826860
You can use this calculator to find out if your setup have a decent consistency. In your case the result is You can survive the loss of no nodes without impacting the application
I think I wasn't clear at response. The replication factor is about how many copies of your data will exists. The consistency level is how many copies your client will wait to be made before get an response from server.
Ex: All your nodes are up. The client make a CQL with CL Quorum, the server will copy the data in 2 nodes (3/2 + 1) and reply to client, in background it will copy the data at the third node as well.
In your example, if you shutdown 2 nodes of a 3 node cluster you will never achieve an QUORUM to make requests (with CL QUORUM), so you have to use consistency level ONE, once the nodes are up again, cassandra will copy the data on them. One thing that can happen is: before cassandra copy the data on other 2 nodes, the client make a request for node1 or node2 and the data is not there yet.
I have a cassandra 2.0.5 cluster setup with 3 nodes. Multiple services use the same cluster with separate keyspace. Due to the large size of the blob entry in the table, a query which goes over all the rows causes an OutOfMemory error and crashes the cluster. This is unacceptable for me as different services use the same cluster and one should not affect others.
Now, there is a way to restrict the number of rows which are loaded in memory at a time per query using the fetchSize parameter with a query with most of the drivers supporting cluster 2.0 protocol.
The default_fetch-size value is 5000, and I want to override this value to something smaller like 500 to avoid OOM error. I cannot assume all clients will use a small fetchSize while issuing an expensive query. Is there a way to do so? I cannot find any such configuration in cassandra.yaml.
Can I set this per keyspace level, so all queries to a particular keyspace will have a smaller default_fetch_size?
To set fetchsize on all statements, you can do that in QueryOptions when you build your session. Something like this,
Session session = Cluster.builder()
.addContactPoint("127.0.0.1")
.withQueryOptions(new QueryOptions().setFetchSize(100))
.build().connect();
If you want to have different fetchsize for different keyspaces then maintain session objects for each keyspace. Something like this,
Session sessionForKeyspace1 = Cluster.builder()
.addContactPoint("127.0.0.1")
.withQueryOptions(new QueryOptions().setFetchSize(100))
.build().connect("keyspace1");
Session sessionForKeyspace2 = Cluster.builder()
.addContactPoint("127.0.0.1")
.withQueryOptions(new QueryOptions().setFetchSize(200))
.build().connect("keyspace2");
What I know is you can only set fetch size of a particular statement. statement.setFetchSize(100);
Hi I'm new to Cassandra. I have a 2 node Cassandra cluster. For reasons imposed by the front end I need...
Total replication of all data on each of the two nodes.
Eventual consistent writes. So the node being written to will respond with an acknowledge to the front end straight away. Not synchronized on the replication
Can anyone tell me is this possible? Is it done in the YAML file? I know there is properties there for consistency but I don't see that any of the Partitioners suit my needs. Where can I set the replication factor?
Thanks
You set the replication factor during creation of the keyspace. So if you use (and plan for the future on using) a single data center set-up, you create the keyspace using cqlsh like so
CREATE KEYSPACE "Excalibur"
WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3};
Check out the documentation regarding the create keyspace. How this is handled internally is related to the snitch definition of the cluster and a strategy option defined per keyspace. In the case of the SimpleStrategy above, this simply assumes a ring topology of your cluster and places the data clockwise in that ring (see this).
Regarding consistency, you can very different levels of consistency for write and read operations in your client/driver during each operation:
Cassandra extends the concept of eventual consistency by offering tunable consistency―for any given read or write operation, the client application decides how consistent the requested data should be.
Read the doc
If you use Java in your clients, and the DatatStax Java driver, you can set the consistency level using
QueryOptions.setConsistencyLevel(ConsistencyLevel consistencyLevel)
"One" is the default setting.
Hope that helps