We get an error "cannot achieve consistency level QUORUM" (details below)
in following configuration:
Two datacenters with 6 nodes each, all nodes on same rack.
It works when CL is set as "Local Quorum".
Basically, as far as we use consistency level that require cross DC consistency, it fails to insert data. "Nodetool status" command shows that all 12 nodes are up and running.
What can be wrong?
Your help is much appreciated!
Thanks
Dimitry
Keyspace
CREATE KEYSPACE test6 WITH replication = {'class': 'NetworkTopologyStrategy', 'C
entralUS': '3', 'EastUs': '3'} AND durable_writes = true;
Query
INSERT INTO glsitems (itemid,itemkey) VALUES('1', 'LL');
Error
cassandra-driver-2.7.2\cassandra\cluster.py", line 3347, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve
consistency level QUORUM" info={'required_replicas':
4, 'alive_replicas':3, 'consistency': 'QUORUM'}
It could be that Cassandra thinks all nodes are in the same datacenter. In this case LOCAL_QUORUM would always work properly but not QUORUM.
Did you correctly configure the snitch ?
Snitch – For multi-data center deployments, it is important to make
sure the snitch has complete and accurate information about the
network, either by automatic detection (RackInferringSnitch) or
details specified in a properties file (PropertyFileSnitch). link
You can find which snitch is used in the cassandra yaml file, property endpoint_snitch.
Here is the datastax documentation about existing snitches with Cassandra 2.0.
Related
We use a multi-data center (DC) cassandra cluster. During write on to the cluster, I want only LOCAL DC to perform writes on its nodes as we are already routing the write requests to the desired DC only based on the source from where write is initiated. So, I want only LOCAL DC to process the write and no other DC to perform the writes on its nodes. But later on by virtue of replication among nodes across DCs, I want the written data to be replicated across DCs. Is this replication across DCs possible when I am restricting the write to only one DC in the first place.If I do not open connections to REMOTE hosts lying in different DCs during my write operation, is data replication possible amongst DCs later on. Why I definitely need replicas of data in all DCs is because during data read from cluster, we want the data to be read from any DC the read request falls on, not necessarily the LOCAL one.
Do anyone has solution to this?
You may want to use Local_Quorum consistency for writes if you want to perform them in only Local DC.
Check keyspace definition for the one you want these restriction. It should have class "Network topology" and RF in both DC. Something like this:
ALTER KEYSPACE <Keyspace_name> WITH REPLICATION =
{'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};
It states that after consistency is satisfied Cassandra will propagate the writes to another DC.
Use Quorum consistency for reads if they are not restricted to one DC but be aware that it might add bit latency because Cassandra has to read data from other data center as well.
For some test purposes I want to break a consistency of data in my test cassandra cluster, consisting of two datacenters.
I assumed that if I use a consistency level equal to LOCAL_QUORUM, or LOCAL_ONE I will achieve this. Let us say I have a cassandra node node11 belonging to DC1:
cqlsh node11
CONSISTENCY LOCAL_QUORUM;
INSERT INTO test.test (...) VALUES (...) ;
But in fact, data appears in all nodes. I can read it from the node22 belonging to the DC2 even with the consistency level LOCAL_*. I've double checked: the nodetool shows me the two datacenters and node11 certainly belongs to the DC1, while node22 belongs to the DC2.
My keyspace test is configured as follows:
CREATE KEYSPACE "test"
WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1' : 2, 'dc2' : 2};
and I have two nodes in each DC respectively.
My questions:
It seems to me that I wrongly understand the idea of these consistency level. In fact they do not prevent from writing data to the different DC's, but just ask for appearing of the data at least in the current datacenter. Is it correct understanding?
More essentially: is any way to perform such a trick and achieve such a "broken" consistency, when I have a different data stored in two datacenters within one cluster?
(At the moment I think that the only one way to achieve that - is to break the ring and do not allow nodes from one DC know anything about nodes from another DC, but I don't like this solution).
LOCAL_QUORUM, this consistency level requires a quorum of acknoledgement received from the local DC but all the data are sent to all the nodes defined in the keyspace.
Even at low consistency levels, the write is still sent to all
replicas for the written key, even replicas in other data centers. The
consistency level just determines how many replicas are required to
respond that they received the write.
https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
I don't think there is proper way to do that
This suggestion is to test scenario only to break data consistency between 2 DCs. (haven't tried but based on my understanding should work)
Write data in one DC (say DC1) with Local* consistency
Before write, keep nodes in DC2 down so DC1 will store hints as DC2 nodes are down.
Let max_hint_window_in_ms (3 hours by default - and you can reduce it) time pass so that DC1 coordinator will delete all the hints
Start DC2 nodes and query with LOCAL* query, the data from DC1 won't be present in DC2.
You can repeat these steps and insert data in DC2 with different values keeping DC1 down so same data will have different values in DC1 and DC2.
Cassandra gives me serious headache. Yesterday, everything was running fine and then I dropped a table, ran a CQLSSTableWriter which somehow threw errors about my Lucene index (for not being on classpath or the like) several times and now, every command I issue in the cqlsh is throwing errors.
CREATE KEYSPACE IF NOT EXISTS mydata WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};
takes a while and then throws:
Warning: schema version mismatch detected, which might be caused by DOWN nodes;
if this is not the case, check the schema versions of your nodes in system.local and system.peers.
OperationTimedOut: errors={}, last_host=XXX.XXX.XXX.20
After that I will create a new table and it will also throw the same error.
cqlsh:mydata> create table test (id text PRIMARY KEY, id2 text);
Warning: schema version mismatch detected, which might be caused by DOWN nodes; if this is not the case, check the schema versions of your nodes in system.local and system.peers.
OperationTimedOut: errors={}, last_host=XXX.XXX.XXX.20
last_host always shows the ip of the host where I run the cqlsh on. I have tried the same commands with different nodes too.
The keyspace and table however is still being created! The error says something about mismatching schema versions, so I made sure and ran:
nodetool describecluster
And the output of it shows that all my nodes are on the same schema. No schema mismatches. I also issued nodetool resetlocalschema before, without any luck though.
When I go ahead and insert some data into the newly created table, following error arises. Note that the insert statement does not return an error.
cqlsh:mydata> insert into test(id, id2) values('test1', 'test2');
cqlsh:mydata> select * from mydata.test ;
Traceback (most recent call last):
File "/usr/bin/cqlsh.py", line 1314, in perform_simple_statement
result = future.result()
File "/usr/share/cassandra/lib/cassandra-driver-internal-only-3.0.0-6af642d.zip/cassandra-driver-3.0.0-6af642d/cassandra/cluster.py", line 3122, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}
Note that I have one datacenter and five nodes. I do not plan to use more than one datacenter in the future. [cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.4.0 | Native protocol v4]
I have also restarted Cassandra multiple times. nodetool status shows that all nodes are up and running. Does anyone have a clue about what's going on?
I fixed this by...
dropping all tables in the keyspace
running alter keyspace mydata WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '1'}; instead of SimpleStrategy
restarting the cassandra service on all nodes
recreating all tables
runnning nodetool repair
Now I am able to insert data and query data again. Still not quite sure what was the cause of all this to be honest though.
I have 6 nodes in my cassandra cluster, all the nodes are up. My keyspace is set up as:
replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true
Doing a read from this cass cluster gives me the error:
java.io.IOException: Exception during execution of SELECT "colA", "colB", "colC", "colD" FROM "keyspacename"."tablename" WHERE token("colA") > ? AND token("colA") <= ? LIMIT 1 ALLOW FILTERING: Not enough replica available for query at consistency LOCAL_ONE (1 required but only 0 alive)
All my nodes are up with a replication factor of 1.. then what is causing this problem?
Also I can cqlsh and do "select" and "insert" in this table.
Anyone know whats going on?
SimpleStrategy: Use for a single data center only. If you ever intend more than one data center, use the NetworkTopologyStrategy.
Or try running the same query with Consistency ONE, ALL, ANY, QUORUM. I mean not with LOCAL_*
Hi I'm new to Cassandra. I have a 2 node Cassandra cluster. For reasons imposed by the front end I need...
Total replication of all data on each of the two nodes.
Eventual consistent writes. So the node being written to will respond with an acknowledge to the front end straight away. Not synchronized on the replication
Can anyone tell me is this possible? Is it done in the YAML file? I know there is properties there for consistency but I don't see that any of the Partitioners suit my needs. Where can I set the replication factor?
Thanks
You set the replication factor during creation of the keyspace. So if you use (and plan for the future on using) a single data center set-up, you create the keyspace using cqlsh like so
CREATE KEYSPACE "Excalibur"
WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3};
Check out the documentation regarding the create keyspace. How this is handled internally is related to the snitch definition of the cluster and a strategy option defined per keyspace. In the case of the SimpleStrategy above, this simply assumes a ring topology of your cluster and places the data clockwise in that ring (see this).
Regarding consistency, you can very different levels of consistency for write and read operations in your client/driver during each operation:
Cassandra extends the concept of eventual consistency by offering tunable consistency―for any given read or write operation, the client application decides how consistent the requested data should be.
Read the doc
If you use Java in your clients, and the DatatStax Java driver, you can set the consistency level using
QueryOptions.setConsistencyLevel(ConsistencyLevel consistencyLevel)
"One" is the default setting.
Hope that helps