we use Cassandra 3.11.1, com.datastax.oss java-driver-core 4.13.0 and Java 13.
we have multiple read micro-services which read data from Cassandra, but got this error:
Cassandra timeout during read query at consistency ALL (8 responses were required but only 7 replica responded)
our queries are mostly just select by primary key, some of queries even specify to set consistency level to local_quorum, wonder why we still ran into this issue, sample queries:
QueryBuilder.selectFrom(Email.TABLE_NAME)
.all()
.whereColumn(Email.COLUMN_EMAIL_ADDRESS)
.isEqualTo(bindMarker())
.limit(bindMarker())
.build()
.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM)
QueryBuilder.selectFrom(Account.TABLE_NAME)
.columns(
AccountDataObject.COLUMN_ACCOUNT_ID,
AccountDataObject.COLUMN_EMAIL_ADDRESS)
.whereColumn(Account.COLUMN_ACCOUNT_ID)
.isEqualTo(bindMarker())
.build()
Related
I have three nodes Syclla cluster. I have keyspace which has 3 replication factor. I use datastax-java-driver 3.6.0 version and Scylla 3.0.0 version. When I tried to read my data with consistency level = LOCAL_QUORUM, I get error message below which is impossible in my opinion. As far as If I use LOCAL_QUORUM 2 nodes is enough for 3 replication factor.
Is it bug or am I missing something?
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (3 responses were required but only 2 replica responded)
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (3 responses were required but only 2 replica responded)
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (3 responses were required but only 2 replica responded)
What happened here is that Scylla chose to do probabilistic read repair and detected mismatch before CL was reached. At this point it started to do repair between all three replicas and failed to read from all of them (either due to an overload or one node crashed/was restarted while operation was ongoing). You can disable probabilistic read repair to avoid it.
I have a table with huge tombstones.So when i performed a spark job (which reads) on that specific table, it gave result without any issues. But i executed same query using cqlsh it gave me error because huge tombstones present in that table.
Cassandra failure during read query at consistency one(1 replica
needed but o replicas responded ,1 failed
I know tombstones should not be there, i can run repair to avoid them , but apart from that why spark succeeded and cqlsh failed. They both use same sessions and queries.
How spark-cassandra connector works? is it different than cqlsh?
Please let me know .
thank you.
The Spark Cassandra Connector is different to cqlsh in a few ways.
It uses the Java Driver and not the python Driver
It has significantly more lenient retry policies
It full table scans by breaking the request up into pieces
Any of these items could be contributing to why it would work in SCC and not in CQLSH.
Environment :
5 machines Cassandra 2.1.15 cluster.
RF = 3, CL = QUORUM
1 machine goes down for more than 3 hours, without the possibility to bring it back
Decide to do noderemove and replace it :
The problem i saw is this :
Did heavy load over the node :
cassandra-stress write n=50000000 cl=QUORUM -rate threads=1000 -node 192.168.0.171,192.168.0.177,192.168.0.178,192.168.0.179,192.168.0.220
At one time gave me the error :
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (3 replica were required but only 2 acknowledged the write)
According to my knowledge QUORUM = RF/2+1 rounded down => 2 replicas should be acquired.
Is this some kind of a bug!? Does it have some kind of negative impact?
Are you certain that cassandra-stress is using your keyspace? If you have not configured it to do so, it must be using default keyspace with replications as many as number of nodes. Try using -schema switch for cassandra-stress.
https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html
We have our Cassandra cluster running on AWS EC2 with 4 nodes in the ring. We have face data inconsistency issue. We changed consistency level two while using "cqlsh" shell, the data inconsistency issue has been solved.
But we dont know "How to set consistency level on Cassandra cluster?"
Consistency level can be set at per session or per statement basis. You will need to check the consistency level of writes and reads, to get a strong consistency your R + W ( read consistency + write consistency ) should be greater than your replication factor.
If you are using Java Driver, you can set default consistency at cluster level using "Cluster.Builder.withQueryOptions()" method.
http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/Cluster.Builder.html#withQueryOptions-com.datastax.driver.core.QueryOptions-
What will be an ideal way to query cassandra by a partition key using the Spark Connector. I am using where to pass in the key but that causes cassandra to add ALLAOW FILTERING under the hood which in turn causes timeouts.
current set up :
csc.cassandraTable[DATA]("schema", "table").where("id =?", "xyz").map( x=> print(x))
here id is the partition(not primary) key
I have a composite primary key and using only the partition key for query
Update :
yes , I am getting an exception with this :
Cassandra failure during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded, 1 failed)
none of my partitions have more than a 1000 records and I am running a single cassandra node
ALLOW FILTERING is not going to affect your query if you use a where clause on the entire partition key. If the query is timing out it may mean your partition is just very large or the full partition key was not specified
EDIT:
Cassandra failure during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded, 1 failed)
Means that the your queries are being sent to machines which do not have a replica of the data you are looking for. Usually this means that the replication of the keyspace is not set correctly or that the connection host is incorrect. The LOCAL part of LOCAL_ONE means that the query is only allowed to succeed if the data is available on the LOCAL_DC.
With this in mind you have 3 options
Change the initial connection target of your queries
Change the replication of your keyspace
Change the consistency level of your queries
Since you only have 1 machine, Changing the replication of your keyspace is probably the right thing to do.