Cassandra ReadTimeOuts - cassandra
I have problem with Cassandra ReadTimeOuts.
Scenario:
3 GB data loaded to Cassandra,
9 nodes of Cassandra's within 1 DataCenter,
Replication equals 3,
Consistency level equals 1,
Cassandra version 2.2.9
link to cassandra.yaml
https://pastebin.com/x0bF7nLf
Tests:
For Testing I am using jmeter plug-in for Cassandra.
Request is a select with condition for row ID within provided list of ID's. List always contains 100 ID's. Each request should always return 100 rows (all ID's are in database).
ID's are random so cache role is reduced.
Sample select:
select * from price.item_vat_posting_group where no in ('B7B7A6','B2DD05','A34751','B4BC7D','C0BB53','D07DCB','C03716','BB99DF','A975C2','C2AE27','AF621C','242448','B30CDA','508336','B44D6B','D07422','AC44EA','C6F34D','9B25AC','C4CF12','AC25BD','C3D9C7','AE7DB2','C5E03E','BF7AC1','B499B5','A7787E','645180','A9BEFE','AFFEA4','A88955','D95B50','B0F9FC','C09174','253953','9ED9CA','CAF896','536951','214502','427776','DA14CB','422282','A4B10A','C56BF5','B373E0','D171EF','C70607','B350AB','9D809B','586563','BF6308','A4BF5A','C42716','C3261C','C45B79','C6FE55','D1F0D4','C483B5','A67D59','DC5898','9BACAD','D9C6B0','D17DAE','D8D4F3','A05946','BBEBA8','A87B37','A13E97','BB7099','A3FC26','C461DF','309810','BF6306','D07603','C59F70','C5906C','A515ED','B50056','A8390E','A0CCC7','BF2713','C6EC7D','D7EB9D','A5D5EB','984076','D88F44','257058','D61635','D40CDE','B0A347','B7617F','D6277E','B4286F','C41F99','D84232','DC1636','BFF15D','DD0972','9B3138');
Scenario 1.
While sending requests by 100 threads in 10 minutes time Cassandra has 5% ReadTimeOuts for total number of handled requests.
Average request time is 100 ms.
Processor usage on each Cassandra node is between 40% - 50%.
Scenario 2.
While sending requests by 4 threads in 24 hours time, about 10 ReadTimeOuts occurs per 100 000 requests.
Processor usage on each Cassandra node is 5%.
In both scenarios Garbage Collector works less then 300 ms.
Error message:
Cassandra time-out during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded)
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_ONE (1 responses were required but only 0 replica responded)
at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:69)
at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258)
Some statistics:
QUESTION:
Is that typical for Cassandra or am I doing something wrong ?
You are using in query, In query put a lot of presser in the coordinator node. When you execute in query, this means that you’re waiting on this single coordinator node to give you a response, it’s keeping all those queries and their responses in the heap, and if one of those queries fails, or the coordinator fails, you have to retry the whole thing.
Instead of using in query use executeAsync with separate query for each no. Now if one query failed retry requires only one small fast query.
Or
Change your data model so that you can specify partition key when using in query.
Note : To much executeAsync at a time can also put presser on your cluster. Check this link https://stackoverflow.com/a/30526719/2320144
More : https://lostechies.com/ryansvihla/2014/09/22/cassandra-query-patterns-not-using-the-in-query-for-multiple-partitions/
Your query isn't efficient because you scan lot of partition.
Each partition stored in different node.
You should scan one or less 10 partitions with range condition.
Change your data model, check theses links :
https://www.datastax.com/dev/blog/the-most-important-thing-to-know-in-cassandra-data-modeling-the-primary-key
https://www.datastax.com/dev/blog/a-deep-look-to-the-cql-where-clause
Related
Cassandra sometimes skips records in SELECT query
My setup is: cassandra 1.2.19 single datacenter cluster with 4 nodes NetworkTopologyStrategy with replication factor of 3 consistency level of writes to the db is set to LOCAL_QUORUM I am trying to iterate all records in a given table and I do so with some legacy application code which fetches the data in batches with consecutive select queries of this type: SELECT * FROM records WHERE TOKEN(partition_key) > last_partition_key_of_previous_batch LIMIT 1000; The problem is that sometimes some records are skipped. I also noticed that those skipped records are old, added months ago to the database. All of the select queries are executed with consistency level ONE. Is it possible that this is the cause? From what I understood about consistency levels when the consistency level for reads is ONE, only one node is asked to execute the query. Is it possible that sometimes the node that executes the query does not contain all the records and that's why sometimes some records are missing?
Changing the consistency level of the query to QUORUM fixed the issue.
Apache Cassandra Reading explanation
I am currently managing a percona xtradb cluster composed by 5 nodes, that hadle milions of insert every day. Write performance are very good but reading is not so fast, specially when i request a big dataset. The record inserted are sensors time series. I would like to try apache cassandra to replace percona cluster, but i don't understand how data reading works. I am looking for something able to split query around all the nodes and read in parallel from more than one node. I know that cassandra sharding can have shard replicas. If i have 5 nodes and i set a replica factor of 5, does reading will be 5x faster?
Cassandra read path The read request initiated by a client is sent over to a coordinator node which checks the partitioner what are the replicas responsible for the data and if the consistency level is met. The coordinator will check is it is responsible for the data. If yes, will satisfy the request. If no, it will send the request to fastest answering replica (this is determined using the dynamic snitch). Also, a request digest is sent over to the other replicas. The node will compare the returning data digests and if all are the same and the consistency level has been met, the data is returned from the fastest answering replica. If the digests are not the same, the coordinator will issue some read repair operations. On the node there are a few steps performed: check row cache, check memtables, check sstables. More information: How is data read? and ReadPathForUsers. Load balancing queries Since you have a replication factor that is equal to the number of nodes, this means that each node will hold all of your data. So, when a coordinator node will receive a read query it will satisfy it from itself. In particular(if you would use a LOCAL_ONE consistency level, the request will be pretty fast). The client drivers implement the load balancing policies, which means that on your client you can configure how the queries will be spread around the cluster. Some more reading - ClientRequestsRead
If i have 5 nodes and i set a replica factor of 5, does reading will be 5x faster? No. It means you will have up to 5 copies of the data to ensure that your query can be satisfied when nodes are down. Cassandra does not divide up the work for the read. Instead it tries to force you to design your data in a way that makes the reads efficient and fast.
Best way to read cassandra is by making sure that each query you generate hits cassandra partition. Which means the first part of your simple primary(x,y,z) key and first bracket of compound ((x,y),z) primary key are provided as query parameters. This goes back to cassandra table design principle of having a table design by your query needs. Replication is about copies of data and Partitioning is about distributing data. https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/archPartitionerAbout.html some references about cassandra modelling, https://www.datastax.com/dev/blog/the-most-important-thing-to-know-in-cassandra-data-modeling-the-primary-key https://www.datastax.com/dev/blog/basic-rules-of-cassandra-data-modeling it is recommended to have 100 MB partitions but not compulsory. You can use cassandra-stress utility to have look report of how your reads and writes look.
Cassandra concurrent read and write
I am trying to understand the Cassandra concurrent read and writes. I come across the property called concurrent_reads (Defaults are 8) A good rule of thumb is 4 concurrent_reads per processor core. May increase the value for systems with fast I/O storage So as per the definition, Correct me If am wrong, 4 threads can access the database concurrently. So let's say I am trying to run the following query, SELECT max(column1) from 'testtable' WHERE duration = 'month'; I am just trying to execute this query, What will be the use of concurrent read in executing this query?
Thats how many active reads can run at a single time per host. This is viewable if you type nodetool tpstats under the read stage. If the active is at pegged at the number of concurrent readers and you have a pending queue it may be worth trying to increase this. Its pretty normal for people to have this at ~128 when using decent sized heaps and SSDs. This is very hardware dependent so defaults are conservative. Keep in mind that the activity on this thread is very fast, usually measured in sub ms but assuming they take 1ms even with only 4, given little's law you have a maximum of 4000 (local) reads per second per node max (1000/1 * 4), with RF=3 and quorum consistency that means your doing a minimum of 2 reads per request so can divide in 2 to think of a theoretical (real life is ickier) max throughput. The aggregation functions (ie max) are processed on the coordinator, after fetching the data of the replicas (each doing a local read and sending response) and are not directly impacted by the concurrent reads since handled in the native transport and request response stages.
From cassandra 2.2 onward, the standard aggregate functions min, max, avg, sum, count are built-in. So, I don't think concurrent_reads will have any effect on your query.
Select All Performance in Cassandra
I'm current using DB2 and planning to use cassandra because as i know cassandra have a read performance greater than RDBMS. May be this is a stupid question but I have experiment that compare read performance between DB2 and Cassandra. Testing with 5 million records and same table schema. With query SELECT * FROM customer. DB2 using 25-30s and Cassandra using 40-50s. But query with where condition SELECT * FROM customer WHERE cusId IN (100,200,300,400,500) DB2 using 2-3s and Cassandra using 3-5ms. Why Cassandra faster than DB2 with where condition? So i can't prove which database is greater with SELECT * FROM customer right? FYI. Cassandra: RF=3 and CL=1 with 3 nodes each node run on 3 computers (VM-Ubuntu) DB2: Run on windows Table schema: cusId int PRIMARY KEY, cusName varchar
If you look at the types of problems that Cassandra is good at solving, then the reasons behind why unbound ("Select All") queries suck become quite apparent.
Cassandra was designed to be a distributed data base. In many Cassandra storage patterns, the number of nodes is greater than the replication factor (I.E., not all nodes contain all of the data). Therefore, limiting the number of network hops becomes essential to modeling high-performing queries. Cassandra performs very well with specific queries (which utilize the partition/clustering key structure), because it can quickly locate the node primarily responsible for the data.
Unbound queries (A.K.A. multi-key queries) incur the extra network time because a coordinator node is required. So one node acts as the coordinator, queries all other nodes, collates data, and returns the result set. Specifying a WHERE clause (with at least a partition key) and while using a "Token Aware" load balancing policy, performs well for two reasons:
A coordinator node is not required.
The node primarily responsible for the range is queried, returning the result set in a single netowrk hop.
tl;dr;
Querying Cassandra with an unbound query, causes it to incur a lot of extra processing and network time that it normally wouldn't have to do, had the query been specified with a WHERE clause.
Even as a troublesome query like a no-condition range query, 40-50s is pretty extreme for C*. Is the coordinator hitting GCs with the coordination? Can you include code used for your test? When you make a select * vs millions of records, it wont fetch them all at once, it will grab the fetchSize at a time. If your just iterating through this, the iterator will actually block even if you used executeAsync initially. This means that every 10k (default) records it will issue a new query that you will block on. The serialized nature of this will take time just from a network perspective. http://docs.datastax.com/en/developer/java-driver/3.1/manual/async/#async-paging explains how to do it in a non-blocking way. You can use this to to kick off the next page fetch while processing the current which would help. Decreasing the limit or fetch size could also help, since the coordinator may walk token ranges (parallelism is possible here but its heuristic is not perfect) one at a time until it has read enough. If it has to walk too many nodes to respond it will be slow, this is why empty tables can be very slow to do a select * on, it may serially walk every replica set. With 256 vnodes this can be very bad.
Cassandra read performance degrade as we increase data on nodes
DB used: Datastax cassandra community 3.0.9 Cluster: 3 x (8core 15GB AWS c4.2xlarge) with 300GB io1 with 3000iops. Write consistency: Quorum , read consistency: ONE Replication factor: 3 Problem: I loaded our servers with 50,000 users and each user had 1000 records initially and after sometime, 20 more records were added to each users. I wanted to fetch the 20 additional records that were added later(Query : select * from table where userID='xyz' and timestamp > 123) here user_id and timestamp are part of primary key. It worked fine when I had only 50,000 users. But as soon as I added another 20GB of dummy data, the performance for same query i.e. fetch 20 additional records for 50,000 users dropped significantly. Read performance is getting degraded with increase in data. As far as I have read, this should not have happened as keys get cached and additional data should not matter. what could be possible cause for this? CPU and RAM utilisation is negligible and I cant find out what is causing the query time to increase. I have tried changing compaction strategy to "LeveledCompaction" but that didn't work either. EDIT 1 EDIT 2 Heap size is 8GB. The 20GB data is added in a way similar to the way in which the initial 4GB data was added (the 50k userIDs) and this was done to simulate real world scenario. "userID" and "timestamp" for the 20GB data is different and is generated randomly. Scenario is that I have 50k userIDs with 1020 rows where 1000 rows were added first and then additional 20 rows were added after some timestamp, I am fetching these 20 messages. It works fine if only 50k userIDs are present but once I have more userIDs (additional 20GB) and I try to fetch those same 20 messages (for initial 50k userIDs), the performance degrades. EDIT 3 cassandra.yaml
Read performance is getting degraded with increase in data. This should only happen when your add a lot of records in the same partition. From what I can understand your table may looks like: CREATE TABLE tbl ( userID text, timestamp timestamp, .... PRIMARY KEY (userID, timestamp) ); This model is good enough when the volume of the data in a single partition is "bound" (eg you have at most 10k rows in a single partition). The reason is that the coordinator gets a lot of pressure when dealing with "unbound" queries (that's why very large partitions are a big no-no). That "rule" can be easily overlooked and the net result is an overall slowdown, and this could be simply explained as this: C* needs to read more and more data (and it will all be read from one node only) to satisfy your query, keeping busy the coordinator, and slowing down the entire cluster. Data grow usually means slow query response, and after a certain threshold the infamous read timeout error. That being told, it would be interesting to see if your DISK usage is "normal" or something is wrong. Give it a shot with dstat -lrvn to monitor your servers. A final tip: depending on how many fields you are querying with SELECT * and on the amount of retrieved data, being served by an SSD may be not a big deal because you won't exploit the IOPS of your SSDs. In such cases, preferring an ordinary HDD could lower the costs of the solution, and you wouldn't incur into any penalty.