When I am trying to execute the below query, I am always getting QueryTimeOutException,
Exception is,
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 0 replica responded)
Query is,
SELECT * FROM my_test.my_table WHERE key_1 = 101 ORDER BY key_2 ASC LIMIT 25;
I am using cassandra version 2.1.0 with 3 nodes, Single DC with replication of 3, cassandra.yaml has all default values and I am having following keyspace and table as schema,
CREATE KEYSPACE my_test
WITH REPLICATION = {
'class' : 'SimpleStrategy',
'replication_factor' : 3
};
CREATE TABLE my_test.my_table (
key_1 bigint,
key_2 bigint,
key_3 text,
key_4 text,
key_5 text,
key_6 text,
key_7 text,
key_8 text,
key_9 text,
key_10 text,
key_11 timestamp,
PRIMARY KEY (key_1, key_2)
);
Currently the table has around 39000 records but initially it has 50000 records, 11000 records has been deleted for some business logic.
One of the solution to avoid such exception is to increase query read time out, But my schema and query are more direct why should I increase my read time out?
Since In my query I have given the partition key (key_1) so it should reach the destination exactly, after that I had specified the start range of parition key,
So it should retrieve with a maximum time of 2seconds, but is not so. But the below query is working fine and retrieved the results less than 1 seconds (Difference is, ASC is not working and DESC is working)
SELECT * FROM my_test.my_table WHERE key_1 = 101 ORDER BY key_2 DESC LIMIT 25;
Again as per schema the cluster key default order is ASC, So retrieving the data in ASC should be faster than DESC order as per cassandra documentation.
But it is reverse in my case.
Again some clues, The following are the queries that has been tried through CQLSH.
The following query is working and retrieved the results less than 1 seconds
SELECT * FROM my_test.my_table WHERE key_1 = 101 AND key_2 > 1 AND key_2 < 132645 LIMIT 1;
But, the following query is not working and throws time out exception,
SELECT * FROM my_test.my_table WHERE key_1 = 101 AND key_2 > 1 AND key_2 < 132646 LIMIT 1;
But, the following queries are working, and retrieved results less than 1 seconds
SELECT * FROM my_test.my_table WHERE key_1 = 101 AND key_2 = 132644;
SELECT * FROM my_test.my_table WHERE key_1 = 101 AND key_2 = 132645;
SELECT * FROM my_test.my_table WHERE key_1 = 101 AND key_2 = 132646;
SELECT * FROM my_test.my_table WHERE key_1 = 101 AND key_2 = 132647;
Strange behaviour any help would be appreciated.
For each key_1 there will be around 1000000 key_2.
And this is what happens when you take the 2 billion cells per partition limit, and try to use all of it. I know I've answered plenty of posts here before by acknowledging that there is a hard limit of 2 billion cells per partition, your (very) wide row will become ungainly and probably time-out long before that. This is what I believe you are seeing.
The solution here, is a technique called "bucketing." Basically, you have to find an additional key to partition your data by. Too many CQL rows are being written to the same data partition, and bucketing would help bring the ratio of partition to clustering keys back to a sane level.
The logical way to go about bucketing, is with a time element. I see your last key is a timestamp. I don't know how many rows each key_1 gets in a day, but let's say that you only get a few thousand every month. In that case, I would create an additional partition key of month_bucket:
CREATE TABLE my_test.my_table (
key_1 bigint,
key_2 bigint,
...
key_11 timestamp,
month_bucket text,
PRIMARY KEY ((key_1,month_bucket) key_2)
);
That would allow you to support a query like this:
SELECT * FROM my_test.my_table
WHERE key_1 = 101 AND month_bucket = '201603'
AND key_2 > 1 AND key_2 < 132646 LIMIT 1;
Again, bucketing on month is just an example. But basically, you need to find an additional column to partition your data on.
Issue got resolved after restarting all the 3 cassandra servers. I don't know what the hell makes trouble.. Since it is in production server couldn't able to get exact Root Cause.
Related
Use case: Find maximum counter value in a specific id range
I want to create a table with these columns: time_epoch int, t_counter counter
The frequent query is:
select time_epoch, MAX t_counter where time_epoch >= ... and time_epoch < ...
This is to find the counter in specific time range. Planning to make time_epoch as primary key. I am not able to query the data. It is always asking for ALLOW FILTERING. Since its a very costly function, We dont want to use it.
How to design the table and query for the use case.
Let's assume that we can "bucket" (partition) your data by day, assuming that enough write won't happen in a day to make the partitions too large. Then, we can cluster by time_epoch in DESCending order. With time based data, storing data in descending order often makes the most sense (as business reqs usually care more about the most-recent data).
Therefore, I'd build a table like this:
CREATE TABLE event_counter (
day bigint,
time_epoch timestamp,
t_counter counter,
PRIMARY KEY(day,time_epoch))
WITH CLUSTERING ORDER BY (time_epoch DESC);
After inserting a few rows, the clustering order becomes evident:
> SELECT * FROM event_counter ;
WHERE day=20210219
AND time_epoch>='2021-02-18 18:00'
AND time_epoch<'2021-02-19 8:00';
day | time_epoch | t_counter
----------+---------------------------------+-----------
20210219 | 2021-02-19 14:09:21.625000+0000 | 1
20210219 | 2021-02-19 14:08:32.913000+0000 | 2
20210219 | 2021-02-19 14:08:28.985000+0000 | 1
20210219 | 2021-02-19 14:08:05.389000+0000 | 1
(4 rows)
Now SELECTing the MAX t_counter in that range should work:
> SELECT day,max(t_counter) as max
FROM event_counter
WHERE day=20210219
AND time_epoch>='2021-02-18 18:00'
AND time_epoch<'2021-02-19 09:00';
day | max
----------+-----
20210219 | 2
Unfortunately there is no better way. Think about it.
If you know cassandra architecture then you would know that your data is spread across multiple nodes based on primary key. only way to filter on values from primary key would be to transverse each node which is essentially what "ALLOW FILTERING" is done.
I've got 3 nodes in 2 datacenters. On each node I'm using the same cqlshrc file having just the following lines:
[connection]
request_timeout = 3600
Replication strategy is as follows :
class:NetworkTopologyStrategy, 'dc1' :2, 'dc2':1
I've inserted more than 100 000 rows in the database. However when I execute
select count(*) from table, I get "operation timed out" in 2 of the 3 nodes. i.e. the query is successful in one node.
Why is the query unsuccessful in 2 out of 3 nodes despite each having same cqlshrc file?
OS: RHEL 6
Cassandra:3.0.14
In cassandra count(*) is a very costly operation, need to scan all the row from all the node just to give you the count and can generate timeout exception.
So Instead of using count(*) maintain a counter table for example :
CREATE TABLE page_view_counts (
counter_value counter,
url_name varchar,
page_name varchar,
PRIMARY KEY (url_name, page_name)
);
Whenever a new row inserted into base table increment the count plus one
UPDATE page_view_counts
SET counter_value = counter_value + 1
WHERE url_name = 'stackoverflow.com' AND page_name = 'questions';
Now to get the count just try like below
SELECT * FROM page_view_counts
WHERE url_name = 'stackoverflow.com' AND page_name = 'questions';
I'm trying to store and retrieve last active sensors by this schema:
CREATE TABLE last_signals (
section bigint,
sensor bigint,
time bigint,
PRIMARY KEY (section, sensor)
);
Row of this table will be updated every few seconds and in the result hot sensors will remain in memtable. But what will happen when I get a run a query like this:
SELECT * FROM last_signals
WHERE section = ? AND time > ?
Limit ?
ALLOW FILTERING;
And the result will be something like this (Ordered by clustering key):
sect | sens | time
------+------+------
1 | 1 | 4
1 | 2 | 3
1 | 4 | 2
1 | 5 | 9
The first Question: Is this result guaranteed to be the same in all version? (I'm using 3.7) and the next one is that how I can change this behavior (with query option, modeling or etc.). Indeed I need to get last writes first without considering clustring-keys order. I think in this case my reads will be much faster.
I don't think there is any way to guarantee order besides using clustering keys. Thus your ALLOW FILTERING query is potentially costly and may even time out. You could consider the following schema:
CREATE TABLE last_signals_by_time (
section bigint,
sensor bigint,
time bigint,
dummy bool,
PRIMARY KEY ((section, sensor), time)
) WITH CLUSTERING ORDER BY (time DESC);
Instead of updates do inserts with TTL so that you do not have to clean up old entries manually. (The dummy field is needed in order for TTL to work)
And then just run your read queries per section/sensors in parallel:
SELECT * FROM last_signals_by_time
WHERE section = ? AND sensor = ?
LIMIT 1;
I'm trying to display the latest values from a list of sensors. The list should also be sortable by the time-stamp.
I tried two different approaches. I included the update time of the sensor in the primary key:
CREATE TABLE sensors (
customerid int,
sensorid int,
changedate timestamp,
value text,
PRIMARY KEY (customerid, changedate)
) WITH CLUSTERING ORDER BY (changedate DESC);
Then I can select the list like this:
select * from sensors where customerid=0 order by changedate desc;
which results in this:
customerid | changedate | sensorid | value
------------+--------------------------+----------+-------
0 | 2015-07-10 12:46:53+0000 | 1 | 2
0 | 2015-07-10 12:46:52+0000 | 1 | 1
0 | 2015-07-10 12:46:52+0000 | 0 | 2
0 | 2015-07-10 12:46:26+0000 | 0 | 1
The problem is, I don't get only the latest results, but all the old values too.
If I remove the changedate from the primary key, the select fails all together.
InvalidRequest: code=2200 [Invalid query] message="Order by is currently only supported on the clustered columns of the PRIMARY KEY, got changedate"
Updating the sensor values is also no option:
update overview set changedate=unixTimestampOf(now()), value = '5' where customerid=0 and sensorid=0;
InvalidRequest: code=2200 [Invalid query] message="PRIMARY KEY part changedate found in SET part"
This fails because changedate is part of the primary key.
Is there any possible way to store only the latest values from each sensor and also keep the table ordered by the time-stamp?
Edit:
In the meantime I tried another approach, to only storing the latest value.
I used this schema:
CREATE TABLE sensors (
customerid int,
sensorid int,
changedate timestamp,
value text,
PRIMARY KEY (customerid, sensorid, changedate)
) WITH CLUSTERING ORDER BY (changedate DESC);
Before inserting the latest value, I would delete all old values
DELETE FROM sensors WHERE customerid=? and sensorid=?;
But this fails because changedate is NOT part of the WHERE clause.
The problem is, I don't get only the latest results, but all the old values too.
Since you are storing in a CLUSTERING ORDER of DESC, it will always be very easy to get the latest records, all you need to do is add 'LIMIT' to your query, i.e.:
select * from sensors where customerid=0 order by changedate desc limit 10;
Would return you at most 10 records with the highest changedate. Even though you are using limit, you are still guaranteed to get the latest records since your data is ordered that way.
If I remove the changedate from the primary key, the select fails all together.
This is because you cannot order on a column that is not the clustering key(s) (the secondary part of the primary key) except maybe with a secondary index, which I would not recommend.
Updating the sensor values is also no option
Your update query is failing because it is not legal to include part of the primary key in 'set'. To make this work all you need to do is update your query to include changedate in the where clause, i.e.:
update overview set value = '5' and sensorid = 0 where customerid=0 and changedate=unixTimestampOf(now())
Is there any possible way to store only the latest values from each sensor and also keep the table ordered by the time-stamp?
You can do this by creating a separate table named 'latest_sensor_data' with the same table definition with exception to the primary key. The primary key will now be 'customerid, sensorid' so you can only have 1 record per sensor. The process of creating separate tables is called denormalization and is a common use pattern particularly in Cassandra data modeling. When you insert sensor data you would now insert data into both 'sensors' and 'latest_sensor_data'.
CREATE TABLE latest_sensor_data (
customerid int,
sensorid int,
changedate timestamp,
value text,
PRIMARY KEY (customerid, sensorid)
);
In cassandra 3.0 'materialized views' will be introduced which will make this unnecessary as you can use materialized views to accomplish this for you.
Now doing the following query:
select * from latest_sensor_data where customerid=0
Will give you the latest value for every sensor for that customer.
I would recommend renaming 'sensors' to 'sensor_data' or 'sensor_history' to make it more clear what the data is. Additionally you should change the primary key to 'customerid, changedate, sensorid' as that would allow you to have multiple sensors at the same date (which seems possible).
Your first approach looks reasonable. If you add "limit 1" to your query, you would only get the latest result, or limit 2 to see the latest 2 results, etc.
If you want to automatically remove old values from the table, you can specify a TTL (Time To Live) for data points when you do the insert. So if you wanted to keep data points for 10 days, you could do this by adding "USING TTL 864000" on your insert statements. Or you could set a default TTL for the entire table.
CQL Execution [returns instantly, assuming uses clustering key index]:
cqlsh:stats> select count(*) from events where month='2015-04' and day = '2015-04-02';
count
-------
5447
Presto Execution [takes around 8secs]:
presto:default> select count(*) as c from cassandra.stats.events where month = '2015-04' and day = timestamp '2015-04-02';
c
------
5447
(1 row)
Query 20150228_171912_00102_cxzfb, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:08 [147K rows, 144KB] [17.6K rows/s, 17.2KB/s]
Why should presto get to process 147K rows when cassandra itself responds with just 5447 rows for the same query [I tried select * too]?
Why presto is not able to use the clustering key optimization?
I tried all possible values like timestamp, date, different formats of dates. Not able to see any effect on number of rows being fetched.
CF Reference:
CREATE TABLE events (
month text,
day timestamp,
test_data text,
some_random_column text,
event_time timestamp,
PRIMARY KEY (month, day, event_time)
) WITH comment='Test Data'
AND read_repair_chance = 1.0;
Added event_timestamp too as a constraint in response to Dain's answer
presto:default> select count(*) from cassandra.stats.events where month = '2015-04' and day = timestamp '2015-04-02 00:00:00+0000' and event_time = timestamp '2015-04-02 00:00:34+0000';
_col0
-------
1
(1 row)
Query 20150301_071417_00009_cxzfb, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:07 [147K rows, 144KB] [21.3K rows/s, 20.8KB/s]
The Presto engine will pushdown simple WHERE clauses like this to a connector (you can see this in the Hive connector), so the question is, why does the Cassandra connector not take advantage of this. To see why, we'll have to look at the code.
The pushdown system first interacts with connectors in the ConnectorSplitManager.getPartitions(ConnectorTableHandle, TupleDomain) method, so looking at the CassandraSplitManager, I see it is delegating the logic to getPartitionKeysSet. This method looks for a range constraint (e.g., x=33 or x BETWEEN 1 AND 10) for every column in the primary key, so in your case, you would need to add a constraint on event_time.
I don't know why the code insists on having a constraint on every column in the primary key, but I'd guess that it is a bug. It should be easy to tweak this code to remove that constraint.