I created a table that has timestamps in it but when I try to Cluster Order By the timestamp variable, it is not ordered properly.
To create the table I wrote:
CREATE TABLE videos_by_tag (
tag text,
video_id uuid,
added_date timestamp,
title text,
PRIMARY KEY ((tag), added_date, video_id))
WITH CLUSTERING ORDER BY (added_date ASC);
And the output I got when doing a SELECT * FROM videos_by_tag is:
tag | added_date | video_id | title
-----------+---------------------------------+--------------------------------------+------------------------------
datastax | 2013-04-16 00:00:00.000000+0000 | 5645f8bd-14bd-11e5-af1a-8638355b8e3a | What is DataStax Enterprise?
datastax | 2013-10-16 00:00:00.000000+0000 | 4845ed97-14bd-11e5-8a40-8338255b7e33 | DataStax Studio
cassandra | 2012-04-03 00:00:00.000000+0000 | 245e8024-14bd-11e5-9743-8238356b7e32 | Cassandra & SSDs
cassandra | 2013-03-17 00:00:00.000000+0000 | 3452f7de-14bd-11e5-855e-8738355b7e3a | Cassandra Intro
cassandra | 2014-01-29 00:00:00.000000+0000 | 1645ea59-14bd-11e5-a993-8138354b7e31 | Cassandra History
(5 rows)
As you can see the dates are out of order. There is a 2012 year value in the middle of the output.
You can fine-tune the display order using the ORDER BY clause. The partition key must be defined in the WHERE clause and the ORDER BY clause defines the clustering column to use for ordering.
Example:
SELECT * FROM videos_by_tag
WHERE tag = 'datastax' ORDER BY added_date ASC;
This is a very common misconception in Cassandra. The data is in fact ordered correctly in the sample data you posted.
The CLUSTERING ORDER applies to the sort order of the rows within a partition -- NOT across ALL partitions.
Using the example you posted, the clustering column added_date is correctly sorted in ascending order for the partition tag = 'datastax':
tag | added_date
-----------+---------------------------------
datastax | 2013-04-16 00:00:00.000000+0000
datastax | 2013-10-16 00:00:00.000000+0000
Similarly, added_date is sorted in ascending order for tag = 'cassandra':
tag | added_date
-----------+---------------------------------
cassandra | 2012-04-03 00:00:00.000000+0000
cassandra | 2013-03-17 00:00:00.000000+0000
cassandra | 2014-01-29 00:00:00.000000+0000
Like I said, the sort order only applies to rows within a partition.
It would be impossible to sort all rows in all partitions because such task does not scale. Imagine if you had billions of partitions in the table across hundreds of nodes. Every time you inserted a new row to any partition, Cassandra has to do a full table scan to sort the data and it just wouldn't make sense to do so. Cheers!
Related
I’m designing a new table in Cassandra
create table student (
studentid text PRIMARY KEY,
department text,
major text,
updatedon timestamp)
I would need to to perform three queries on this table
Query all data (findByAll)
Query all data based on major, planning on adding a secondary index on this column
Query data based on time range I.e updated on column
I can achieve this using a composite primary key, however I also want rows to be uniquely identifiable based on id only. For example:
Row 1 :
1| engineering | electrical | 01-01-2021
If the student were to a different major?
1| engineeering | mechanical | 02-02-2021
I would like to perform an upsert where only the major and updated on columns would change.
My conundrum is I don’t understand what I should have as my primary key if I want to perform range queries on updatedon, where rows a uniquely identified by id only.
I came across a bucketing approach but wasn’t sure if that would add additional complexity to my simple/minimal design.
It looks like you're approaching it backwards by starting with the table design. When modelling your data in Cassandra, it sounds counter-intuitive but you need to start with the application queries first and design tables against those queries.
Let me illustrate by listing all your app queries and designing a table for each of them.
APP QUERY 1 - Query all data (findByAll)
If your intention is to retrieve all the records to display them, this is a bad idea in Cassandra since it will require a full table scan. I'm aware that developers are used to doing this on toy applications with a small amount of data but in Cassandra, data is distributed across nodes so full table scans don't scale.
Think of situations where you have a million or more records with hundreds of nodes in the cluster. It doesn't make sense for an app to wait for the query to finish retrieving all records.
APP QUERY 2 - Query all data based on major, planning on adding a secondary index on this column
Adding an index on major isn't a good idea if performance matters to you. You should design a table specifically optimised for this query. For example:
CREATE TABLE students_by_major (
major text,
studentid text,
department text,
updatedon timestamp,
PRIMARY KEY (major, studentid)
)
In this table, each major partition has 1 or more rows of studentid. For example:
major | studentid | department | updatedon
------------------------+-----------+-------------+---------------------------------
computer science | 321 | science | 2020-01-23 00:00:00.000000+0000
electrical engineering | 321 | engineering | 2020-02-24 00:00:00.000000+0000
electrical engineering | 654 | engineering | 2019-05-06 00:00:00.000000+0000
chemical engineering | 654 | engineering | 2019-07-08 00:00:00.000000+0000
arts | 987 | law | 2020-09-12 00:00:00.000000+0000
civil engineering | 654 | engineering | 2019-02-04 00:00:00.000000+0000
APP QUERY 3 - Query data based on time range I.e updated on column
You'll only be able to do a range query on updatedon if the column is defined in the primary key.
APP QUERY 4 - If the student were to do a different major?
You can have a table where each student has multiple rows of majors:
CREATE TABLE majors_by_student (
studentid text,
major text,
department text,
updatedon timestamp,
PRIMARY KEY (studentid, major)
)
For example, student ID 654 has updated their major 3 times:
cqlsh> SELECT * FROM majors_by_student WHERE studentid = '654';
studentid | updatedon | department | major
-----------+---------------------------------+-------------+------------------------
654 | 2019-07-08 00:00:00.000000+0000 | engineering | chemical engineering
654 | 2019-05-06 00:00:00.000000+0000 | engineering | electrical engineering
654 | 2019-02-04 00:00:00.000000+0000 | engineering | civil engineering
QUERY 5 - You want to perform range queries on updatedon where rows are uniquely identified by studentid only.
CREATE TABLE community.updated_majors_by_student (
studentid text,
updatedon timestamp,
department text,
major text,
PRIMARY KEY (studentid, updatedon)
)
Using student 654 above as an example, you can do a range query for any updates made after April 30 with:
SELECT * FROM updated_majors_by_student WHERE studentid = '654' AND updatedon > '2019-04-30 +0000';
Note that since updatedon is a timestamp, you need to specify the timezone for precision and +0000 is the TZ for UTC.
studentid | updatedon | department | major
-----------+---------------------------------+-------------+------------------------
654 | 2019-07-08 00:00:00.000000+0000 | engineering | chemical engineering
654 | 2019-05-06 00:00:00.000000+0000 | engineering | electrical engineering
To keep the tables above in sync, you need to use CQL BATCH statements as I've described in this article -- https://community.datastax.com/articles/2744/. Cheers!
I have a table with a structure like this:
CREATE TABLE kaefko.se_vi_f55dfeebae00d2b3 (
value text PRIMARY KEY,
id text,
popularity bigint);
With data that looks like this:
value | id | popularity
--------+------------------+------------
rally | 4eff16cb91f96cd6 | 2
reddit | 11aa39686ed66ba5 | 3
red | 552d7e95af481415 | 1
really | 756bfa499965863c | 1
right | c5850c6b08f7966b | 1
redis | 7f1d251f399442d7 | 1
And I've created a materialized view that should sort these values by the popularity from the biggest to the smallest ones:
CREATE MATERIALIZED VIEW kaefko.se_vi_f55dfeebae00d2b3_by_popularity AS
SELECT *
FROM kaefko.se_vi_f55dfeebae00d2b3
WHERE popularity IS NOT null
PRIMARY KEY (value, popularity)
WITH CLUSTERING ORDER BY (popularity DESC);
But the data in the materialized view looks like this:
value | popularity | id
--------+------------+------------------
rally | 2 | 4eff16cb91f96cd6
reddit | 3 | 11aa39686ed66ba5
really | 1 | 756bfa499965863c
right | 1 | c5850c6b08f7966b
redis | 1 | 7f1d251f399442d7
As you can see there are two main issues:
Data is not sorted as defined in the materialized view
There is just a part of all data in the materialized view
I'm not very experienced in Cassandra and I've already spent hours trying to find the reason why this happens with no avail. Could somebody please help me? Thank you <3
__
I'm using ScyllaDB 4.1.9-0 and cqlsh shows this:
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Alex's comment is 100% correct, the order is within the partition.
PRIMARY KEY (value, popularity)
WITH CLUSTERING ORDER BY (popularity DESC);
This means that the ordering of popularity is descending only for values where the 'value' field is the same - if I was to alter the data you used to show what this would look like as an example, you would get the following:
value | popularity | id
--------+------------+------------------
rally | 3 | 4eff16cb91f96cd6
rally | 2 | 11aa39686ed66ba5
really | 3 | 756bfa499965863c
really | 2 | c5850c6b08f7966b
really | 1 | 7f1d251f399442d7
The order is on a per partition key basis, not globally ordered.
I create a table in Cassandra for monitoring insert from an application.
My partition key is an int composed by year+month+day, my clustering key a timestamp and after that my username and some others fields.
I would like to display the last 5 inserts but it's seems that the partition key go before the "order by desc".
How can I get the correct result ? Normaly clustering key induces the order so why I get this result? (Thank in advance)
Informations :
Query : select tsp_insert, txt_name from ks_myKeyspace.myTable limit 5;
Result :
idt_day | tsp_insert | txt_name
----------+--------------------------+----------
20161028 | 2016-10-28 15:21:09+0000 | Jean
20161028 | 2016-10-28 15:21:01+0000 | Michel
20161028 | 2016-10-28 15:20:44+0000 | Quentin
20161031 | 2016-10-31 09:24:32+0000 | Jacquie
20161031 | 2016-10-31 09:23:32+0000 | Gabriel
Wanted :
idt_day | tsp_insert | txt_name
----------+--------------------------+----------
20161031 | 2016-10-31 09:24:32+0000 | Jacquie
20161031 | 2016-10-31 09:23:32+0000 | Gabriel
20161028 | 2016-10-28 15:21:09+0000 | Jean
20161028 | 2016-10-28 15:21:01+0000 | Michel
20161028 | 2016-10-28 15:20:44+0000 | Quentin
My table :
CREATE TABLE ks_myKeyspace.myTable(
idt_day int,
tsp_insert timestamp,
txt_name text, ...
PRIMARY KEY (idt_day, tsp_insert)) WITH CLUSTERING ORDER BY (tsp_insert DESC);
Ultimately, you are seeing the current order because you are not using a WHERE clause. You can see what's going on if you use the token function on your partition key:
aploetz#cqlsh:stackoverflow> SELECT idt_day,tsp_insert,token(idt_day),txt_name FROM mytable ;
idt_day | tsp_insert | system.token(idt_day) | txt_name
----------+---------------------------------+-----------------------+----------
20161028 | 2016-10-28 15:21:09.000000+0000 | 810871225231161248 | Jean
20161028 | 2016-10-28 15:21:01.000000+0000 | 810871225231161248 | Michel
20161028 | 2016-10-28 15:20:44.000000+0000 | 810871225231161248 | Quentin
20161031 | 2016-10-31 09:24:32.000000+0000 | 5928478420752051351 | Jacquie
20161031 | 2016-10-31 09:23:32.000000+0000 | 5928478420752051351 | Gabriel
(5 rows)
Results in Cassandra CQL will always come back in order of the hashed token value of the partition key (which you can see by using token). Within the partition keys, your CLUSTERING ORDER will be enforced.
That's key to understand... Result set ordering in Cassandra can only be enforced within a partition key. You have no control over the order that the partition keys come back in.
In short, use a WHERE clause on your idt_day and you'll see the order you expect.
It seems to me that you are getting the whole thing wrong. Partition keys are not used for ordering data, they are used only to know the location of your data in the cluster, specifically the node. Moreover, the order really matters inside a partition only...
Your query results really are unpredictable. Depending on which node is faster to answer (assuming a cluster and not a single node), you can get every time a different result. You should try to avoid selecting without partition restrictions, they don't scale.
You can however change your queries and perform one select per day, then you'd query for ordered data (your clustering key) in an ordered manner ( you manually chose the order of the days in your queries). And as a side note it would be faster because you could query multiple partitions in parallel.
I am new to cassandra. I am using cassandra-3.0 and datastax java driver for development. I would like to know whether cassandra provide any option to fecth the data based on rowkey range?
something like
select * from <table-name> where rowkey > ? and rowkey < ?;
If not, is there any other option in cassandra ( java/cql) to fetchdata based on row ranges?
Unfortunately, there really isn't a mechanism in Cassandra that works in the way that you are asking. The only way to run a range query on your partition keys (rowkey) is with the token function. This is because Cassandra orders its rows in the cluster by the hashed token value of the partition key. That value would not really have any meaning for you, but it would allow you to "page" through the a large table without encountering timeouts.
SELECT * FROM <table-name>
WHERE token(rowkey) > -9223372036854775807
AND token(rowkey) < -5534023222112865485;
The way to go about range querying on meaningful values, is to find a value to partition your rows by, and then cluster by a numeric or time value. For example, I can query a table of events by date range, if I partition my data by month (PRIMARY KEY(month,eventdate)):
aploetz#cqlsh:stackoverflow> SELECT * FROM events
WHERE monthbucket='201509'
AND eventdate > '2015-09-19' AND eventdate < '2015-09-26';
monthbucket | eventdate | beginend | eventid | eventname
-------------+--------------------------+----------+--------------------------------------+------------------------
201509 | 2015-09-25 06:00:00+0000 | B | a223ad16-2afd-4213-bee3-08a2c4dd63e6 | Hobbit Day
201509 | 2015-09-25 05:59:59+0000 | E | 9cd6a265-6c60-4537-9ea9-b57e7c152db9 | Cassandra Summit
201509 | 2015-09-22 06:00:00+0000 | B | 9cd6a265-6c60-4537-9ea9-b57e7c152db9 | Cassandra Summit
201509 | 2015-09-20 05:59:59+0000 | E | b9fe9668-cef2-464e-beb4-d4f985ef9c47 | Talk Like a Pirate Day
201509 | 2015-09-19 06:00:00+0000 | B | b9fe9668-cef2-464e-beb4-d4f985ef9c47 | Talk Like a Pirate Day
(5 rows)
I'm using Cassandra 2.1.5.
I'm creating table using:
create table dummy2(
id timeuuid,
time timestamp,
primary key (id, time)
) with clustering order by (time desc);
I inserted into the table four records:
insert into dummy2 (id, time) values (now(), 1000000);
insert into dummy2 (id, time) values (now(), 2000000);
insert into dummy2 (id, time) values (now(), 3000000);
insert into dummy2 (id, time) values (now(), 4000000);
I get results:
id | time
--------------------------------------+--------------------------
e1fa7a80-1e64-11e5-8bf5-55cdf06f740f | 1970-01-01 08:33:20+0800
e3bbb280-1e64-11e5-8bf5-55cdf06f740f | 1970-01-01 08:50:00+0800
e5ceb400-1e64-11e5-8bf5-55cdf06f740f | 1970-01-01 09:06:40+0800
e0719090-1e64-11e5-8bf5-55cdf06f740f | 1970-01-01 08:16:40+0800
which looks like a tree map order, or random...
If I change the id type from "timeuuid" to "text", then the ordering works just fine:
id | time
-------+--------------------------
hello | 1970-01-01 09:06:40+0800
hello | 1970-01-01 08:50:00+0800
hello | 1970-01-01 08:33:20+0800
hello | 1970-01-01 08:16:40+0800
Is it by design or a bug? Or I'm using it in a wrong way?
Yes, this is the way Cassandra is designed to work. Clustering order only works within a partition. That's because each partition key is hashed into a token to determine where it should be stored in the cluster (to provide optimal data distribution). The rows within each partition are then written on-disk in their clustering order.
So in your first example, each row is sorted by time within each id. Of course, as each partition key (id) is different, you have no way of seeing that. But in your second example, your partition keys are the same, so your results are clustered by time.
"which looks like a tree map order, or random..."
They are ordered by their hashed token values, you can see this by using the token function:
aploetz#cqlsh:stackoverflow2> SELECT token(id),id,time FROM dummy3;
token(id) | id | time
----------------------+-------+--------------------------
-3758069500696749310 | hello | 1969-12-31 19:06:40-0600
-3758069500696749310 | hello | 1969-12-31 18:50:00-0600
-3758069500696749310 | hello | 1969-12-31 18:33:20-0600
-3758069500696749310 | hello | 1969-12-31 18:16:40-0600
(4 rows)
Or perhaps a better example:
aploetz#cqlsh:stackoverflow2> SELECT token(id),id,time FROM dummy2;
token(id) | id | time
----------------------+--------------------------------------+--------------------------
-5795426230130619993 | e1fa7a80-1e64-11e5-8bf5-55cdf06f740f | 1969-12-31 18:33:20-0600
-2088884548269216731 | e3bbb280-1e64-11e5-8bf5-55cdf06f740f | 1969-12-31 18:50:00-0600
8496311684589314797 | e5ceb400-1e64-11e5-8bf5-55cdf06f740f | 1969-12-31 19:06:40-0600
8930307282139899213 | e0719090-1e64-11e5-8bf5-55cdf06f740f | 1969-12-31 18:16:40-0600
(4 rows)
Earlier this year I wrote an article for PlanetCassandra on this frequently mis-understood topic: We Shall Have Order! Give it a read and see if that helps point you in the right direction.