If I have a List field in Cassandra and two people write to it at the same time, is it a simple last write wins or will it merge the writes?
For example: [a, b, c, d]
User1 -> [b, a, c, d] (move b to index 0)
User2 -> [a, b, d, c] (move c to index 3)
Will Cassandra merge the results and end up with [b, a, d, c] or will it use last write wins to the microsecond?
You will get the merge result
Every write data to cassandra, a timestamp associated with each column is also inserted. when you execute read query, timestamps are used to pick a "winning" update within a single column or collection element.
What if I have a truly concurrent write with the same time stamp? In the unlikely case that you precisely end up with two time stamps that match in its microsecond, you might end up with a bad version but Cassandra ensures that ties are consistently broken by comparing the byte values.
Cassandra store list (collection) different than normal column.
Example :
CREATE TABLE friendlists (
user text PRIMARY KEY,
friends list <text>
);
If we insert some dummy data :
user | friends
----------+-------------------------
john | [doug, patricia, scott]
patricia | [john, lucifer]
The internal representation:
RowKey: john
=> (column=, value=, timestamp=1374687324950000)
=> (column=friends:26017c10f48711e2801fdf9895e5d0f8, value='doug', timestamp=1374687206993000)
=> (column=friends:26017c11f48711e2801fdf9895e5d0f8, value='patricia', timestamp=1374687206993000)
=> (column=friends:26017c12f48711e2801fdf9895e5d0f8, value='scott', timestamp=1374687206993000)
=> (column=friends:6c504b60f48711e2801fdf9895e5d0f8, value='matt', timestamp=1374687324950000)
=> (column=friends:6c504b61f48711e2801fdf9895e5d0f8, value='eric', timestamp=1374687324950000)
-------------------
RowKey: patricia
=> (column=, value=, timestamp=1374687352290000)
=> (column=friends:3b817b80f48711e2801fdf9895e5d0f8, value='john', timestamp=1374687243064000)
Here the internal column name is more complicated because a UUID is appended to the name of the CQL field "friends". This is used to keep track of the order of items in the list.
Every time you insert data cassandra with below query :
INSERT INTO friendlists (user , friends ) VALUES ( 'patricia', ['john', 'lucifer']);
//or
UPDATE friendlists SET friends = ['john', 'lucifer'] where user = 'patricia';
Will create a tombstone with a less timestamp than current, it tells that the previous data has been deleted. So if concurrent insert happened with the same exact timestamp both data are ahead of tombstone so both data will live.
Source :
http://mighty-titan.blogspot.com/2012/06/understanding-cassandras-consistency.html
http://opensourceconnections.com/blog/2013/07/24/understanding-how-cql3-maps-to-cassandras-internal-data-structure-sets-lists-and-maps/
Related
id
itemId
correlationId
1
A
2
B
A
3
A
4
C
B
5
D
B
6
E
D
Hello, I have a Notes database with a similar structure like the table above. This table contains a unique id for each document. It also contains an itemId, which is not unique, and a correlationId which creates a relation to another item (itemId).
I want to select all documents, which have a specific itemId (in this example A) and every document which is correlated to this itemId (directly or indirectly). The result should look similar to my table.
I was wondering if I need to write multiple requests or if it is possible to write only one formula.
SELECT #Contains(itemId; "A");
SELECT #Contains(correlationId;"A") => retrieve itemId: B ==>SELECT #Contains(correlationId;"B") => retrieve itemId: C, D ==> SELECT #Contains(correlationId;"C") (no result) and SELECT #Contains(correlationId;"D")
Considering three rows, (a, b), (a, f) and (c, e).
Sadly, under some flake partitioning algorithm, the partition key value a and c hashes to the same value.
In this case, how does Cassandra internally order the rows?
Will it be in:
(a,b)
(a,f)
(c,e)
or in
(a, b)
(c, e)
(a, f)
First of all, if the partition keys are different, there will be no chance of having same token (Hash Value). Murmur3Partitioner will take care of uniqueness of token (hash).
row 1 (a,b) - a is partition key in this case. Hash will be calculated using a
row 2 (a,f) - a is partition key in this case. Hash will be calculated using a
row 3 (c,e) - c is partition key in this case. Hash will be calculated using c
Row number 1 and 2 are having same row kay (partition key), now this is collision. But cassandra has time stamp associated with every cell (a cell is smallest unit of storage in cassandra).Every cell has three properties Name, Value and Time Stamp. Cassandra use time stamp to merge these tow rows in to one row. The latest value will be persisted. From above example, for row kay a, value f will be persisted. Cassandra compaction process will take care of merging the rows.
If I create a table like this in Cassandra
CREATE TABLE example (
key1 text PRIMARY KEY,
map1 map<text,text>,
list1 list<text>,
set1 set<text>
);
and insert some data like this
INSERT INTO example (
key1,
map1,
list1,
set1
) VALUES (
'john',
{'patricia':'555-4326','doug':'555-1579'},
['doug','scott'],
{'patricia','scott'}
);
and look at the storage using CLI, I will see this
RowKey: john
=> (column=, value=, timestamp=1374683971220000)
=> (column=map1:doug, value='555-1579', timestamp=1374683971220000)
=> (column=map1:patricia, value='555-4326', timestamp=1374683971220000)
=> (column=list1:26017c10f48711e2801fdf9895e5d0f8, value='doug', timestamp=1374683971220000)
=> (column=list1:26017c12f48711e2801fdf9895e5d0f8, value='scott', timestamp=1374683971220000)
=> (column=set1:'patricia', value=, timestamp=1374683971220000)
=> (column=set1:'scott', value=, timestamp=1374683971220000)
Now my question is this: what is the first row in CLI output? what does it mean? why it does not have any Column nor Value but has a timestamp?
The "row marker" was introduced [1] so the row doesn't disappear when you remove (set a column to null) the last column. Aligned with how traditional SQL implementations behaves)
You have also found out how cassandra represents collections under the hood.
Remember that
Map keys should be unique (solved)
List can contain duplicates (solved by appending a uuid)
Set should not contain duplicates (solved)
[1] https://issues.apache.org/jira/browse/CASSANDRA-4361
This because while using cassandra-cli, you get a thrift representation of the rows.
First information are for primary key, as your is just a partition key, it's the same as the row key.
So you have your value : as rowKey (John in your example) then the timestamp
You will have more readable result set if you usin cqlsh instead.
you can find more detail here :
https://thelastpickle.com/blog/2013/01/11/primary-keys-in-cql.html
I hope this helps
I have a table (let's call it T) set up with a PRIMARY KEY like the following:
PRIMARY KEY ((A, B), C, ....);
I want to query it like the following:
SELECT * FROM T WHERE A = ? and C <= ? PER PARTITION LIMIT 1 ALLOW FILTEIRNG;
(Note that C is a timstamp value. I am essentially asking for the most recent rows across all partitions whose first partition key belongs to my input).
This works with the allow filtering command, and it makes sense why I need it; I do not know beforehand the partition keys B, and I do not care - I want all of them. Therefore, it makes sense that Cassandra would need to scan the entire partition to give me the results, and it also makes sense why I would need to specify it to allow filtering for this to occur.
However, I have read that we should avoid 'ALLOW FILTERING' at all costs, as it can have a huge performance impact, especially in production environments. Indeed, I only use allow filtering very sparingly in my existing applications, and this is usually for one-off queries that calculate something of this nature.
My quesiton is this: is there a way to restructure this table or query to avoid filtering? I am thinking it is impossible, as I do not have knowledge of the keys that make up B beforehand, but I want to double check just to be sure. Thanks!
You cannot efficiently make that query if (A, B) is your partition key. your key would need to be ((A), B) (dropping clustering keys). Then SELECT * FROM T WHERE A = ?. If only care about the latest, then A, B would always be replaced with the most recent.
Another option if looking to get the A,B tuples from a time is to create a table thats indexed by time and have the tuples be clustering columns from there like ((time_bucket), A, B, C). time_bucket being a string like 2018-04-06:00:00:00 that contains all the events for that day. Then when you query like:
> CREATE TABLE example (time_bucket text, A int, B int, C int, D int, PRIMARY KEY ((time_bucket), A, B, C)) WITH CLUSTERING ORDER BY (A ASC, B ASC, C DESC);
> INSERT INTO example (time_bucket, A, B, C, D) VALUES ('2018-04', 1, 1, 100, 999);
> INSERT INTO example (time_bucket, A, B, C, D) VALUES ('2018-04', 1, 1, 120, 999);
> INSERT INTO example (time_bucket, A, B, C, D) VALUES ('2018-04', 1, 1, 130, 999);
> INSERT INTO example (time_bucket, A, B, C, D) VALUES ('2018-04', 1, 2, 130, 999);
> SELECT * FROM example WHERE time_bucket = '2018-04' GROUP BY time_bucket, A, B;
time_bucket | a | b | c | d
-------------+---+---+-----+-----
2018-04 | 1 | 1 | 130 | 999
2018-04 | 1 | 2 | 130 | 999
You would get the 1st result from each of the rows in the time bucket partition whose clustering by A and B. If you make the partitions small enough (use finer grain time buckets, like hours or 15 minutes or something, depending on data rate) its more acceptable to use ALLOW FILTERING here then like:
SELECT * FROM example WHERE time_bucket = '2018-04' AND A = 1 AND C < 120 GROUP BY time_bucket, A, B ALLOW FILTERING ;
time_bucket | a | b | c | d
-------------+---+---+-----+-----
2018-04 | 1 | 1 | 100 | 999
Because its all within one partition and within a bounded size (monitor it closely with tablestats/max partition size). Make sure always querying with time_bucket though so it doesnt become a range query. You want to make sure you do not end up going through too many things without returning a result (which is one of dangers of allow filtering).
I am trying to practice Cassandra using this example (under Composite Columns paragraph):
So, I have created table tweets and it looks like following:
cqlsh:twitter> SELECT * from tweets;
tweet_id | author | body
--------------------------------------+-------------+--------------
73954b90-baf7-11e4-a7d0-27983e9e7f51 | gwashington | I chopped...
(1 rows)
Now I am trying to populate timeline, which is a related table using CQL and I am not sure how to do it. I have tried SQL approach, but it did not work:
cqlsh:twitter> INSERT INTO timeline (user_id, tweet_id, author, body) SELECT 'gmason', 73954b90-baf7-11e4-a7d0-27983e9e7f51, author, body FROM tweets WHERE tweet_id = 73954b90-baf7-11e4-a7d0-27983e9e7f51;
Bad Request: line 1:55 mismatched input 'select' expecting K_VALUES
So I have two questions:
How to populate timeline table with SQL, so that it would relate to tweets?
How do I make sure that Timeline Physical Layout will be created as shown in that example?
Thanks.
EDIT:
This is explanation for my question #2 above (the picture is taken from here):
tldr;
Use cqlsh COPY to export tweets, modify the file, use COPY to import timeline.
Use cassandra-cli to verify the physical structure.
Long version...
I'll go a different way on this one, and suggest that it will probably be easier using the native COPY command in cqlsh.
I followed the similar examples found here. After creating the tweets and timeline tables in cqlsh, I inserted rows into tweets as indicated. My tweets table then looked like this:
aploetz#cqlsh:stackoverflow> SELECT * FROM tweets;
tweet_id | author | body
--------------------------------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------
05a5f177-f070-486d-b64d-4e2bb28eaecc | gmason | Those gentlemen, who will be elected senators, will fix themselves in the federal town, and become citizens of that town more than of your state.
b67fe644-4dbe-489b-bc71-90f809f88636 | jmadison | All men having power ought to be distrusted to a certain degree.
819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1 | gwashington | To be prepared for war is one of the most effectual means of preserving peace.
I then exported them like this:
aploetz#cqlsh:stackoverflow> COPY tweets TO '/home/aploetz/tweets_20150223.txt'
WITH DELIMITER='|' AND HEADER=true;
3 rows exported in 0.052 seconds.
I then edited the tweets_20150223.txt file, adding a user_id column on the front and copying a couple of rows, like this:
userid|tweet_id|author|body
gmason|05a5f177-f070-486d-b64d-4e2bb28eaecc|gmason|Those gentlemen, who will be elected senators, will fix themselves in the federal town, and become citizens of that town more than of your state.
jmadison|b67fe644-4dbe-489b-bc71-90f809f88636|jmadison|All men having power ought to be distrusted to a certain degree.
gwashington|819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1|gwashington|To be prepared for war is one of the most effectual means of preserving peace.
jmadison|819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1|gwashington|To be prepared for war is one of the most effectual means of preserving peace.
ahamilton|819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1|gwashington|To be prepared for war is one of the most effectual means of preserving peace.
ahamilton|05a5f177-f070-486d-b64d-4e2bb28eaecc|gmason|Those gentlemen, who will be elected senators, will fix themselves in the federal town, and become citizens of that town more than of your state.
I saved that file as timeline_20150223.txt, and imported it into the timeline table, like this:
aploetz#cqlsh:stackoverflow> COPY timeline FROM '/home/aploetz/timeline_20150223.txt'
WITH DELIMITER='|' AND HEADER=true;
6 rows imported in 0.016 seconds.
Yes, timeline will be a wide-row table, partitioning on user_id and then clustering on tweet_id. I verified the "under the hood" structure by running the cassandra-cli tool, and listing the timeline column family (table). Here you can see how the rows are partitioned by user_id, and each column has the tweet_id uuid as a part of its name:
-
[default#stackoverflow] list timeline;
Using default limit of 100
Using default cell limit of 100
-------------------
RowKey: ahamilton
=> (name=05a5f177-f070-486d-b64d-4e2bb28eaecc:, value=, timestamp=1424707827585904)
=> (name=05a5f177-f070-486d-b64d-4e2bb28eaecc:author, value=676d61736f6e, timestamp=1424707827585904)
=> (name=05a5f177-f070-486d-b64d-4e2bb28eaecc:body, value=54686f73652067656e746c656d656e2c2077686f2077696c6c20626520656c65637465642073656e61746f72732c2077696c6c20666978207468656d73656c76657320696e20746865206665646572616c20746f776e2c20616e64206265636f6d6520636974697a656e73206f66207468617420746f776e206d6f7265207468616e206f6620796f75722073746174652e, timestamp=1424707827585904)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:, value=, timestamp=1424707827585715)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:author, value=6777617368696e67746f6e, timestamp=1424707827585715)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:body, value=546f20626520707265706172656420666f7220776172206973206f6e65206f6620746865206d6f73742065666665637475616c206d65616e73206f662070726573657276696e672070656163652e, timestamp=1424707827585715)
-------------------
RowKey: gmason
=> (name=05a5f177-f070-486d-b64d-4e2bb28eaecc:, value=, timestamp=1424707827585150)
=> (name=05a5f177-f070-486d-b64d-4e2bb28eaecc:author, value=676d61736f6e, timestamp=1424707827585150)
=> (name=05a5f177-f070-486d-b64d-4e2bb28eaecc:body, value=54686f73652067656e746c656d656e2c2077686f2077696c6c20626520656c65637465642073656e61746f72732c2077696c6c20666978207468656d73656c76657320696e20746865206665646572616c20746f776e2c20616e64206265636f6d6520636974697a656e73206f66207468617420746f776e206d6f7265207468616e206f6620796f75722073746174652e, timestamp=1424707827585150)
-------------------
RowKey: gwashington
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:, value=, timestamp=1424707827585475)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:author, value=6777617368696e67746f6e, timestamp=1424707827585475)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:body, value=546f20626520707265706172656420666f7220776172206973206f6e65206f6620746865206d6f73742065666665637475616c206d65616e73206f662070726573657276696e672070656163652e, timestamp=1424707827585475)
-------------------
RowKey: jmadison
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:, value=, timestamp=1424707827585597)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:author, value=6777617368696e67746f6e, timestamp=1424707827585597)
=> (name=819d95e9-356c-4bd5-9ad0-8cd36a7aa5e1:body, value=546f20626520707265706172656420666f7220776172206973206f6e65206f6620746865206d6f73742065666665637475616c206d65616e73206f662070726573657276696e672070656163652e, timestamp=1424707827585597)
=> (name=b67fe644-4dbe-489b-bc71-90f809f88636:, value=, timestamp=1424707827585348)
=> (name=b67fe644-4dbe-489b-bc71-90f809f88636:author, value=6a6d616469736f6e, timestamp=1424707827585348)
=> (name=b67fe644-4dbe-489b-bc71-90f809f88636:body, value=416c6c206d656e20686176696e6720706f776572206f7567687420746f206265206469737472757374656420746f2061206365727461696e206465677265652e, timestamp=1424707827585348)
4 Rows Returned.
Elapsed time: 35 msec(s).
In order to accomplish this you will need to use an ETL tool. Use either Hadoop or Spark. There is no INSERT/SELECT in CQL and this is for a reason. In a real world you will need to execute 2 inserts from your application - one into each table.
You will just have to believe that when you have primary key with partition key and clustering key, this would store the data in a wide row format.