I had two following tables ( taken from Cassandra Definitve Guide , https://gist.github.com/jeffreyscarpenter/761ddcd1c125dfb194dc02d753d31733 } - What is guaranteed with respect to the folloowing tables assuming they had the same partition key ?
Can we safely assume the data for both the tables present in the same node as long as the partition key is same ? as both tables contain same partition key.
Ok , and as tables are different from each other , will they be stored in different partitions or same partition in the "same" node
https://gist.github.com/jeffreyscarpenter/761ddcd1c125dfb194dc02d753d31733
CREATE TABLE hotel.pois_by_hotel (
poi_name text,
hotel_id text,
description text,
PRIMARY KEY ((hotel_id), poi_name)
) WITH comment = 'Q3. Find pois near a hotel';
CREATE TABLE hotel.available_rooms_by_hotel_date (
hotel_id text,
date date,
room_number smallint,
is_available boolean,
PRIMARY KEY ((hotel_id), date, room_number)
) WITH comment = 'Q4. Find available rooms by hotel / date';
if both tables have the same partition key, then the same value will be mapped into the same token. If tables are in the same keyspace, then yes - they will be on the same node(s). If they are in the different keyspaces, then there could be a partial overlap - if replication factor is different, for example, one keyspace has higher RF (like, KS1 has RF=2, and KS2 has RF=3, then 2 nodes will have replicas for both keyspaces, and 3rd node will have only for KS2).
Each table will have its own set of the files on disk, so although they have the same "logical partitions", on disk they are in different files. You can always look into data files, something like, /var/lib/cassandra/data/<keyspace>/<table>-<table-uuid>/
Related
I have the following scenario;
My data has the id field, and this field constantly increasing.
When event created, id is assigned = 1 automatically.
Then 2, 3, 4 and so on.
When data that has the id = 1 is generated, then it will never be generated again.
I want to store this dat ain Cassandra. I can set primary key as the id field, but i dont know how cassandra will create partitions for each record?
Will it create one partition for each record?
Or will it create range partition by primary key. For example; id from 1 to 100 is the first partition, 100-200 is the second partition etc.
In Cassandra, the partition key uniquely identifies a single partition (record) in the table. For clarification, the primary key:
must have 1 partition key
zero or more clustering columns
So the primary key doesn't equate to a range of partitions.
Compared to traditional RDBMS which have two-dimensional tables, Cassandra tables have the traditional 2D tables but can also be 3D or more. The power of Cassandra is that tables can be multi-dimensional meaning each partition can have one or more rows (it can have thousands).
If you're interested, I've explained this in a bit more detail with examples in this post -- https://community.datastax.com/questions/6171/. Cheers!
Cassandra Data Modeling Query
Hello,
The data model i am working on is as below with different tables for same data data set for satisfying different kinds of query. The data mainly stores event data of some campaigns sent out on multiple channels like email, web, mobile app, sms etc. Events can include page visits, email opens, link clicks etc for different subscribers.
Table 1:
(enterprise_id int, domain_id text, campaign_id int, event_category text, event_action text, datetime timestamp, subscription_id text, event_label text, ........) (many more columns not part of primary key)
PRIMARY KEY ((enterprise_id,campaign_id),domain_id, event_category, event_action, datetime, subscription_id))
CLUSTERING ORDER BY (domain_id DESC, event_category DESC, event_action DESC, datetime DESC, subscription_id DESC)
Keys and Data size for Table 1:
I have partition key as enterprise_id + campaign_id . Each enterprise can have several campaigns . The datastore may have data for few hundred campaigns. Each campaign can have upto 2-3 million records. Hence there may be 3000 partitions across 100 enterprises and each partition having 2-3 miilion records.
Cassandra Queries: Query always with partition key + primary key including the datetime field. The subscription id is included in primary key to keep each record unique as we can have multiple records with similar values for rest of the keys in primary key. enterprise_id +c ampaign_id is always available as a filter in the queries.
Table 2:
(enterprise_id int, domain_id text, event_category text, event_action text, datetime timestamp, subscription_id text, event_label text, campaign_id int........) (many more columns not part of primary key)
PRIMARY KEY (enterprise_id, domain_id, event_category, event_action, datetime, subscription_id))
CLUSTERING ORDER BY (domain_id DESC, event_category DESC, event_action DESC, datetime DESC, subscription_id DESC)
Keys and Data size for Table 2) : I have partition key as enterprise_id only. Each enterprise can have several campaigns . May be few hundred campaigns. Each campaign can have upto 2-3 Mn records. In this case the partition is quite big with data for all campaigns in a single partition. can have upto 800 - 900 million entries
Cassandra Queries: Query always with partition key + primary key upto datetime. The subscription id is included in primary key to keep each record unique as we can have multiple records with similar values for rest of the keys in primary key. In this case, data has to be queries across campaigns and we may not have campaign_id as a filter in the queries.
Table 3:
(enterprise_id int, subscription_id text, domain_id text, event_category text, event_action text, datetime timestamp, event_label text, campaign_id int........) (many more columns not part of primary key)
PRIMARY KEY (enterprise_id, subscription_id, domain_id, event_category, event_action, datetime, ))
CLUSTERING ORDER BY ( subscription_id DESC, domain_id DESC, event_category DESC, event_action DESC, datetime DESC,)
Keys and Data size for Table 3) : I have partition key as enterprise_id. Each enterprise can have several campaigns . May be few hundred campaigns. Each campaign can have upto 2-3 Mn records. In this case the partition is quite big with data for all campaigns in a single partition. can have upto 800 -900 million entries
Cassandra Queries: Query always with partition key + primary key as subscription_id only. Should be able to query directly on enterprise_id + subscription_id.
My Queries:
Size of data on each partition: With Table 2) and Table 3) i may end up with more than 800 -900 million rows per partition. As per my reading it is not ok to have so many entries per partition. How can i achieve my use case in this scenario? Even if i create multiple partitions based on some data like a week_number (1-52 in a year), the query will need to query across all partitions and end up using a IN clause with all week numbers which is as good as scanning all data.
Is it ok to have multiple tables with same partition key and different primary keys with Clustering order change? For example in Table 2 and Table 3 the hash will be on enterprise_id and will lead to same node. However only the clustering key order has changed and will allow me to query directly on the required key. Will the data be in different physical partitions for Table2 and Table3 in such a scenario? Or if it maps to same partition number how will cassandra internally distinguish between the two tables?
Is it ok to use ALLOW FILTERING if i specify the partition key. For example i can avoid the need for creating Table 3 and use table 2 for query on subscription_id directly if i use ALLOW FILTERING on Table 2. What will be the impact again.
First of all, please only as one question per question. Given the length and detail required for your answers, this post is unlikely to provide long term value for future users.
As per my reading it is not ok to have so many entries per partition. How can I achieve my use case in this scenario?
Unfortunately, if partitioning on a time component will not work, then you'll have to find some other column to partition the data by. I've seen rows-per-partition work ok in the range of 50k to 20k. Most of those use cases on the higher end had small partitions. It looks like your model has many columns, so I'd be curious as to the average partition size. Essentially, find a column to partition on which keeps your partition sizes in the 10MB to 1MB range.
Is it ok to have multiple tables with same partition key and different primary keys with Clustering order change?
Yes, this is perfectly fine.
Will the data be in different physical partitions for Table2 and Table3 in such a scenario? Or if it maps to same partition number how will cassandra internally distinguish between the two tables?
The partition is hashed into a number ranging from +/- 2^63. That number will then be compared to the partition ranges mapped to all nodes, and then the query will be sent to that node. So all the partition does, is determine which node is responsible for the data.
The tables have their data files written to different directories, based on table name. So Cassandra distinguishes between the tables by the table name provided in the query. Nothing you need to worry about.
Is it ok to use ALLOW FILTERING if I specify the partition key.
I would still recommend against it if you're concerned about performance. But the good thing about using the ALLOW FILTERING directive while specifying a full partition key, will indeed prevent Cassandra from reading multiple nodes to build the result set. So that should be ok. The only drawback here, is that Cassandra stores/reads data from disk by the defined CLUSTERING ORDER, and using ALLOW FILTERING obviously complicates that process (forcing random reads vs. sequential reads).
I am using the Secondary index for one of the column in Cassandra table.,
Say I have a 5 node cluster (192.168.1.1, 192.168.1.2, 192.168.1.3, 192.168.1.4, 192.168.1.5) with the Keyspace replication factor as '3' and considering the following table,
CREATE TABLE nodestat (
uniqueId text,
totalCapacity int,
physicalUsage int,
flashMode text,
timestamp timestamp,
primary key (uniqueId, timestamp))
with clustering order by (timestamp desc);
In this, I have the value of uniqueId as 'test', which means I just have only one partition named 'test'.
When I perform the getEndPoints, I could see that the data resides in only 3 nodes.
./nodetool getendpoints keyspacename nodestat test
192.168.1.1
192.168.1.2
192.168.1.3
So my partition data is available in 3 nodes, I did the secondary index on one of the columns,
CREATE CUSTOM INDEX nodeIp_idx ON nodestat(flashMode)
So now when I perform
select * from nodestat where uniqueId = 'test' AND flashMode = 'yes'
How many nodes will it go to collect the data?
select * from nodestat where uniqueId = 'test' AND flashMode = 'yes'
Based on this query, you are using partition key along with a secondary index. Hence it will behave like a normal query based on the chosen consistency level. That is if "local_one" only one node will be enough to respond and if "local_quorum" a quorum of nodes in that dc will have to respond. Secondary index will further assist to narrow down the resultset.
Remember secondary index are local to data in every node of that cluster and hence present in all nodes of the cluster. Additional reference here.
In short, there is no direct correlation of Replication factor to Secondary index.
The problem is Cassandra cannot handle a lot of tables per cluster (> 1000). I was looking for any means to reduce the number of tables, and one of them was grouping multiple tables that share the same structure to gether.
Let say if we have two table A and B
create table A (
key text,
value text,
primary key(key)
)
and
create table B (
key text,
value text,
primary key(key)
)
We can group them together by adding one more partition key
create table Shared (
original_table_name text, // either 'A' or 'B'
key text,
value text,
primary key(original_table_name, key)
)
My question is, is it a good pattern and what are the consequences of modelling data this way?
Please elaborate what you mean by alot of tables, because our production is running with 50+ tables, and I don't see any issue with it.
Anyways, if your application is using atlot of tables, then most probable cause of it it, normalized table. In cassandra you should always create denormalized tables, because of no join facility. Cassandra is built for very fast writes, so, you can count on it and not worry about that.
Now regarding the new design, I don't see any problem with that, only thing is your partition key should be combination of (table_name, key) and not just table_name so that it will be evenly distributed across nodes.
And ofcourse to query each time, you will have to specify table_name + key.
Using older versions of Cassandra, we were expected to create our own sorted rows using a special row of columns, because columns are saved sorted in Cassandra.
Is Cassandra 3.0 with CQL using the same concept when you create a PRIMARY KEY?
Say, for example, that I create a table like so:
CREATE TABLE my_table (
created_on timestamp,
...,
PRIMARY KEY (created_on)
);
Then I add various entries like so:
INSERT INTO my_table (created_on, ...) VALUES (1, ...);
...
INSERT INTO my_table (created_on, ...) VALUES (9, ...);
How does Cassandra manage the sort on the PRIMARY KEY? Will that happens on all nodes, or only one set (what I call a set is the number of replicates, so if you have a cluster of 100 nodes with a replication factor of 4, would the primary key appear on 100 nodes, 25, or just 4? With older versions, it would only be on 4 nodes.)
In your case the primary key is the partition key, which used to be the row key. Which means the data your are inserting will be present on 4 out of 100 nodes if the replication factor is set to 4.
In CQL you can add more columns to the primary key, which are called clustering keys. When querying C* with CQL the result set might contain more than one row for a partition key. Those rows are logical and are stored in the partition of which they share the partition key (but vary in their clustering key values). The data in those logical rows is replicated as the partition is.
Have a look at the example for possible primary keys in the official documentation of the CREATE TABLE statement.
EDIT (row sorting):
C* keeps the partitions of a table in the order of their partition key values' hash code. The ordering is therefor not straight forward and results for range queries by partition key values are not what you would expect them to be. But as partitions are in fact ordered you still can do server side pagination with the help of the token function.
That said, you could employ the ByteOrderedPartitioner to achieve lexical ordering of your partitions. But it is very easy to create hotspots with that partitioner and it is generally discouraged to use it.
The rows of a given partition are ordered by the actual values of their clustering keys. Range queries on those behave as you'd expect them to.