I am using the Secondary index for one of the column in Cassandra table.,
Say I have a 5 node cluster (192.168.1.1, 192.168.1.2, 192.168.1.3, 192.168.1.4, 192.168.1.5) with the Keyspace replication factor as '3' and considering the following table,
CREATE TABLE nodestat (
uniqueId text,
totalCapacity int,
physicalUsage int,
flashMode text,
timestamp timestamp,
primary key (uniqueId, timestamp))
with clustering order by (timestamp desc);
In this, I have the value of uniqueId as 'test', which means I just have only one partition named 'test'.
When I perform the getEndPoints, I could see that the data resides in only 3 nodes.
./nodetool getendpoints keyspacename nodestat test
192.168.1.1
192.168.1.2
192.168.1.3
So my partition data is available in 3 nodes, I did the secondary index on one of the columns,
CREATE CUSTOM INDEX nodeIp_idx ON nodestat(flashMode)
So now when I perform
select * from nodestat where uniqueId = 'test' AND flashMode = 'yes'
How many nodes will it go to collect the data?
select * from nodestat where uniqueId = 'test' AND flashMode = 'yes'
Based on this query, you are using partition key along with a secondary index. Hence it will behave like a normal query based on the chosen consistency level. That is if "local_one" only one node will be enough to respond and if "local_quorum" a quorum of nodes in that dc will have to respond. Secondary index will further assist to narrow down the resultset.
Remember secondary index are local to data in every node of that cluster and hence present in all nodes of the cluster. Additional reference here.
In short, there is no direct correlation of Replication factor to Secondary index.
Related
I had two following tables ( taken from Cassandra Definitve Guide , https://gist.github.com/jeffreyscarpenter/761ddcd1c125dfb194dc02d753d31733 } - What is guaranteed with respect to the folloowing tables assuming they had the same partition key ?
Can we safely assume the data for both the tables present in the same node as long as the partition key is same ? as both tables contain same partition key.
Ok , and as tables are different from each other , will they be stored in different partitions or same partition in the "same" node
https://gist.github.com/jeffreyscarpenter/761ddcd1c125dfb194dc02d753d31733
CREATE TABLE hotel.pois_by_hotel (
poi_name text,
hotel_id text,
description text,
PRIMARY KEY ((hotel_id), poi_name)
) WITH comment = 'Q3. Find pois near a hotel';
CREATE TABLE hotel.available_rooms_by_hotel_date (
hotel_id text,
date date,
room_number smallint,
is_available boolean,
PRIMARY KEY ((hotel_id), date, room_number)
) WITH comment = 'Q4. Find available rooms by hotel / date';
if both tables have the same partition key, then the same value will be mapped into the same token. If tables are in the same keyspace, then yes - they will be on the same node(s). If they are in the different keyspaces, then there could be a partial overlap - if replication factor is different, for example, one keyspace has higher RF (like, KS1 has RF=2, and KS2 has RF=3, then 2 nodes will have replicas for both keyspaces, and 3rd node will have only for KS2).
Each table will have its own set of the files on disk, so although they have the same "logical partitions", on disk they are in different files. You can always look into data files, something like, /var/lib/cassandra/data/<keyspace>/<table>-<table-uuid>/
For monitoring some distributed software I insert their monitoring data into Cassandra table. The columns are metric_type, metric_value, host_name, component_type and time_stamp. The scenario is I collect all the metrics for all the nodes in every second. The time in uniform for all nodes and their metrics. The keys(that differentiate rows) are host_name, component_type, metric_type and time_stamp. I design my table like below:
CREATE TABLE metrics (
component_type text,
host_name text,
metric_type text,
time_stamp bigint,
metric_value text,
PRIMARY KEY ((component_type, host_name, metric_type), general_timestamp)
) WITH CLUSTERING ORDER BY (time_stamp DESC)
where component_type, host_name and metric_type are partitions key and time_stamp is clustering key.
The metrics table is suitable for the queries that gets some data according to their timestamp just for a host_name or a metric_type or a component_type, as using partition keys Cassandra will find the partition that data are stored and using clustering key will fetch data from that partition and this is the optimal case for Cassandra queries.
Besides that, I need a query that fetches all data just using time_stamp. For example :
SELECT * from metrics WHERE time_stamp >= 1529632009872 and time_stamp < 1539632009872 ;
I know the metric table is not optimal for the above query, because it should search every partition to fetch data. I guess in this situation we should design another table with the time_stamp as partition key, so data will be fetched from one or some limited number of partitions. But I am not certain about some aspects:
Is it optimal to set time_stamp as partition key? because of I insert data into the database every second and the partition key numbers will be a lot!
I need my queries to be interval on time_stamp and I know interval conditions are not allowed in partition keys, just allowed on clustering keys!
So what is the best Cassandra data model for such time series data and query?
Using time_stamp as partition key is not optimal in my opinion, as it would create a lot of partitions.
I would propose 2 solutions:
1) Go with a "week_first_day" as partition key. You would have to compute the correct week_first_day keys on your application side and then emit multiple select queries.
2) You could use ElasticSearch on top of cassandra. Cassandra remains the primary data source, but you have the freedom, to do complex selects. If you are interested, I would recommend to take a look at Elassandra .
I am considering Cassandra as an intermediate storage during my ETL job to perform data deduplication.
Let's imagine I have a stream of events, each of them have some business entity id, timestamp and some value. I need to get only latest value in terms of in-event timestamp for each business key, but events may come unordered.
My idea was to create staging table with business id as a partition key and timestamp as a clustering key:
CREATE TABLE sample_keyspace.table1_copy1 (
id uuid,
time timestamp,
value text,
PRIMARY KEY (id, time)
) WITH CLUSTERING ORDER BY ( time DESC )
Now if I insert some data in this table I can get latest value for some given partition key:
select * from table1 where id = 96b29b4b-b60b-4be9-9fa3-efa903511f2d limit 1;
But that would require to issue such query for every business key I'm interested in.
Is there some effective way I could do it in CQL?
I know we have an ability to list all available partition keys (by select distinct id from table1). So if I look into storage model of Cassandra, getting first row for each partition key should not be too hard.
Is that supported?
If you're using a version after 3.6, there is an option on your query named PER PARTITION LIMIT (CASSANDRA-7017) which you can set to 1. This won't auto complete in cqlsh until 3.10 with CASSANDRA-12803.
SELECT * FROM table1 PER PARTITION LIMIT 1;
In a word: no.
The partitioning key is why Cassandra can work essentially any amount of data: It decides where to put/look for data using the hash of the partitioning key. That is why CQL SELECTs always need to do an equality filter on the entire partitioning key. In order to find the first time for each id, Cassandra would have to ask all nodes for any partition of the data, then perform a complex operation on each of them. Relational databases allow this, Cassandra does not. All it allows are full table scans (SELECT * from table1), or partition scans (SELECT DISTINCT id FROM table1), but those cannot* be linked to any complex operation.
*) I am omitting ALLOW FILTERING here, since it does not help in this context.
Using older versions of Cassandra, we were expected to create our own sorted rows using a special row of columns, because columns are saved sorted in Cassandra.
Is Cassandra 3.0 with CQL using the same concept when you create a PRIMARY KEY?
Say, for example, that I create a table like so:
CREATE TABLE my_table (
created_on timestamp,
...,
PRIMARY KEY (created_on)
);
Then I add various entries like so:
INSERT INTO my_table (created_on, ...) VALUES (1, ...);
...
INSERT INTO my_table (created_on, ...) VALUES (9, ...);
How does Cassandra manage the sort on the PRIMARY KEY? Will that happens on all nodes, or only one set (what I call a set is the number of replicates, so if you have a cluster of 100 nodes with a replication factor of 4, would the primary key appear on 100 nodes, 25, or just 4? With older versions, it would only be on 4 nodes.)
In your case the primary key is the partition key, which used to be the row key. Which means the data your are inserting will be present on 4 out of 100 nodes if the replication factor is set to 4.
In CQL you can add more columns to the primary key, which are called clustering keys. When querying C* with CQL the result set might contain more than one row for a partition key. Those rows are logical and are stored in the partition of which they share the partition key (but vary in their clustering key values). The data in those logical rows is replicated as the partition is.
Have a look at the example for possible primary keys in the official documentation of the CREATE TABLE statement.
EDIT (row sorting):
C* keeps the partitions of a table in the order of their partition key values' hash code. The ordering is therefor not straight forward and results for range queries by partition key values are not what you would expect them to be. But as partitions are in fact ordered you still can do server side pagination with the help of the token function.
That said, you could employ the ByteOrderedPartitioner to achieve lexical ordering of your partitions. But it is very easy to create hotspots with that partitioner and it is generally discouraged to use it.
The rows of a given partition are ordered by the actual values of their clustering keys. Range queries on those behave as you'd expect them to.
For secondary index queries that the partition key is specified in the WHERE clause, does the secondary index lookup hits all cluster nodes, or just the node of the specified partition key?
If the latter is correct, then secondary index will be a good fit also for high cardinality fields (only for queries that satisfies the partition key).
EDIT: For example, for the following feed schema, query of a specific feed (feed_id specified) to retrieve existing or deleted feed items should be very efficient:
CREATE TABLE my_feed (
feed_id int,
item_id timeuuid,
is_deleted boolean,
data text,
PRIMARY KEY (feed_id, item_id)
) WITH CLUSTERING ORDER BY (item_id DESC);
CREATE INDEX my_feed_is_deleted_idx ON my_feed (is_deleted);
==> SELECT * FROM my_feed WHERE feed_id=1 AND is_deleted=false; --efficient?
If you hit a partition key first, then it won't be a cluster wide operation. Only the target partition will be hit. If you have wide rows with many rows in a partition, a secondary index will be an efficient way to filter them down once a partition is hit.
When and when not to use a secondary index and why is covered here: https://docs.datastax.com/en/dse/6.8/cql/cql/cql_using/useWhenIndex.html