I am running, 'nodetool cfstat' and it returns me list of Keyspace and cfstats for each column family on that node/machine. The cfstats results has, SSTable count value for each column families. My question is, Will SSTable value for column family be same across node, specially for those CF, whose SSTable count is 0? The reason is if SSTable for a column family is 0, then It is safe to drop those column families.
The cfstats output is per node, so is only valid for the node that nodetool connected to. To get the total SSTable count, you will need to sum them across all nodes. A column family with no SSTables on any node is empty.
Because the counts are per node, they will be inflated by replication. So e.g. the sum of all key counts will be approximately a factor of the replication factor higher than the actual key count.
Related
I need to check which tables are empty in Cassandra over a few keyspaces and there are over 20 tables. I could do a count on every single table but that's a little troublesome...
Is there a way to see the counts for every single table across the different keyspaces without typing all the 20+ queries? I have the tables in a comma-delimited list if that helps.
Edit: I used python to help with this but am interested in a Cassandra solution.
You could also try it at the command line level with nodetool tablestats:
ยป bin/nodetool tablestats stackoverflow | grep "Table\:\|partitions"
Table: cart_product
Number of partitions (estimate): 1
Table: keyvalue
Number of partitions (estimate): 0
Table: last_message_by_group
Number of partitions (estimate): 2
Table: mytable
Number of partitions (estimate): 5
Table: temps_by_item
Number of partitions (estimate): 2
Table: users
Number of partitions (estimate): 1
Granted, this is only reflective of the node that the command is on. But you should be able to ascertain whether or not a table is empty by this or several of the other statistics available in the tablestats output.
I do a select with tracing ON and see:
Skipped 0/1 non-slice-intersecting sstables
included 0 due to tombstones [ReadStage-<N>]
So is it working to ignore tombstones? The trace:
Read 0 live rows and 2 tombstone cells
is clear: it is reading tombstones
Let's say there was a Column A.
You added value x to Column A.
Then you deleted Column A.
Instead of immediately deleting value x, Cassandra will add a marker for Column A which is called tombstone. The tombstone is also an individual record in itself just like the original value x.
Let's say the two updates were written in different sstables (Cassandra storage).
Now when you are reading the value, Cassandra will get the value x and the tombstone for Column A. It will see that tombstone was written after the value x so it will not return any value.
Skipped 0/1 non-slice-intersecting sstables
included 0 due to tombstones
This is basically confirming the same.
Based on talking to some Cassandra admins:
" Skipping sstables is Cassandra telling us it eliminated the tombstones efficiently, this is ok
" Deleting everything in a partition in general helps ensure Cassandra is not bogged down with tombstones
I was trying to read the data from 100 partitions of one of the column family of Cassandra from Spark and it has 150-200 MB partition size. So, when I try to run this SparkJob on this column family read latency of rest of the column families also hamper. What could be the possible reasons for such behavior and solution for the same?
I am new to cassandra, As per my understanding depending on the configured partitioner(murmur3partitioner or randomaccess partitioner) there is a partitions limit per table. if we configure keyspace with murmur3partitioner which would enforce the partitions limit of 2^63 partitions per table. while inserting the row, if the new insertion tries to create new partition beyond the limit, the insertion would fail(means if I get unique combinations of row keys more than 2^63 per table).
Can anyone please clarify, Is my understanding about partitions limit on column family is correct ?
And also as per my understanding there is no way to increase the partitions limit even by adding nodes into the cluster, please correct me if I am wrong.
The range of values for the murmur3 partitioner is actually -2^63 to +2^63-1 That's a massive number. You aren't going to run out of values in any practical sense. No worries.
is this 2 billion cells per partition limit still valid?
http://wiki.apache.org/cassandra/CassandraLimitations
Let's say you save 16 bytes on average per cell. Then you "just" can persist 16*2e9 bytes = 32 GB of data (plus column name) on one machine!?
Or if you imagine a quadratic table you will be able to store 44721 rows with 44721 columns each!?
Doesn't really sound like Big Data.
Is this correct?
Thanks!
Malte
The 2 billion cell limit is still valid and you most likly want to remodel your data if you start seeing that many cells per partition.
The maximum number of cells (rows x columns) in a single partition is
2 billion.
A partition is defined by they partition key in CQL and will define where a particular piece of data will live. For example if I had two nodes with a fictional range of 0-100 and 100-200. Partition keys which hashed to between 0 and 100 would reside on the first node and those with hashed value of between 100 and 200 would reside on the second node. In reality Cassandra uses the Murmur3 algorithm to hash primary keys generating values between -2^63 and 2^63-1.
The real limitation tends to be based on how many unique values you have for your partition key. If you don't have a good deal of uniqueness within a single column many users combine columns to generate more uniqueness(composite primary key).
http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/create_table_r.html
More info on hashing and how C* holds data.
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architecturePartitionerAbout_c.html