cassandra table looks empty in cqlsh, but nodetool cfstats thinks otherwise - cassandra

Question

Using nodetool cfstats I can see that a particular table (table1) is using 59mb and has 545597 keys. Another related table (table2) is using 568mb and has 2,506,141 keys.
Using cqlsh, when I do select count( * ) from table1 it pauses for about 7 seconds then returns a count of 0. However, if I do select count( * ) from table2 it pauses for much longer and then returns a count of 2,481,669.
I also tried select * from table1 and select * from table2. The first takes 7 seconds then returns nothing. The second instantly starts paging through results.
I'm well aware these are expensive operations, however this is on a single dev server which has only this one Cassandra instance. It's a cluster of 1 and not meant for production. I just want to figure out why the values in table1 are invisible.
Is it possible that table1 actually has no values in it? That shouldn't be possible given that I just ran a job to add a bunch of values to it. I also ran "nodetool compact", so that should have eliminated all the tombstones and the cfstats should show what's actually there, right? Here are the cfstats for table1 after I ran nodetool compact:
SSTable count: 1
Space used (live): 59424392
Space used (total): 59424392
Space used by snapshots (total): 73951087
Off heap memory used (total): 806762
SSTable Compression Ratio: 0.28514022725059224
Number of keys (estimate): 545597
Memtable cell count: 393204
Memtable data size: 17877650
Memtable off heap memory used: 0
Memtable switch count: 3
Local read count: 5
Local read latency: 0.252 ms
Local write count: 545804
Local write latency: 0.013 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 611792
Bloom filter off heap memory used: 611784
Index summary off heap memory used: 180202
Compression metadata off heap memory used: 14776
Compacted partition minimum bytes: 216
Compacted partition maximum bytes: 310
Compacted partition mean bytes: 264
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 6.0
Maximum tombstones per slice (last five minutes): 7
If it helps, I'm using apache cassandra 2.2.0 on a linux server.

Answer 1

Cassandra saves all the data in files (sstables). For speeds, writes append data at the end of the files (the index certainly works differently, but they do not describe how those function...)
The deletion of data (or expiration in your case) does not remove the data from the files because it would otherwise mean a lot of large moves and tons of I/O. So instead of they just mark the entry as "dead" (hence they are called tombstones).
Once in a while, the compaction system comes in (Assuming you did not turn it off against that table) and compacts tables. That means it reads from the start of the file and moves live entries over dead ones. More or less, something like this assuming B gets deleted at some point (columns left to right represent different points in time):
Creation Deletion Compaction
A A A
B B-tombstone C
C C
If your table has too many tombstones, the compaction may fail (I do not understand why it could fail, but that's what I read). A table that fails compaction is marked as "do not ever compact", which is a big problem, if you ask me. And a table with half a million keys could very well be failing.
While the table is in the "Deletion" state (includes tombstones), a SELECT that goes over a tombstone still creates a TombStone memory object (do not ask me why, I have no idea, it looks like Cassandra would not work right otherwise...) Hence, the 7 seconds to read all the tombstones and create Java objects for each one of them.
The CQL interface includes a TRACE feature that can be used to see the number of tombstones you have in a table. It prints out a bunch of things that you'd like to know about.
TRACE ON;
SELECT COUNT( * ) FROM table1;