Does sstableloader insert pairs, replicated over different sstables, uniquely? - cassandra

Question

I used sstableloader to import snapshots from a cluster of 4 nodes configured to replicate four times. The folder structure of the snapshots is:
<keyspace>/<tablename>/snapshots/<timestamp>
Ultimately there were 4 timestamps in each snapshot folder, one for each node. They appeared in the same snapshot-directory, because I tar-gzipped them and extracted the snapshots of all nodes in the same directory.
I noticed that sstableloader couldn't handle this, because the folder should end with / as an assumption of the tool. Hence I restructured the folders to
<timestamp>/<keyspace>/<tablename>
And then I applied sstableloader to each timestamp:
sstableloader -d localhost <keyspace>/<tablename>
This seems hacky, as I restructured the folder, and I agree, but I couldn't get the sstableloader tool to work otherwise. If there is a better way, please let me know.
However, this worked:
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of <keyspace>/<tablename>/<keyspace>-<tablename>-ka-953-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-911-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-952-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-955-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-951-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-798-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-954-Data.db <keyspace>/<tablename>/<keyspace>-<tablename>-ka-942-Data.db to [/127.0.0.1]
progress: [/127.0.0.1]0:8/8 100% total: 100% 0 MB/s(avg: 7 MB/s)
Summary statistics:
Connections per host: : 1
Total files transferred: : 8
Total bytes transferred: : 444087547
Total duration (ms): : 59505
Average transfer rate (MB/s): : 7
Peak transfer rate (MB/s): : 22
So I repeated the command for each timestamp (and each keyspace and each tablename), and all the data got imported on the single-node setup of my laptop (default after installing cassandra on ubuntu from ppa).
Possibly important to note, before importing with sstableloader I initialized the keyspace with replication 1, instead of 3 on the 4-node-cluster server(s).
CREATE KEYSPACE <keyspace> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;
Nevertheless, I noticed this:
$ du -sh /var/lib/cassandra/data/<keyspace>/<tablename>-e08e2540e82a11e4a64d8d887149c575/
6,4G /var/lib/cassandra/data/<keyspace>/<tablename>-e08e2540e82a11e4a64d8d887149c575/
However, when I query the size of the snapshots:
$ du -sh 142961465*/<keyspace>/<tablename>
2,9G 1429614655449/<keyspace>/<tablename>
3,1G 1429614656562/<keyspace>/<tablename>
2,9G 1429614656676/<keyspace>/<tablename>
2,7G 1429614656814/<keyspace>/<tablename>
The snapshots have a total size of 11.6GB, with replication 3 the essential part of the data should be ~3.9GB, however the /var/lib/cassandra/data/<keyspace>/<tablename>-e08e2540e82a11e4a64d8d887149c575/ folder is significantly larger. Why is this the case? How smart is cassandra / sstableloader? Are different redundant pairs filtered somehow?

Answer 1

You're almost certainly seeing Cassandra doing the right thing: It's importing each sstable, and letting timestamp resolution win.
It's probably the case that you various sstables had various older versions of data: older sstables had obsolete, shadowed cells, and newer sstables had new, live cells. As sstableloader pushes that data into the cluster, the oldest data is written first, and then obsoleted by the newer data as it's replayed. If there are deletes, then there will also be tombstones, which actually ADD space usage on top of everything else.
If you need to purge that obsolete data, you can run compaction (either using nodetool compact if that's an option for you - your data set is small enough it's probably fine - or something like http://www.encql.com/purge-cassandra-tombstones/ to do a single sstable at a time, if you're space constrained).

Answer 2

We were having a similar issue:
nodetool cleanup
nodetool compact keyspace1.tabel1 (Note: Manual compaction is not recommended as per Cassandra Documentation, we did this as part of migration)
We also found that sstableloader was creating very large files, we used sstablesplit to break down table into smaller files
https://cassandra.apache.org/doc/latest/cassandra/tools/sstable/sstablesplit.html