I'm trying to use SSTableLoader to migrate tables from a Cassandra 2.1 cluster to a Cassandra 3.11 and while some SStables are loaded successfully, I keep hitting weird errors with others.
I'm trying to repeatedly load the same SSTables, sometimes I get a generic
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
Other times
org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
However, if I check in system.log I always find this error:
java.lang.IllegalArgumentException: No column name component found in
cell name
Tried searching for the error above, did not find any helpful details.
I tried nodetool repair on the tables in the 2.x cluster but after a couple of hours it looks like it's still not done. I would try to use nodetool scrub but not sure if this will result in data loss.
The problem is that the sstable format changed in Cassandra 3.0 and so you can't just stream sstables in the 2.1 format to a 3.x node as the sstable format.
The official (and better) way is that you would first need to upgrade your node(s) to a newer C* version and then run nodetool upgradesstables -a.
Related
I'm trying to migrate data from Cassandra to ScyllaDB from snapshot using sstableloader and data in some tables gets loaded without any error but when verifying count using PySpark, it gives less rows in ScyllaDB than in Cassandra. Help needed!
I work at ScyllaDB
There are two tools that can be used to help find the differences:
https://github.com/scylladb/scylla-migrate (https://github.com/scylladb/scylla-migrate/blob/master/docs/scylla-migrate-user-guide.md) you can use the check mode to find the missing rows.
https://github.com/scylladb/scylla-migrator is a tool for migration from alive CQL clusters one to another (Cassandra --> Scylla) will work that also supports validation (https://github.com/scylladb/scylla-migrator#running-the-validator). There is a blog series on using this tool https://www.scylladb.com/2019/02/07/moving-from-cassandra-to-scylla-via-apache-spark-scylla-migrator/.
Please post a bug on https://github.com/scylladb/scylla/issues if indeed there are missing rows.
Solved this problem by using nodetool repair on Cassandra keyspace, took snapshot and loaded the snapshot in ScyllaDB using sstableloader.
I see some data inconsistency after moving data to a new cluster.
Old cluster has 9 nodes in total and each has got 2+ TB of data on it.
New cluster has same set of nodes as old and configuration is same.
Here is what I've performed in order:
nodetool snapshot.
Copied that snapshot to destination
Created a new Keyspace on Destination Cluster.
Used sstableloader utility to load.
Restarted all nodes.
After successful completion of transfer, I ran few queries to compare(Old vs New Cluster) and found out that the new cluster is not consistent but the data I see is properly distributed on each node (nodetool status).
Same query returns different sets of results for some of the partitions and I get zero rows first time, second time 100 rows,200 rows and eventually it becomes consistent for few partitions and record count matches with old cluster.
Few partitions have no data in the new cluster where as old cluster has data for those partitions.
I tried running queries on cqlsh with CONSISTENCY ALL but the problem still exist.
Did i miss any important steps to consider before and after?
Is there any procedure to find out the root cause of this?
I am currently running "nodetool repair" but I doubt if that could solve as I tried with Consistency ALL.
Highly Appreciate your help!
The fact that the results eventually becomes consistent indicates that the replicas are out-of-sync.
You can verify this by reviewing the logs around the time that you were loading data, particularly for dropped mutations. You can also check the output of nodetool netstats. If you're seeing blocking read repairs, that's another confirmation that the replicas are out-of-sync.
If you still have other partitions you can test, enable TRACING ON in cqlsh when you query with CONSISTENCY ALL. You will see if there are digest mismatches in the trace output which should also trigger read repairs. Cheers!
[EDIT] Based on your comments below, it sounds like you possibly did not load the snapshots from ALL the nodes in the source cluster with sstableloader. If you've missed loading SSTables to the target cluster, then that would explain why data is missing.
In Cassandra 2.1 cluster, data format is ka and post upgrade to Cassandra 3.11, I see the new sstables are written in md format. For the time-series data that is going to be expired in 3 months time, can I skip running the nodetool upgradesstables?
I validated the data reads are working fine from the older ka format sstables after upgrade. The reason I want to skip the upgrade is from other threads, I know it is going to take a lot of time for format conversion and anyway this data is going to expire in 3 months.
I don't think it's mandatory to run nodetool sstablesupgrade, Cassandra 3 will be able to work with old SSTables, but you lose a lot of advantages of Cassandra 3 (for instance, space consumption is significantly reduced). Also Datastax has a warning in their upgrade documentation:
WARNING: Failure to upgrade SSTables when required results in a significant performance impact and increased disk usage and possible data loss. Upgrading is not complete until the SSTables are upgraded.
We are using cassandra 2.1.14. Currently large partition warning is seen on system.hints table.
How to make sure that system.hints table doesn't have wide partitions ?
Note that we don't want to upgrade to cassandra 3 now.
Is there a periodic way to clean up system.hints ?
Will this cause I/O spike in cassandra cluster ?
Log:
Compacting large partition system/hints:
10ad72eb-0240-4b94-b73e-eb4dc2aa759a (481568345 bytes)
How to make sure that system.hints table doesn't have wide partitions?
There's not really much you can do about that. system.hints is partitioned on target_id which is the hostID of the target node. If 10000 hints build up for one node, there's really no other place for them to go.
Is there a periodic way to clean up system.hints?
As mentioned above, hints should TTL after 3 hours. This failsafe is meant to keep the system.hints table from getting too out of control. But it's not at all fool-proof.
One way to be sure, would be to clear them out via nodetool:
nodetool truncatehints
Will this cause I/O spike in cassandra cluster?
Running nodetool truncatehints is fairly innocuous. I haven't noticed a spike from running it before.
I have a table with huge tombstones.So when i performed a spark job (which reads) on that specific table, it gave result without any issues. But i executed same query using cqlsh it gave me error because huge tombstones present in that table.
Cassandra failure during read query at consistency one(1 replica
needed but o replicas responded ,1 failed
I know tombstones should not be there, i can run repair to avoid them , but apart from that why spark succeeded and cqlsh failed. They both use same sessions and queries.
How spark-cassandra connector works? is it different than cqlsh?
Please let me know .
thank you.
The Spark Cassandra Connector is different to cqlsh in a few ways.
It uses the Java Driver and not the python Driver
It has significantly more lenient retry policies
It full table scans by breaking the request up into pieces
Any of these items could be contributing to why it would work in SCC and not in CQLSH.