I want to perform a TRUNCATE on multiple tables with around 25 million records, in clustered, multi-data-center environment. I would just like some advice on steps to take before/after the truncate to ensure that there aren't huge discrepancies between the nodes.
According to this, a TRUNCATE deletes the SSTable holding the data. Does this mean that I'll need to set my consistency level to ALL before the truncate? Is a nodetool repair necessary after the operation?
Any advice would be greatly appreciated.
cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4
Ensure that all nodes are up before issuing this command.
Truncate will naturally propogate out over the cluster as long as all nodes remain up and available.
Truncate will automatically run as if CONSISTENCY is set to ALL which ensures that the command only returns after all nodes have deleted their data and fail if a node cannot be reached.
Repair is not necessary as there will not be any data to repair after the operation.
Related
I am currently writing tests in golang and I want to get rid of all the data of tables after finishing tests. I was wondering if it is possible to flush the data of all tables in cassandra.
FYI: I am using 3.11 version of Cassandra.
The term "flush" is ambiguous in this case.
In Cassandra, "flush" is an operation where data is "flushed" from memory and written to disk as SSTables. Flushing can happen automatically based on certain triggers or can be done manually with the nodetool flush command.
However based on your description, what you want is to "truncate" the contents of tables. You can do this using the following CQL command:
cqlsh> TRUNCATE ks_name.table_name
You will need to iterate over each table in the keyspace. For more info, see the CQL TRUNCATE command. Cheers!
I see some data inconsistency after moving data to a new cluster.
Old cluster has 9 nodes in total and each has got 2+ TB of data on it.
New cluster has same set of nodes as old and configuration is same.
Here is what I've performed in order:
nodetool snapshot.
Copied that snapshot to destination
Created a new Keyspace on Destination Cluster.
Used sstableloader utility to load.
Restarted all nodes.
After successful completion of transfer, I ran few queries to compare(Old vs New Cluster) and found out that the new cluster is not consistent but the data I see is properly distributed on each node (nodetool status).
Same query returns different sets of results for some of the partitions and I get zero rows first time, second time 100 rows,200 rows and eventually it becomes consistent for few partitions and record count matches with old cluster.
Few partitions have no data in the new cluster where as old cluster has data for those partitions.
I tried running queries on cqlsh with CONSISTENCY ALL but the problem still exist.
Did i miss any important steps to consider before and after?
Is there any procedure to find out the root cause of this?
I am currently running "nodetool repair" but I doubt if that could solve as I tried with Consistency ALL.
Highly Appreciate your help!
The fact that the results eventually becomes consistent indicates that the replicas are out-of-sync.
You can verify this by reviewing the logs around the time that you were loading data, particularly for dropped mutations. You can also check the output of nodetool netstats. If you're seeing blocking read repairs, that's another confirmation that the replicas are out-of-sync.
If you still have other partitions you can test, enable TRACING ON in cqlsh when you query with CONSISTENCY ALL. You will see if there are digest mismatches in the trace output which should also trigger read repairs. Cheers!
[EDIT] Based on your comments below, it sounds like you possibly did not load the snapshots from ALL the nodes in the source cluster with sstableloader. If you've missed loading SSTables to the target cluster, then that would explain why data is missing.
I am starting with cassandra and have had some problems. I create keyspaces and tables to go playing, if I delete them and then run a describe keyspace they keep appearing to me. Other times I delete them and it tells me they don't exist, but I can't create them either because it says it exists.
Is there a way to clear that "cache" or something similar?
I would also like to know if through cqlsh I can execute a .cql file that is on my computer.
[cqlsh 5.0.1 | Cassandra 3.11.0 | CQL spec 3.4.4 | Native protocol v4]
This may be due to the eventually consistent nature of Cassandra. If you are on a small test cluster and just playing around you could try doing CONSISTENCY ALL in cqlsh which will force the nodes to become consistent.
You should always run delete or drop command manually with CONSISTENCY ALL so that it will reflect all the nodes and DCs. Also you need to wait for a moment to replicate into the cluster. Once replicated you will not get deleted data else you need to run a repair in the cluster.
I have a table with huge tombstones.So when i performed a spark job (which reads) on that specific table, it gave result without any issues. But i executed same query using cqlsh it gave me error because huge tombstones present in that table.
Cassandra failure during read query at consistency one(1 replica
needed but o replicas responded ,1 failed
I know tombstones should not be there, i can run repair to avoid them , but apart from that why spark succeeded and cqlsh failed. They both use same sessions and queries.
How spark-cassandra connector works? is it different than cqlsh?
Please let me know .
thank you.
The Spark Cassandra Connector is different to cqlsh in a few ways.
It uses the Java Driver and not the python Driver
It has significantly more lenient retry policies
It full table scans by breaking the request up into pieces
Any of these items could be contributing to why it would work in SCC and not in CQLSH.
I have a Cassandra datacenter which I'd like to run a full repair on. The datacenter is used for analytics/batch processing and I'm willing to sacrifice latencies to speed up a full repair (nodetool repair). Writes to the datacenter is moderate.
What are my options to make the full repair faster? Some ideas:
Increase streamthroughput?
I guess I could disable autocompation and decrase compactionthroughput temporarily. Not sure I'd want to that, though...
Additional information:
I'm running SSDs but haven't spent any time adjusting cassandra.yaml for this.
Full repairs are run sequentially by default. The state and differences of the nodes' datasets are stored in binary trees. Recreating these is the main factor here. According to this datastax blog entry, "Every time a repair is carried out, the tree has to be calculated, each node that is involved in the repair has to construct its merkle tree from all the sstables it stores making the calculation very expensive."
The only way I see to significantly increase the speed of a full repair is to run it in parallel or repair subrange by subrange. Your tag implies that you run Cassandra 2.0.
1) Parallel full repair
nodetool repair -par, or --parallel, means carry out a parallel repair.
According to the nodetool documentation for Cassandra 2.0
Unlike sequential repair (described above), parallel repair constructs the Merkle tables for all nodes at the same time. Therefore, no snapshots are required (or generated). Use a parallel repair to complete the repair quickly or when you have operational downtime that allows the resources to be completely consumed during the repair.
2) Subrange repair
nodetool accepts start and end token parameters like so
nodetool repair -st (start token) -et (end token) $keyspace $columnfamily
For simplicity sake, check out this python script that calculates tokens for you and executes the range repairs:
https://github.com/BrianGallew/cassandra_range_repair
Let me point out two alternative options:
A) Jeff Jirsa pointed to incremental repairs.
These are available starting with Cassandra 2.1. You will need to perform certain migration steps before you can use nodetool like this:
nodetool repair -inc, or --incremental means do an incremental repair.
B) OpsCenter Repair Service
For the couple of clusters at my company itembase.com, we use the repair service in DataStax OpsCenter which is executing and managing small range repairs as a service.