I have a cluster which I am considering enabling incremental repair on. If anything goes wrong I'd like to disable incremental repair on every node. How do I do that?
Turn node off and use sstablerepairedset to remove the repair time for each sstable so that they will all be candidates for future compactions.
find '/path/cassandra/data/keyspace/table/' -iname "*Data.db*" > sstables.txt
sudo -u cassandra sstablerepairedset --is-unrepaired -f sstables.txt
Then just go back to using repair with no -inc or in later versions use the -full flag
Related
What would be the best way how to stop Cassandra nodes before application updates and database updates, and to afterwards start them? Currently I'm going through each node VM and stopping them using the command sudo service cassandra stop and then after the update is done, I'm doing the same, just starting them with command sudo service cassandra start. With few nodes it's OK, but I have 20 nodes on a server, and it takes some time to do that.
Is there any better/more efficient way to do that?
There are so many ways to do this - use for loop in shell + ssh, use pssh, etc. You just need to remember that it's better to perform rolling restart of the individual Cassandra nodes, so you need to have to wait until Cassandra starts again. Something like:
for host in `cat your-host-list.txt`; do
ssh user#${host} 'sudo service cassandra stop && your_update_command && sudo service cassandra start
done
or slightly easier with pssh:
pssh -l user -h your-host-list.txt -p 1 -t 0 'sudo service cassandra stop && your_update_command && sudo service cassandra start
-p 1 tells to execute only one command in parallel
t 0 instructs to wait for execution without timeout
We do run repair -pr on every DSC 2.1.15 node within gc_grace like this:
nodetool -h localhost repair -pr -par mykeyspc
But in the log it says full=true:
[2017-02-12 00:00:01,683] Starting repair command #11, repairing 256 ranges for
keyspace mykeyspc (parallelism=PARALLEL, full=true)
Would have expected that a -pr didn't run a full repair or how to read this log?
It means full as in "not incremental". Can think of it as its fully repairing the data in those ranges, not just the unrepaired data. It is confusing argument naming. The -pr means its just repairing the primary ranges though so you still need to do that on each node.
When I run nodetool clearsnapshot I get the normal "Requested clearing snapshot(s)" message, but the snapshot is never removed. What can I do to troubleshoot why this is occurring? Is it acceptable for me to just manually remove the snapshot directories from the tablespace directories as a workaround for this?
nodetool clearsnapshot 1472489985541
Requested clearing snapshot(s) for [1472489985541]
nodetool listsnapshots | awk '{print $1}' | grep ^1 | sort -u
1472489985541
1473165734236
1473690660090
1474296554367
Is it acceptable for me to just manually remove the snapshot directories from the tablespace directories as a workaround for this?
Yes, you can always safely remove the snapshots directories manually. They are just hard links to actual SSTables
In order to delete a snapshot from all keyspaces using the snapshot name, you must specify the -t flag in your clearsnapshot command.
I am using Jconsole for monitoring Cassandra. I can get value like how much load each keyspace is having.
I want to find out disk space usage for each node in a cluster by remotely.
Is there any way to do so?
A shell script can do the trick
for i in node1_ip node2_ip ... nodeN_ip
do
ssh user#$i "du -sh /var/lib/cassandra/data" >> /tmp/disk_usage.txt
done
Replace /var/lib/cassandra/data if your data folder is put somewhere else
Is there a way i can remove memsql node from the webUI itself ?
I mean removing 3 or 4 different memsql nodes on different systems is tiresome.
At this time MemSQL Ops only supports removing nodes from the command line. To make this easier, you can iterate over the MemSQL id's you are interested in. For example here is how I would delete all of the leaf nodes (escaped line break added for clarity):
memsql-ops memsql-list -q -r leaf \
| xargs -n 1 memsql-ops memsql-delete --delete-without-prompting
Or, if you just want to remove the nodes from the web-ui, but not delete the actual nodes you can use memsql-unmonitor in the same way as Joseph Victor mentioned.
memsql-ops memsql-list -q -r leaf \
| xargs -n 1 memsql-ops memsql-unmonitor
Hopefully we can get full management capabilities into the MemSQL Ops UI in an upcoming release.
Try memsql-ops memsql-unmonitor on the terminal?