Autocompactions can be enabled or disabled using nodetool enableautocompaction and disableautocompaction. But is there any way to know the status? I do not see any nodetool command which will show the status.
There is no mechanism to tell short of taking a heap dump currently. Best option is just to use nodetool enableautocompaction if you want it on regardless to be safe or setting alerting on compaction pending tasks.
I think you are searching for one of the below commands:
1.CompactionHistory
Description: Provides the history of compaction operations.
CompactionStats
Provide statistics about a compaction. The total column shows the total number of uncompressed bytes of SSTables being compacted. The system log lists the names of the SSTables compacted.
As suggested Chris, nodetool compactionstats will probably help if autocompaction enabled then you can see some running task and pending task may be 0 or any number but if autocompaction disabled then you can see many pending task and no running task on nodetool compactionstats.
Related
While running full repair on a cassandra cluster with 15 nodes, RF=3 and 3racks(single datacenter) using command ./nodetool repair -pr -full -seq I can see multiple validation compactions running at the same time (>10). Is there any way to limit simultaneous validations in cassandra 3.11.1 like we can limit normal compactions?
As the cluster size has increased, I limited repairs to run table by table and also used -pr and -seq to restrict load on the nodes. But now, the load is very high due to concurrent validation compactions. Need a way to restrict concurrent validation compactions to reduce load on nodes during repairs. I'm also exploring reaper to manage repairs but need to find some workaround for the load issues till I use reaper.
If you're seeing (validation) compactions becoming cumbersome, there are two settings that you should look at:
compaction_throughput_mb_per_sec
concurrent_compactors
compaction_throughput_mb_per_sec
This is the main tuneable setting for compaction. I mentioned this setting in a related answer here: Advise on stopping compaction to reduce slowness
I would recommend checking this setting, and then reducing it until contention is resolved. Or, you could try to set compaction throughput to 1 (the lowest setting) during the day. Then, raise it back up once business hours are over.
% bin/nodetool setcompactionthroughput 1
% bin/nodetool getcompactionthroughput
Current compaction throughput: 1 MB/s
But definitely check it first, just to see what you're running at, and maybe consider halving that and check the effect.
concurrent_compactors
So this defaults to the smaller of (number of disks, # number of cores), with a minimum of 2 and a maximum of 8. There is some solid advice out there around forcing this to a value of 1 if you're using spinning disks, and maybe set it to 4 for SSDs. The default is usually fine, but if it's too high, compactions can overwhelm disk I/O.
tl;dr;
Focus on compaction throughput for now. My advise is to check it, lower it, observe it, and repeat until things improve.
I was trying to make a configuration change in cassandra-env.sh file.
For the change to be effective I have to bounce my cassandra nodes. But the nodes I want to bounce are running compactions.
So what will happen to these pending tasks if I bounce my nodes when the compactions are in progress?
As said before, compactions will stop when you bounce the nodes. But it will pick up once you start the nodes again. No warn on that. If you have really long compactions ongoing, you might want to wait for those to finish.
nodetool compactionstats -H is your friend to check the current status and expected ETA of the current compactions.
If you want your nodes to startup faster, flush nodetool flush, drain nodetool drain and then stop the node. (This way you clear the commitlog).
Its very simple: your pending compaction will be failed for the dependent nodes. For development or test environment, you may do whatever you want but for production environment we prefer, complete your all pending compaction tasks and then go for the changes.
If you are in a hurry then go for Nodetool Stop, it'll stop your compaction process then go for the changes.
I have scheduled incremental repair for everyday. But while the repair is going on, our monitoring system reports COMPACTIONEXECUTOR_PENDING tasks.
I am wondering, if I can introduce a check, to see, if compaction is not running, before I trigger repair.
I should be able to check if compaction is running by parsing output of nodetool netstats and compactionstats command output.
I will proceed with repair if both of the following checks passes:
nodetool netstats output contains Not sending any streams.
nodetool compactionstats output contains pending tasks: 0
But I want to get some expert opinion before I proceed.
Is my understanding correct?
I don't want to get into situation, in which, these checks are failing always and repair process is not getting triggered at all.
Thanks.
Compaction is occurring regularly in Cassandra. So I'm a bit scared that only triggering repair when pending_compactions=0 will result in repair not running enough. But it depends on your traffic of course, e.g. if you have few writes you won't do many compactions. You should probably add a max wait time for pending_compactions=0 so that after a specified time if the condition is not true repair will run anyway.
To answer your question. Nodetool uses JMX to fetch MBeans in Cassandra. You can see all available MBeans here: http://cassandra.apache.org/doc/latest/operating/metrics.html
You want this MBean:
org.apache.cassandra.metrics:type=Compaction name=PendingTasks
You can create your own JMX Client like this: How to connect to a java program on localhost jvm using JMX?
Or you can use jmxterm: https://github.com/jiaqi/jmxterm
My understanding is you could use it like this:
java -jar jmxterm-1.0.0-uber.jar
get -b org.apache.cassandra.metrics:type=Compaction name=PendingTasks
I noticed an increase in the number of open files on my cassandra cluster and went to check the health of it. Nodetool status reported only 300gb in use per node of the 3TB each has allocated.
Shortly there after i began to see HEAP OOM errors showing up in the cassandra logs.
These nodes had been running for 3-4 months no issue, but had a series of test data populate and then dropped from them.
After checking the harddrives via the df command i was able to determine they were all between 90-100% filled in a jboded scenario.
edit: further investigation shows that the remaining files are in the 'snapshot' subfolder and the data subfolder itself has no db tables.
My question is, has anyone seen this? Why did compaction not free these tombstones? Is this a bug?
Snapshots aren't tombstones - they are a backup of your data.
As Highstead says you can drop any unused snapshots via the clearsnapshot command.
You can disable the automatic snapshot facility via the cassandra.yaml
https://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__auto_snapshot
Also check if you may have non-default true for snapshot_before_compaction
Snapshots occur over the lifetime of the cassandra cluster. These snapshots are not captured in a nodetool status but still occupy space. In this case the snapshots consuming all the space were created when a table was dropped.
To retrieve a list of current snapshots use the command nodetool listsnapshots
This feature can be disabled through editing /etc/cassandra/cassandra-env.sh and setting auto_snapshot to false. Alternatively these snapshots can be purged via the command nodetool clearsnapshot <name>.
Currently I am debugging performance issue with Apache Cassandra. when Memtable for the column family is filled, it is queued to be flushed to SSTable. This flushing happens often when you perform massive writes.
When this queue is filled up, writes are blocked until next successful completion of flush. This indicates that your node cannot handle writes it is receiving.
Is there a matrix in nodetool indicating this behaviour? In other words, I want a data indicating a node cannot keep up with writes it is receiving.
Thanks!!
Thats not really true for a couple years. The active memtable is switched and a new memtable takes its position as live. New mutations occur on this live memtable while the "to be flushed" memtables are included in local reads. The MemtableFlushWriter thread pool has the flush tasks queued on it. So you can see how many are pending there (under tpstats). The mutations backing up you can also see under the MutationStage.
Ultimately
nodetool tpstats
Is likely what your looking for.
I want a data indicating a node cannot keep up with writes it is receiving.
Your issue is likely bound to disk I/O not being able to handle the throughput --> flushes of memtables queue up --> writes are blocked
the command dstat is your friend to investigate I/O issues. Some others linux commands may be also handy. Read this excellent blog post from Amy Tobey: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html
Is there a matrix in nodetool indicating this behaviour?
nodetool tpstats
I believe you're looking for tp (thread pool) stats.
nodetool tpstats
Typically blocked FlushWriters indicates that your storage system is having trouble keeping up with the write workload. Are you using spinning disks by chance? You'll also want to keep an eye on iostat in this case as well.
Here's the docs for tpstats: https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsTPstats.html