I have a machine the commit logs keeps increasing upto 7.8 GB and still growing, I checked a property commitlog_total_space_in_mb: 8192 which is commented in cassandra.yaml. I suspect it has to be the default.
1) What is the problem in growing commit log size?
2) Is it says my memtable treshold is not reached?
EDITED :
memtable_cleanup_threshold = 1/(memtable_flush_writers + 1) * (memtable_offheap_space_in_mb + memtable_heap_space_in_mb)
where recommended values are,
memtable_flush_writers - Smaller of number of disks or number of cores with a minimum of 2 and a maximum of 8, so in our case it is '8'
memtable_offheap_space_in_mb - 1/4 of the heap size, so in our case it is 2GB
memtable_heap_space_in_mb - 1/4 of the heap size, so in our case it is 2GB
so calculation will be,
memtable_cleanup_threshold = 1/(8 + 1) * 4096
memtable_cleanup_threshold = 455MB
Why does it didn't flush reaching to 455 MB and remove the commit log?
Yes, 8192MB (or 1/4 of log file disk space, whichever is smaller -- could apply if you have a smaller server) is the default. Source: Cassandra documentation on commitlog_total_space_in_mb.
To answer your questions:
(1) If the commitlog files continue to grow, you can run out of disk space.
(2) The configured threshold has not yet been met.
Edited after your additional questions to add:
Commitlogs aren't deleted when memtables are flushed.
Note the file size is preallocated based on your configuration size -- I think you already figured that out, but noting that here if anyone else tries to observe the file size via ls or similar.
If you nodetool drain or restart, they will be cleared. Otherwise, they will continue to grow to the max size and rotate around.
Here is a test to see what happens if you force a flush:
nodetool tablestats keyspace.table | grep "Memtable data size"
Memtable data size: 1292049
cat /var/lib/cassandra/commitlog/CommitLog-A.log | wc -l
10418
cat /var/lib/cassandra/commitlog/CommitLog-B.log | wc -l
0
nodetool flush
nodetool tablestats keyspace.table | grep "Memtable data size"
Memtable data size: 0
cat /var/lib/cassandra/commitlog/CommitLog-A.log | wc -l
10419
cat /var/lib/cassandra/commitlog/CommitLog-B.log | wc -l
0
nodetool drain
nodetool tablestats keyspace.table | grep "Memtable data size"
Memtable data size: 0
cat /var/lib/cassandra/commitlog/CommitLog-A.log | wc -l
no such file
cat /var/lib/cassandra/commitlog/CommitLog-B.log | wc -l
0
You see similar results if it flushes automatically based on memtable configuration. Throughout the following observed flushes, the commitlog was not purged either:
Related
I have 3 node cluster and upon checking the nodetool status; Load is just less than 100 GB on all three nodes. The replication factor is two and percentage own is 65-70% for all three.
However when I inspected the /data directory it is having index.db files for size more than 400 GB and the total size of the keyspace directory is more than 700GB.
Any idea on why a huge gap??
Let me know if any extra details are required :)
PS: nodetool listsnapshots command shows an empty list (No snapshots)
Analysis - we tried redeploying the setup but still the same results; tried researching this topic but no luck.
Expectation - I was expecting this difference in the load and the size of data directory to be negligible if not zero.
For example, the dir "/app/source"
There is an 100GB filesystem mount on "/"
So when I use "df /app/source" I can get the capacity is 100GB.
Then there is a dir "/app/source/video" and an 100GB filesystem mount on it.
Is there any easy way to get the real capacity (200GB) of "/app/source" ?
/app/source don't have a capacity of 200G, thus you cannot expect to see it. It's "real" capacity is 100G as the underlying disk capacity is 100G. If you think it has a capacity of 200G, then you expect to be able to store 200G of data in /app/source but you cannot ! You can store 100G in /app/source and 100G in /app/source/video
Maybe you would like to really merge the capacity of both partitions, for this you could use LVM.
Trying to merge only the reported numbers, which you could do with a simple script or alias (see below), would give you bad information and then you may try to add files on a full partition.
If at the end you still need the added total, maybe something like this can help:
# df -h --total /app/source /app/source/video | grep total | awk -F" " '{print $2}'
I have an 6 node cluster , each node is of 1000 GB in size. But the size of one node reached to 1000 GB randomly.On analysis i found only one key space gets filled & only 1 table of this keyspace size get increased from 200 GB to 800 GB (In 24 hours ) , which means someone execute operations on this table only . I want to figure out what operations had perform on this node which leads to this size increment ?
Are there any logs which can be looked at to see what operations were performed?
I guess how I would do this is to use "nodetool tablehistograms" to prove that you have large partitions for the table. Then I would go to the table directory and run "sstablemetadata" on some of the data files, locating ones that displays some large partition sizes.
One trick you could do once you find sstables that have larger partitions is:
sstabledump <sstable> | grep -n "\"key\" :"
What that will do is show you the line number every time the key switches, the larger the gap between lines, the more rows there are.
Here is an example:
sstabledump aa-483-bti-Data.db | grep -n "\"key\" :"
4: "key" : [ "PROCESSING" ],
65605: "key" : [ "PENDING" ],
8552007: "key" : [ "COMPLETED" ],
As you can see, the gap between PENDING and COMPLETED was much larger than PROCESSING and PENDING (65k lines v.s. 8M lines). So this tells me that the PROCESSING partition is relatively small compared to PENDING. The only mystery is how large is the COMPLETED one as there is no "ending" line. To get the total line count, run:
sstabledump aa-483-bti-Data.db | wc -l
16316029
Total line count is 16M. So COMPLETED goes from 8M to 16M, or about 8M lines. So the COMPLETED partition is large as well, about as large as the PENDING partition.
Looking at sstablemetadata to see if that matches up with the output, I see that it does:
sstablemetadata aa-483-bti-Data.db
Partition Size:
Size (bytes) | Count (%) Histogram
943127 (921.0 kB) | 1 ( 33) OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
129557750 (123.6 MB) | 1 ( 33) OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
155469300 (148.3 MB) | 1 ( 33) OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
I see two relatively large partitions and one small one. Bingo.
Maybe some of those can help you get to the bottom of your large partition(s).
With DataStax Enterprise, you should be able to turn on the Database Auditing feature. In fact, by configuring a logger class of CassandraAuditWriter, all activity gets written to the audit_log table in the dse_audit keyspace.
The data is organized by this PRIMARY KEY: ((date, node, day_partition), event_time); and has columns like username,table_name,keyspace_name,operation and others.
Check out the DataStax docs on that for configuration and query options.
As for (open source) Apache Cassandra, we use Ericsson's Cassandra Audit plugin for this functionality. By adding in the project's JAR, and making a couple of adjustments to the cassandra.yaml file, you can view the audit.logs for records like:
15:42:41.655 - client:'10.0.110.1'|user:'flynn'|status:'ATTEMPT'|operation:'DELETE FROM ecks.ectbl WHERE partk = ?'
I have a table in cassandra where I saved data using clients TTL = 1 month (tables TTL is 0), the table is configured with time window compaction strategy.
Everyday Cassandra cleaned one single sstable containing expired data from one month ago.
Recently I changed the clients TTL to 15 days, I was expecting cassandra to clean two sstables a day at some point, and release the space. But it keeps cleaning one sstable a day and keeping 15 days of dead data.
How do I know?
for f in /data/cassandra/data/keyspace/table-*/*Data.db; do meta=$(sudo sstablemetadata $f); echo -e "Max:" $(date --date=#$(echo "$meta" | grep Maximum\ time | cut -d" " -f3| cut -c 1-10) '+%m/%d/%Y') "Min:" $(date --date=#$(echo "$meta" | grep Minimum\ time | cut -d" " -f3| cut -c 1-10) '+%m/%d/%Y') $(echo "$meta" | grep droppable) ' \t ' $(ls -lh $f | awk '{print $5" "$6" "$7" "$8" "$9}'); done | sort
This command list all the sstables
Max: 05/19/2018 Min: 05/18/2018 Estimated droppable tombstones: 0.9876591095477787 84G May 21 02:59 /data/cassandra/data/pcc/data_history-c46a3220980211e7991e7d12377f9342/mc-218473-big-Data.db
Max: 05/20/2018 Min: 05/19/2018 Estimated droppable tombstones: 0.9875830312750179 84G May 22 15:25 /data/cassandra/data/pcc/data_history-c46a3220980211e7991e7d12377f9342/mc-221915-big-Data.db
Max: 05/21/2018 Min: 05/20/2018 Estimated droppable tombstones: 0.9876636061230402 85G May 23 13:56 /data/cassandra/data/pcc/data_history-c46a3220980211e7991e7d12377f9342/mc-224302-big-Data.db
...
For now I have been triggering the compactions manually using JMX, but I want all erased as it would normally do.
run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction /data/cassandra/data/keyspace/sstable_path
I think I figured it out. Had to run manual compaction on the oldest and newest sstable with all it's content expired, both sstables at the same time.
After a couple of days it cleanead everything.
How do I know it was running? Because when I tried to run forceUserDefinedCompaction on any other sstable in between it always returned null.
EDIT:
It didn't work, again Sstable count keeps increasing.
EDIT:
Using sstableexpiredblockers pointed to the sstables blocking the rest of compactions. After compacting these manually it automatically compacted the rest.
On one node out of 8, the blocking sstable wasn't been unlocked after compacting, so a "nodetool scrub" did the job (which scrubs all the sstables).
Problem Statement:-
I am getting this below exception-
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException:
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace
quota of /tmp is exceeded: quota=659706976665600 diskspace consumed=614400.1g
So I just wanted to know how much is the size of /tmp directory currently and because of that I am getting this exception. How can I see the free space in /tmp?
Update:-
bash-3.00$ df -h /tmp
Filesystem size used avail capacity Mounted on
rpool/tmp 10G 2.2G 7.8G 22% /tmp
I am puzzled right now why I am getting that exception then as it clearly states above that I have space available.
You can do (For SunOS)
# du -sh /tmp
To see how much it uses now, but that you already saw.
To see how much total, free and used space is on the partition where /tmp resides you can use:
# df -h /tmp
Note that filling up space is not the only thing that can prevent writing to filesystem.
Running out of inodes is another popular reason.
You can check that with
# df -i /tmp
for a in ls; do du -ch ${a} | grep total ; echo ${a}; done
try this but it takes time if the size of dir is in GBs
Managing HDFS Space Quotas
It’s important to understand that in HDFS, there must be enough quota space to accommodate an entire block. If the user has, let’s say, 200MB free in their allocated quota, they can’t create a new file, regardless of the file size, if the HDFS block size happens to be 256MB. You can set the HDFS space quota for a user by executing the setSpace-Quota command. Here’s the syntax:
$ hdfs dfsadmin –setSpaceQuota <N> <dirname>...<dirname>
The space quota you set acts as the ceiling on the total size of all files in a directory. You can set the space quota in bytes (b), megabytes (m), gigabytes (g), terabytes (t) and even petabytes (by specifying p—yes, this is big data!). And here’s an example that shows how to set a user’s space quota to 60GB:
$ hdfs dfsadmin -setSpaceQuota 60G /user/alapati
You can set quotas on multiple directories at a time, as shown here:
$ hdfs dfsadmin -setSpaceQuota 10g /user/alapati /test/alapati