How To Figure Out Cassandra Extra Off-Heap Memory Usage - cassandra

Question

I have nodes which have 32GB Ram. I set 20 GB heap size. I am aware of cassandra uses off-heap for memtable, cache etc. Even though usage of memory is very low by memtable, cache etc, somehow cassandra uses 27GB memory. How can I figure out, how cassandra uses that extra 7GB memory?

Answer 1

You can get metrics with JConsole
Or you can use Jolokia agent. For that download and set up jolokia agent to your Cassandra node. E.g.on Linux
mkdir /opt/jolokia
cd /opt/jolokia
wget https://github.com/rhuss/jolokia/releases/download/v1.4.0/jolokia-1.4.0-bin.tar.gz
tar -xf jolokia-1.4.0-bin.tar.gz
Add the agent path as a JVM option to the end of your cassandra-env.sh file and restart cassandra
echo 'JVM_OPTS="$JVM_OPTS -javaagent:/opt/jolokia/jolokia-1.4.0/agents/jolokia-jvm.jar"' >> /etc/conf/cassandra/cassandra-env.sh
Restart Cassandra.
Then you can query metrics like:
Total on heap memory:
wget http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=Memory/HeapMemoryUsage
Total off heap memory:
wget http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=Memory/NonHeapMemoryUsage
Off heap memory used by memtables:
wget http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=MemtableOffHeapSize
And also for Bloomfilter, IndexSummary and Compression metadata:
wget http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=BloomFilterOffHeapMemoryUsed
wget http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=IndexSummaryOffHeapMemoryUsed
wget http://localhost:8778/jolokia/read/org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=CompressionMetadataOffHeapMemoryUsed
UPDATE:
Example response from Jolokia endpoint:
{
"request":{
"mbean":"org.apache.cassandra.metrics:keyspace=*,name=CompressionMetadataOffHeapMemoryUsed,scope=*,type=Table",
"type":"read"
},
"value":{
"org.apache.cassandra.metrics:keyspace=my_keyspace,name=CompressionMetadataOffHeapMemoryUsed,scope=my_table_name,type=Table":{
"Value":832
},
"org.apache.cassandra.metrics:keyspace=system,name=CompressionMetadataOffHeapMemoryUsed,scope=compaction_history,type=Table":{
"Value":64
},
"org.apache.cassandra.metrics:keyspace=my_keyspace,name=CompressionMetadataOffHeapMemoryUsed,scope=my_table_name2,type=Table":{
"Value":8184
},
...
}
}