Why cassandra fails with OutOfMemoryError during sstables compaction - cassandra

Question

Hi may be this is a stupid question, but I did not find the answer via google.
So what I have:
java 1.7
cassandra 1.2.8 running in single node with -Xmx1G and -Xms1G without any changes to yaml file
I've created next test column family:
CREATE COLUMN FAMILY TEST_HUGE_SF
WITH comparator = UTF8Type
AND key_validation_class=UTF8Type;
Then I try to insert rows in this column family.
I use astyanax lib to access cassandra:
final long START = 1;
final long MAX_ROWS_COUNT = 1000000000; // 1 Billion
Keyspace keyspace = AstyanaxProvider.getAstyanaxContext().getClient();
ColumnFamily<String, String> cf = new ColumnFamily<>(
"TEST_HUGE_SF",
StringSerializer.get(),
StringSerializer.get());
MutationBatch mb = keyspace.prepareMutationBatch()
.withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
for (long i = START; i<MAX_ROWS_COUNT; i++) {
long t = i % 1000;
if (t == 0) {
System.out.println("pushed: " + i);
mb.execute();
Thread.sleep(1);
mb = keyspace.prepareMutationBatch()
.withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
}
ColumnListMutation<String> clm = mb.withRow(cf, String.format("row_%012d", i));
clm.putColumn("col1", i);
clm.putColumn("col2", t);
}
mb.execute();
So as you can see from code, I try to insert 1 Billion rows, each one contains two columns, each column contains simple long value.
After inserting ~ 122 million rows, - cassandra crashed with OutOfMemoryError.
In logs there is next:
INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,334 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941,]. 865 252 169 bytes to 901 723 715 (~104% of original) in 922 963ms = 0,931728MB/s. 26 753 257 total rows, 26 753 257 unique. Row merge counts were {1:26753257, 2:0, 3:0, 4:0, }
INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,337 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1571] 2014-08-08 08:31:46,167 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1571,1,main]
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
INFO [CompactionExecutor:1570] 2014-08-08 08:31:46,994 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-3213,]. 34 773 524 bytes to 35 375 883 (~101% of original) in 44 162ms = 0,763939MB/s. 1 151 482 total rows, 1 151 482 unique. Row merge counts were {1:1151482, 2:0, 3:0, 4:0, }
INFO [CompactionExecutor:1570] 2014-08-08 08:31:47,105 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1570] 2014-08-08 08:31:47,110 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1570,1,main]
java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
As I see cassandra crashes during sstables compaction.
Does this mean that to handle more rows cassandra needs more heap space?
I expected that lack of heap space will only affect performance. Can someone describe, why my expectations are wrong?

Answer 1

Someone else noted this - 1GB heap is very small. With Cassandra 2.0, you could look into this tuning guide for further information:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html
Another consideration is how garbage collecting is being handled. In the cassandra log directory, there should be also GC logs indicating how often and how long the collections were. You can monitor them live using jvisualvm, if you want.