I'm getting OutOfMemoryError when run compaction on some big sstables in production, table size is around 800 GB, compaction on small sstables is working properly though.
$ noodtool compact keyspace1 users
error: Direct buffer memory
-- StackTrace --
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.cassandra.io.compress.BufferType$2.allocate(BufferType.java:35)
Java heap memory(Xms and Xmx) have been set to 8 GB, wondering if I should increase Java heap memory to 12 or 16 GB?
It's not the Heap size, but it's instead so-called "direct memory" - you need to check what amount you have (it's could be specified by something like this -XX:MaxDirectMemorySize=512m, or it will take the same max size as heap). You can increase it indirectly by increasing the heap size, or you can control it explicitly via -XX flag. Here is the good article about non-heap memory in Java.
Related
As per cassandra-env.sh the default heap memory allocation for a 440G Total RAM should be 32765M (Maximum CAP before JVM Swithches to 64 bit reference).
So, why is it showing 32210157568 bytes(30718M) when I query "java -XX:+PrintCommandLineFlags -version" or "java -XX:+PrintFlagsFinal -version | grep -iE 'MaxHeapSize'"
Why is there difference, of around 2G.
FYI: jvm.options files was default & using DSE 5.1.3.
java -XX:+PrintFlagsFinal has nothing to do with Cassandra, and I don't know why you mention cassandra-env.sh. Anyway, let me answer the main part of the question.
In JDK 8, when -Xmx is not specified, the maximum heap size is estimated as
MaxHeapSize = min(1/4 RAM, max_heap_for_compressed_oops)
In your case the server has plently of RAM, so the default heap size is limited by the maximum possible size supported by zero-based compressed oops, that is, 32 GB.
The heap obviously cannot start at zero address (null page is reserved by the OS), and the default heap alignment is 2 MB, so we must subtract at least 2 MB.
Then, JDK prefers to allocate the heap at HeapBaseMinAddress, which is equal to 2 GB on Linux. This provides some space to grow the native heap of the process. For this reason JVM reduces the default maximum heap size by HeapBaseMinAddress.
That's why the final computed heap size is equal to
32 GB - 2 MB - 2 GB = 32210157568
If you give up the requirement for the zero-based compressed oops, you may set -XX:HeapBaseMinAddress=0. In this case the computed heap size would be
32 GB - 2MB = 32766 MB
I am running my cassandra cluster having memory 32 GB on each node,
And row cache capacity (row_cache_size_in_mb) 5GB,
Just want to know, does 5gb memory ram is reserved for row caching from my heap??
It will let it grow to that size over time. Can use nodetool info to see the current size and limit and nodetool setcachecapacity to change it at runtime. Note that its kinda an estimate though and heap can grow a bit larger. I would be sure to test that the row_cache is actually improving things though since in a lot of cases having no row cache can be faster.
A lot of the discussions I found on the internet on resource allocation was about the max memory config for --executor-memory, taking into account a few memory overheads.
But I would imagine that for simple job like reading in a 100MB file and then count # of rows, with a cluster of a total 500GB memory available across nodes, I shouldn't ask for # of executors and memory allocation that, with all memory overheads accounted for, could take all 500GB memory, right? Even 1 executor of 3GB or 5GB memory seems to be an overkill. How should I think about the right memory size for a job?
Thank you!
What is the maximum limit of cache in spark. How much data can it hold at once?
See this. It is 0.6 x (JVM heap space - 300MB) by default.
I may be wrong but to my understanding here is calculation
What is executer memory. Lets say it is 1 GB.
Then heap size is 0.6 of it which 600 MB
Then 50% of heap size is cache. i.,e 300 MB.
http://spark.apache.org/docs/latest/tuning.html#memory-management-overview in this, they must have assumed executor memory is 500 MB. In fact, for local executor memory default size is 500 MB. If it executer memory is 500 MB then only 150 MB is allocated to cache
Its Actually totally depends on executor memory. Spark will take as much as large part of the RDD in memory and the rest will be fetched and recomputed on the fly each time they're needed. It is totally configurable and you can check it here
We have a 3-node cassandra cluster on AWS. These nodes are running cassandra 1.2.2 and have 8GB memory. We didn't change any of the default heap or GC settings. So each node is allocating 1.8GB of heap space. The rows are wide; each row stores around 260,000 columns. We are reading the data using Astyanax. If our application tries to read 80,000 columns each from 10 or more rows at the same time, some of the nodes run out of heap space and terminate with OOM error. Here is the error message:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.duplicate(HeapByteBuffer.java:107)
at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:50)
at org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:60)
at org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:126)
at org.apache.cassandra.db.filter.ColumnCounter$GroupByPrefix.count(ColumnCounter.java:96)
at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:164)
at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:294)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1363)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1220)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1132)
at org.apache.cassandra.db.Table.getRow(Table.java:355)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
ERROR 02:14:05,351 Exception in thread Thread[Thrift:6,5,main] java.lang.OutOfMemoryError: Java heap space
at java.lang.Long.toString(Long.java:269)
at java.lang.Long.toString(Long.java:764)
at org.apache.cassandra.dht.Murmur3Partitioner$1.toString(Murmur3Partitioner.java:171)
at org.apache.cassandra.service.StorageService.describeRing(StorageService.java:1068)
at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:1192)
at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.getResult(Cassandra.java:3766)
at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.getResult(Cassandra.java:3754)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722) ERROR 02:14:05,350 Exception in thread Thread[ACCEPT-/10.0.0.170,5,main] java.lang.RuntimeException: java.nio.channels.ClosedChannelException
at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:893) Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:211)
at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:99)
at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:882)
The data in each column is less than 50 bytes. After adding all the column overheads (column name + metadata), it should not be more than 100 bytes. So reading 80,000 columns from 10 rows each means that we are reading 80,000 * 10 * 100 = 80 MB of data. It is large, but not large enough to fill up the 1.8 GB heap. So I wonder why the heap is getting full. If the data request is to big to fill in a reasonable amount of time, I would expect Cassandra to return a TimeOutException instead of terminating.
One easy solution is to increase the heap size, but that will only mask the problem. Reading 80MB of data should not make a 1.8 GB heap full.
Is there some other Cassandra setting that I can tweak to prevent the OOM exception?
No, there is no write operation in progress when I read the data. I am
sure that increasing the heap space may help. but I am trying to
understand why reading 80MB of data is making a 1.8GB heap full.
Cassandra uses Heap and OfHeap chaching.
First loading of 80MB userdata may result in 200-400 MB of Java Heap usage. (which vm? 64 bit?)
Secondly this memory is added to memory allready used for caches. It seemes that cassandra does not frees that caches to serve your private query. Could make sence for overal throughput.
Did you meanwhile solved your problem by increasing MaxHeap?