I would like to tune Cassandra for heavy read scenario with skinny rows (5-50 columns). The idea is to use row cache, and enable key cache just in case - when data is to large for row cache.
I have dual Intel Xeon server with 24GB RAM (3 in ring, two data centers - gives 6 machines in total)
Those are changes that I've made to default configuration:
cassandra-env.sh
#JVM_OPTS="$JVM_OPTS -ea"
MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="500M"
cassandra.yaml
# do not persist caches to disk
key_cache_save_period: 0
row_cache_save_period: 0
key_cache_size_in_mb: 512
row_cache_size_in_mb: 14336
row_cache_provider: SerializingCacheProvider
The idea it to dedicate 6GB to Cassandra JVM, 0.5GB to key cache (out of 6GB heap), and 14GB to row cache as off-heap.
OS has still 4GB which should be enough, since there is running only one JVM process and it should have overhead of max 2GB.
Is this setup optimal? Any hints?
Thanks,
Maciej
I'm using 1.1.6 version.
SerializingCacheProvider will save cache data at Native Heap area.
That area is not for GC inspect. so It will not be occurred GC.
Your row_cache_size_in_mb setting is for SerializingCache's reference object.
That reference is saved using FreeableMemory(It is in 1.1.x. but after 1.2, it changed).
In other words, Your real cache value is not calculated when calculating row_cache_size_in_mb.
At the result If you want to calculate row_cache_size_in_mb, try to set from minimal size.
In my case, when I set 500mb, each node was using 2G old gen.(in according to deal which data set)
Run the heapspace_calculator and use the suggested value as an initial heap configuration. Monitor your heap usage with "nodetool info".
Try to use short column names and merge columns when possible.
This setup works just fine - I've tested it.
Related
We are using cassandra in order to collect the data from thingsboard. The memory it started with was 4GB (after executing the systemctl status for cassandra) and after 15 hours it has reached up to 9.3GB.
I want to know why is there this much increase in memory and is there any way to control it or to restrict it to use fixed amount of memory without the data being lost.
Check this for setting max heap size used . But tune cassandra gc properly when you change this.
I have a Cassandra installation which contains a table with no more then 110k records.
I'm getting quite a lot of troubles querying the data using PDI 5.3 (the latest version). I am constantly getting out of memory on Cassandra side.
Granted that the server I have Cassandra installed is not the greatest, 4Gb RAM and only 2 cores, I would still expect to perform this simple task without issues.
In cassandra /conf/cassandra-env.sh, I've configured:
MAX_HEAP_SIZE="4G"
HEAP_NEWSIZE="200M"
and now the maximum number of rows I can query is 80k.
The documentation suggests to set MAX_HEAP_SIZE to 1/4th of the machines RAM. But for me that meant 1G and only about 20k rows to query.
I am able to tell how many rows I can query by limiting the select, with the limit keyword, inside the Cassandra input step in PDI.
Are there any other parameters I can tweak to get better performance? This is a development server, on production I'll be expecting queries with 1mil+ rows.
Server on which Cassandra is installed: Red Hat Enterprise Linux Server release 6.6 (Santiago)
Cassandra version: apache-cassandra-2.1.2
Edit: versions updated.
Sacrifice IO for Memory (since memory is killing you):
lower key / row caches if they are enabled (key cache is on by default)
if you carry out lots of deletes you can lower gc_grace_seconds to remove tombstones quicker (assuming you many range scans which you do if you fetch 80k rows, this can help)
Some other ideas:
Paginate (Select 0-10k of 80k, then 10-20k etc.
Check sizes of memtables, if they are too large lower them.
Use tracing to verify what you are retrieving (tombstones can cause lots of overhead)
This thread suggests lowering the commit_log size, but the commit log was heavily revamped and moved offheap in 2.1 and shouldn't be such an issue anymore.
We are trying to create a prototype to the Cassandra Datastax community edition and java driver.
I've tried to measure the latency of simple retrieve and update using the Sample from Cassandra Java Driver (simplex keyspace).
I have two data centers with one Rack per data center. Each Rack contains 3 nodes.
I have 6 nodes (VMs) in total.
I've configured key_cache_size_in_mb to 10 in order tuning the retrieve/update operations.
In summary we are trying to tune the sample operations to get around 5 ms latency for read/update operation.
Following the latency that we managed to achieve:
19 milliseconds elapsed to retrieve playlist table.
title album artist
Memo From Turner Performance Mick Jager
Updating simplex.playlist
14 milliseconds elapsed to update songs table.
14 milliseconds elapsed to retrieve songs table.
title album artist tags
La Petite Tonkinoise' Bye Bye Blackbird' JosŽphine Baker
What are the tunings that should be done in order improve the performance and achieving better latency than above?
Your direction/insight would be highly appreciated.
Thanks in advance,
Erwin
Some performance optimization tips/best practices:
Larger the number of nodes, better the distribution and C* performs better
64-bit JVMs perform better than 32-bit (Use Oracle JVM 1.6 at least u22)
physical environments, minimum is 8GB, but anything between 16-32 GB, 8-core processors
at least two disks, one for the commit log and the other for the data directories
Commit Log + data directory on same volumes – avoid this. The biggest performance gain for write is to put commit log in a separate disk drive. Commit log is 100% sequential, while data reads are random from data directories. I/O contention between commit log & SSTables may deteriorate commit log writes and SSTable reads. But this does not apply to SSDs or EC2.
JVM parameters tuning (on a 8GB RAM system)
Heap tuning
-Xms${MAX_HEAP_SIZE}
-Xmx${MAX_HEAP_SIZE} – default to 40-50% of available physical memory – 4 GB
-Xmn${HEAP_NEWSIZE} - default to 25% of java heap – 1GB
GC tuning
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseParallelGC
-XX:SurvivorRatio=4
-XX:MaxTenuringThreshold=0
Synch the clocks on all nodes – As C* adds timestamp t each coumn value, it is must to synch clocks across the ring using NTP daemon or script. NTP known to drift the clocks across datacenters.
Use Key cache sparingly, as it has highest possible performance gains with least memory footprint, as it stores only the key and data location. Saves one file I/O seek.
update column family my_column_family with keys_cached=50000;
Use RF=3, it’s a best practice, Write/Read consistency level = QUORUM is a best practice
on Linux, you can locate cassandra.sh, which is used to start the Cassandra process. This is where we add the GC params as well as the JVM memory settings. (backup the file first) i assume, you have 4GB allocated to cassandra process. Assuming you have a 8GB system memory, allocate -Xmx4096m to Cassandra process.
https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh?source=cc
you can tuning options coded in section "# GC tuning options"
key_cache_size_in_mb - this setting can be found in the cassandra.yaml file and will applicable to all column families in your keyspace or else set at CF level. You need to know approx size of your rows and work out the calculations. e.g. for 1 million rows to be cached with avg row size of 100 bytes with 25 columns each of 4 bytes, you need to set it as 100 mb (1 mn * 100 bytes)
We are currently doing some stress tests with ab tool. The single inserts are doing fine in cassandra. However, when it comes to batch inserts, I'm currently dealing with java out of memory error: Java Heap Space.
I have a virtual box machine with Ubuntu server 13.04 installed in it with 2G of memory
I don't know much about internal configuration in cassandra.
I'm just making a batch insert with size 100(100 insert in a BATCH).
After the I see this error, I have no longer cqlsh access, no nodetool access for almost 1 hour.
How can I fix this error in heavy loads ?
NOTE : It doesn't happen on single inserts with a HTTP POST requests.
NOTE : In my column family, I have a key with TimeUUIDType and the column values are int s and varchar s
UPDATE : Test results show that I didn't have anything wrong before 6000 requests. However, when it comes to 7000, the php code throws the following;
Error connecting to 127.0.0.1: Thrift\Exception\TTransportException: TSocket: timed out reading 4 bytes from 127.0.0.1:9160
Morever, cassandra logs the following in heavy loads;
WARN [ScheduledTasks:1] 2013-06-28 03:43:07,931 GCInspector.java (line 142)
Heap is 0.9231763795560355 full. You may need to reduce memtable and/or cache sizes.
Cassandra will now flush up to the two largest memtables to free up memory. Adjust
flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to
do this automatically
The batch doesn't sound like a large enough dataset to cause the memory problem, so this sounds like a problem with the JVM on the virtual machine. How much memory have you allocated to it?
You can check by starting JConsole (just type jconsole in the terminal / prompt) and viewing the 'Memory' tab, specifically the value under Max:
You can also get some solid details about what caused the crash thanks to the XX:+HeapDumpOnOutOfMemoryError parameter included in C*'s startup script, its basically a log file storing the stacktrace that caused the memory problem.
Typically the heap size is calculated automatically by the calculate_heap_sizes() function in cassandra-env.sh. You can however override the number that function generated by setting MAX_HEAP_SIZE to a different value. The same variable is used on lines 174 & 175 in cassandra-env.sh JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}" for setting the min and max heap size.
I have a cassandra setup with 6 Node Ring single DC with RF:6 and Read:CL:1. Now at times if a particular node gets lot of requests which in turn would be passed on to all the nodes (cause of RF) it gets on to Compaction with CMS and then finally with ParNew where the whole ring gets in a hanged situation which in turn making it pretty unusable. I found that we could only solve it with increasing Heap size or tweaking in the cassandra code for only reading from the local node(as RF:6 would guarantee every node having the same data although repair etc have to be dealt separately).
How to calculate Heap Size for Cassandra node(I have two Keyspace's with total 14 CF's apart from System CF's).As per cassandra wiki this should be the heap size atleast : memtable_throughput_in_mb * 3 * number of hot CFs + 1G + internal caches where memtable_throughput_in_mb=128mb for my setup. The max row size for a particular CF should matter here. I am not using any row or key cache. Can someone suggest me the same.