Cassandra prototype, latency issue - cassandra

Question

We are trying to create a prototype to the Cassandra Datastax community edition and java driver.
I've tried to measure the latency of simple retrieve and update using the Sample from Cassandra Java Driver (simplex keyspace).
I have two data centers with one Rack per data center. Each Rack contains 3 nodes.
I have 6 nodes (VMs) in total.
I've configured key_cache_size_in_mb to 10 in order tuning the retrieve/update operations.
In summary we are trying to tune the sample operations to get around 5 ms latency for read/update operation.
Following the latency that we managed to achieve:
19 milliseconds elapsed to retrieve playlist table.
title album artist
Memo From Turner Performance Mick Jager
Updating simplex.playlist
14 milliseconds elapsed to update songs table.
14 milliseconds elapsed to retrieve songs table.
title album artist tags
La Petite Tonkinoise' Bye Bye Blackbird' JosŽphine Baker
What are the tunings that should be done in order improve the performance and achieving better latency than above?
Your direction/insight would be highly appreciated.
Thanks in advance,
Erwin

Answer 1

Some performance optimization tips/best practices:
Larger the number of nodes, better the distribution and C* performs better
64-bit JVMs perform better than 32-bit (Use Oracle JVM 1.6 at least u22)
physical environments, minimum is 8GB, but anything between 16-32 GB, 8-core processors
at least two disks, one for the commit log and the other for the data directories
Commit Log + data directory on same volumes – avoid this. The biggest performance gain for write is to put commit log in a separate disk drive. Commit log is 100% sequential, while data reads are random from data directories. I/O contention between commit log & SSTables may deteriorate commit log writes and SSTable reads. But this does not apply to SSDs or EC2.
JVM parameters tuning (on a 8GB RAM system)
Heap tuning
-Xms${MAX_HEAP_SIZE}
-Xmx${MAX_HEAP_SIZE} – default to 40-50% of available physical memory – 4 GB
-Xmn${HEAP_NEWSIZE} - default to 25% of java heap – 1GB
GC tuning
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseParallelGC
-XX:SurvivorRatio=4
-XX:MaxTenuringThreshold=0
Synch the clocks on all nodes – As C* adds timestamp t each coumn value, it is must to synch clocks across the ring using NTP daemon or script. NTP known to drift the clocks across datacenters.
Use Key cache sparingly, as it has highest possible performance gains with least memory footprint, as it stores only the key and data location. Saves one file I/O seek.
update column family my_column_family with keys_cached=50000;
Use RF=3, it’s a best practice, Write/Read consistency level = QUORUM is a best practice
on Linux, you can locate cassandra.sh, which is used to start the Cassandra process. This is where we add the GC params as well as the JVM memory settings. (backup the file first) i assume, you have 4GB allocated to cassandra process. Assuming you have a 8GB system memory, allocate -Xmx4096m to Cassandra process.
https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh?source=cc
you can tuning options coded in section "# GC tuning options"
key_cache_size_in_mb - this setting can be found in the cassandra.yaml file and will applicable to all column families in your keyspace or else set at CF level. You need to know approx size of your rows and work out the calculations. e.g. for 1 million rows to be cached with avg row size of 100 bytes with 25 columns each of 4 bytes, you need to set it as 100 mb (1 mn * 100 bytes)