Cassandra Map collection type read latency - cassandra

Question

I have a three-node Cassandra cluster with replication factor 3 and consistency level LOCAL_QUORUM. My table consists of two columns of MAP<BLOB, BLOB> type. Εach map contains up to 100 entries. I'm writing (append) into both maps and I'm reading from one (1R/1W per transaction).
After a few hours of writing and reading across 500k partitions, the table statistics were as follows:
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 10.00 35.43 2346.80 1597 60
75% 10.00 51.01 4055.27 2299 72
95% 12.00 105.78 17436.92 6866 215
98% 12.00 182.79 36157.19 6866 215
99% 12.00 454.83 52066.35 8239 215
Min 5.00 3.31 379.02 104 3
Max 14.00 186563.16 322381.14 9887 310
So far, so good. The next step was to create 30 million new partitions.
After about 15 hours of writing (in random partitions) I noticed a massive TPS drop (about 2k):
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 2.00 51.01 20924.30 1916 50
75% 3.00 73.46 43388.63 1916 60
95% 4.00 126.93 89970.66 1916 60
98% 4.00 219.34 107964.79 2299 72
99% 4.00 379.02 129557.75 6866 179
Min 0.00 3.97 51.01 104 3
Max 8.00 186563.16 322381.14 9887 310
Performing the first test again across 500k partitions, read latency remained high:
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 5.00 51.01 30130.99 1916 60
75% 6.00 73.46 62479.63 1916 60
95% 7.00 152.32 129557.75 1916 60
98% 8.00 263.21 155469.30 3311 103
99% 8.00 545.79 186563.16 6866 179
Min 3.00 3.97 454.83 104 3
Max 10.00 107964.79 557074.61 9887 310
Read latency increases, even more, when workload involves writing a counter column(into another table):
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 10.00 42.51 62479.63 1916 50
75% 10.00 61.21 107964.79 1916 60
95% 12.00 105.78 186563.16 1916 60
98% 12.00 182.79 223875.79 3311 103
99% 12.00 379.02 268650.95 6866 179
Min 6.00 4.77 545.79 104 3
Max 14.00 129557.75 557074.61 9887 310
What would be the probable causes? Could the map column type be the root cause? Any suggestions(configuration or schema changes)?
I am using prepared statements:
Fetch row by partition ID
SELECT id,attr,uids FROM user_profile WHERE id=:id
Update map entries
UPDATE user_profile SET attr=attr+:attr, attr=attr-:attrstoremove, uids=uids+:newuserids, md=md+:metadata, md=md-:attrstoremove, up=:up WHERE id=:id
Increase counter
UPDATE user_profile_counter SET cnt=cnt+:cnt WHERE cnt_id=:cnt_id AND id=:id;
This is my schema:
CREATE TABLE IF NOT EXISTS PROFILING.USER_PROFILE
(
-- PARTITION KEY
ID TEXT, -- PROFILE ID
-- DATA
ATTR MAP<BLOB, BLOB>, --USER ATTRIBUTES
MD MAP<BLOB, BLOB>, --METADATA PER ATTRIBUTE
UIDS SET<TEXT>,
UP TIMESTAMP, --LAST_UPDATE
PRIMARY KEY (ID)
) WITH caching = {
'keys' : 'ALL',
'rows_per_partition' : '1'
};
CREATE TABLE IF NOT EXISTS PROFILES.USER_PROFILE_COUNTER
(
-- PARTITION KEY
ID TEXT, -- PROFILE ID
-- CLUSTERING KEY
CNT_ID BLOB, -- COUNTER ID
-- DATA
CNT COUNTER, -- COUNTER VALUE
PRIMARY KEY (ID, CNT_ID)
) WITH caching = {
'keys' : 'ALL',
'rows_per_partition' : '10' };
The data are encrypted. Here is a row sample(consisting of three map entries):
YkceUdD6qEvOLw3Wgd8zWA |{0x95f56f594522:
0xacb7f42c7f0ac8187f17a8f2c04e5065, 0xa365a3dc007d:
0x24252727706b5065f9e1f65efec7ced8, 0xf0d55b110f87:
0x5a5ef3b0a041af8c7acf4040333afc96} |
{0x95f56f594522:
0x000d31363333333334363636323639, 0xa365a3dc007d:
0x000d31363333333431323938363735, 0xf0d55b110f87:
0x000d31363333333431323938363735} |
{'46TyNYCKTplibRyAfFsNRPQbvfQINNIIY4WmItuPayfvjDjEp49bnXSXLmD9hAm9'} |
2021-10-04 09:54:58.675000+0000
Cluster info (per node)
Total memory 18GB(heap 5GB)
6 CPU cores
Versions
Cassandra 3.11
DataStax Java driver 4.11.3
JDK 16
Edit
After a few more tests, a high number of disk IOPS was observed, most of them read operations. More specifically, 20k IOPS and 1500MBps bandwidth were observed. I tried to reduce the number of SSTable touched by using the LeveledCompactionStrategy and lowering the chunk_length_in_kb parameter to 4KB but there was not much difference.
Note that SAN storage is used. I know it's an anti-pattern but unfortunately, we have no other option. Any ideas?

Answer 1

So to at least provide some theories before the weekend, let me just touch on a couple of things.
JDK 16
So Cassandra 3.11 won't run on anything higher than JDK 8, so I'm guessing this is from the client side?
Total memory 18GB(heap 5GB)
I suspect that the nodes would perform better with a larger heap. On an 18GB instance, I would bump that up to somewhere around 8GB-10GB. 9GB will give you a happy 50% of RAM, so that might work well.
If the heap size ends up exceeding 8GB in size, I would go with G1GC (Cassandra 3 still defaults to CMS GC). If you feel more comfortable staying on CMS, then I'd make the heap new size (Xmn) somewhere around 40% to 50% of the max (Xmx) and initial (Xms) sizes.
three-node Cassandra cluster with replication factor 3
It's also quite possible that you need more than 3 nodes to handle this load. The Cassandra nodes might be filling up with write backpressure, and dying a slow death. After bumping the heap size, I'd try this next.
Have you had a look at the disk I/O metrics (IOPS in particular)? The partition sizes are good, and so is the query, so maybe the disk is slow?
Not sure if you're running on shared infra or not (ex: OpenStack), but one thing I've run into in the past, was noisy neighbors. You have instance that you're sharing compute or storage with, which is taking up all the bandwidth. I'd check with your infrastructure team, and see if that's an issue.
Also, while working on that shared infra environment, I once ran into an issue where we found out that the storage scheduler had put all drives for one cluster on the same physical volume. Hence, we didn't have any real hardware availability, plus the cluster's high write throughput was basically choking itself out.
Have a look at some of that, and see if you find anything.
Edit 20211114
Yes provisioning Cassandra’s drives on a SAN is going to be difficult to glean performance off of. I can think of two things to try:
Direct your commitlog to a different drive (hopefully a local, non-SAN) than the data.
Provision each node’s data drive to a different SAN.