Column counts are larger than 1996099046, unable to calculate percentiles - cassandra

Question

while I am running TableHistograms getting below message:
NodeTool TableHistograms keyspace TableName
Column counts are larger than 1996099046, unable to calculate percentiles
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 0.00 0.00 0.00 268650950 NaN
75% 0.00 0.00 0.00 3449259151 NaN
95% 0.00 0.00 0.00 25628284214 NaN
98% 0.00 0.00 0.00 44285675122 NaN
99% 0.00 0.00 0.00 44285675122 NaN
Min 0.00 0.00 0.00 105779 0
Max 0.00 0.00 0.00 442856751229223372036854776000
Cassandra version:
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
Replication factor 3
4 node cluster
Getting the above message in one node only
Tried repairing the table but failed with streaming error :
40328:ERROR [StreamReceiveTask:53] 2019-06-10 13:54:33,684 StreamSession.java:593 - [Stream #c9214180-8b82-11e9-90ce-399bac480141] Streaming error occurred on session with peer <IP ADDRESS>
40329-java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
40330- at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:51) ~[apache-cassandra-3.11.2.jar:3.11.2]
40331- at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:373) ~[apache-cassandra-3.11.2.jar:3.11.2]
40332- at org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:383) ~[apache-cassandra-3.11.2.jar:3.11.2]
40333- at org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:270) ~[apache-cassandra-3.11.2.jar:3.11.2]
40334- at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:216) ~[apache-cassandra-3.11.2.jar:3.11.2]
40335- at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
40336- at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_144]
40337- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
40338- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
--
0354:ERROR [Reference-Reaper:1] 2019-06-10 13:54:33,907 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State#7bd8303d) to class org.apache.cassandra.io.util.ChannelProxy$Cleanup#1084465868:PATH/talename-5b621cd0c53311e7a612ffada4e45177/mc-26405-big-Index.db was not released before the reference was garbage collected
Table description includes :
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
Any idea why it is happening? Any help or suggestion is welcome.

Answer 1

You cannot have 2 billion cells in a partition. Also having a secondary index on a table with a 44gb partition is going to have issues for multiple reasons. There really isn't much you can do to fix this short of dropping your index and building a new data model to migrate into. You could build a custom version of Cassandra to ignore that exception but something else will come up very soon as you are at the extreme limits of whats even theoretically possible. You are already past a point that I am surprised is running.
If the streaming error is from repairs you can ignore it while you fix your data model. If it's from bootstrapping I think you will need a custom version of Cassandra to stay running in meantime (or can just ignore the down node you are replacing). Keep in mind node failures are a serious threat to you now as bootstrapping likely will not work. When you put so much in a single partition it cannot be scaled out so there are limited options.