Apache Cassandra unrequired hint files present in hints folder - cassandra

Question

On Apache Cassandra 3.11.1, during cassandra decommission I noticed that the node tries to send hints. I checked the hints folder and found hints that are very old or are present for already removed nodes. I decided to delete them so they are not part of the decommission process, but after decommissioning a node, the process fails and the node gets stuck in UL state.
Stacktrace:
ERROR [HintsDispatcher:3] 2022-12-12 17:58:39,364 CassandraDaemon.java:228 - Exception in thread Thread[HintsDispatcher:3,1,RMI Runtime]
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/xxxx-1670867104313-1.hints
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.ChecksummedDataInput.open(ChecksummedDataInput.java:77) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsReader.open(HintsReader.java:78) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatcher.create(HintsDispatcher.java:73) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:273) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:260) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:238) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:217) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_131]
at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_131]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/0891d19f-7ba9-4fc6-973c-79f98253cf4e-1670867104313-1.hints
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.8.0_131]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 25 common frames omitted
Final Error:
ERROR [RMI TCP Connection(350)-127.0.0.1] 2022-12-12 18:05:33,190 StorageService.java:3954 - Error while decommissioning node
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/xxxx-1670867104313-1.hints
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.ChecksummedDataInput.open(ChecksummedDataInput.java:77) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsReader.open(HintsReader.java:78) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatcher.create(HintsDispatcher.java:73) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:273) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:260) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:238) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:217) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_131]
at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_131]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/0891d19f-7ba9-4fc6-973c-79f98253cf4e-1670867104313-1.hints
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.8.0_131]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 25 common frames omitted
How can I avoid such a scenario?
I tried nodetool truncatehints, but it does nothing on the node.

Answer 1

At this point, I would just run a nodetool removenode on it. The removenode process will:
get rid of a stuck DL/UL node.
move data to its new token ranges from the other replicas.
What happens in case all data was already migrated from the leaving node? will the removenode command then stream again and create duplicate data?
Yes, it streams it again, but it doesn’t create “duplicate data;” not in the same way that in-place writes do.
So a cleanup will clear out this re-streamed data?
No. The streaming happens at the SSTable file level. So essentially a whole new file is being streamed during the decomm/remove process. If there was already duplicate, in-place-updated data present, it still will be there on its new node. If there was only a single write at the partition level, then that's all that will be there. If it already exists, this process will simply write one file over another, and Cassandra will never know it was even there.
When you add or remove a node, a token range calculation is triggered. This means that some nodes end up containing data that they are no longer responsible for (because another node now is). Running a nodetool cleanup removes data which:
has already been streamed somewhere else.
remains on a node which
will never use it.