Can't trace hinted handoff in Cassandra - cassandra

Question

I'm trying to emulate hinted handoff using Cassandra cluster in Docker.
Hinted handoff is active:
root#2f5aa8d649e2:/# nodetool statushandoff
Hinted handoff is running
The keyspace has a replication factor of 3:
cqlsh> DESCRIBE KEYSPACE imdb;
CREATE KEYSPACE imdb WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '2', 'dc2': '1'} AND durable_writes = true;
Then I shut one node down, turn on tracing and insert a new row:
cqlsh:imdb> insert into movies_by_actor (actor, movie_id, character, movie_title, salary) values ('TomHanks', uuid(), 'Character', 'Title', 1000);
Tracing session: e4a2cc20-42ce-11e7-bd49-cf534e0135c6
activity | timestamp | source | source_elapsed | client
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+------------+----------------+-----------
Execute CQL3 query | 2017-05-27 11:23:22.466000 | 172.13.0.2 | 0 | 127.0.0.1
Parsing insert into movies_by_actor (actor, movie_id, character, movie_title, salary) values ('TomHanks', uuid(), 'Character', 'Title', 1000); [Native-Transport-Requests-1] | 2017-05-27 11:23:22.467000 | 172.13.0.2 | 364 | 127.0.0.1
Preparing statement [Native-Transport-Requests-1] | 2017-05-27 11:23:22.467000 | 172.13.0.2 | 727 | 127.0.0.1
Determining replicas for mutation [Native-Transport-Requests-1] | 2017-05-27 11:23:22.468000 | 172.13.0.2 | 1354 | 127.0.0.1
Sending MUTATION message to /172.13.0.3 [MessagingService-Outgoing-/172.13.0.3-Small] | 2017-05-27 11:23:22.468000 | 172.13.0.2 | 1722 | 127.0.0.1
Sending MUTATION message to /172.13.0.6 [MessagingService-Outgoing-/172.13.0.6-Small] | 2017-05-27 11:23:22.468000 | 172.13.0.2 | 1722 | 127.0.0.1
MUTATION message received from /172.13.0.2 [MessagingService-Incoming-/172.13.0.2] | 2017-05-27 11:23:22.469000 | 172.13.0.3 | 30 | 127.0.0.1
MUTATION message received from /172.13.0.2 [MessagingService-Incoming-/172.13.0.2] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 35 | 127.0.0.1
Appending to commitlog [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.3 | 294 | 127.0.0.1
Appending to commitlog [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 292 | 127.0.0.1
Adding to movies_by_actor memtable [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 486 | 127.0.0.1
Enqueuing response to /172.13.0.2 [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 660 | 127.0.0.1
REQUEST_RESPONSE message received from /172.13.0.3 [MessagingService-Incoming-/172.13.0.3] | 2017-05-27 11:23:22.470000 | 172.13.0.2 | 3659 | 127.0.0.1
Processing response from /172.13.0.3 [RequestResponseStage-2] | 2017-05-27 11:23:22.470000 | 172.13.0.2 | 3820 | 127.0.0.1
Sending REQUEST_RESPONSE message to /172.13.0.2 [MessagingService-Outgoing-/172.13.0.2-Small] | 2017-05-27 11:23:22.472000 | 172.13.0.6 | 3533 | 127.0.0.1
REQUEST_RESPONSE message received from /172.13.0.6 [MessagingService-Incoming-/172.13.0.6] | 2017-05-27 11:23:22.473000 | 172.13.0.2 | 34 | 127.0.0.1
Processing response from /172.13.0.6 [RequestResponseStage-3] | 2017-05-27 11:23:22.473000 | 172.13.0.2 | 523 | 127.0.0.1
Request complete | 2017-05-27 11:23:22.469919 | 172.13.0.2 | 3919 | 127.0.0.1
As seen from the log, coordinator node 172.13.0.2 processes the request and contacts nodes 172.13.0.3 and 172.13.0.6. I would expect node 172.13.0.2 to save a hinted handoff, since the third node is unavailable. But when I check the system.hints table, it is empty:
cqlsh:imdb> select * from system.hints;
target_id | hint_id | message_version | mutation
-----------+---------+-----------------+----------
(0 rows)
The consistency level is set to default ONE. Could someone explain where the hinted handoff is stored, if at all?

Answer 1

Latest version of cassandra does not store hints in system.hints table.
Hints are stored in flat files from cassandra 3.0. If using cassandra version greater than 3.0 you must look into hints directory configured in cassandra.yaml
# Directory where Cassandra should store hints.
# If not set, the default directory is $CASSANDRA_HOME/data/hints.
hints_directory: "C:/Program Files/DataStax-DDC/data/hints"
# How often hints should be flushed from the internal buffers to disk.
# Will *not* trigger fsync.
hints_flush_period_in_ms: 10000
Check above 2 values in your cassandra.yaml and look for hints in it.
Hinted Handoff in cassandra 3.0