During cassandra decommission I noticed that the node tries to send hints which takes extremely long time and never completes. I checked the hints folder and found hints that are more than 9+ months old. I am not sure why those old hints are still present in the folder so I decided to delete them. After I deleted them I noticed the following entry in the system.log
INFO [HintsDispatcher:1070] 2021-07-08 11:32:01,056 HintsDispatchExecutor.java:141 - Transferring all hints to /10.199.190.233: 7935f1b5-4725-4dc2-ad6d-b883d53d907d
ERROR [HintsDispatcher:1070] 2021-07-08 11:32:01,061 CassandraDaemon.java:207 - Exception in thread Thread[HintsDispatcher:1070,1,RMI Runtime]
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /data/cassandra/data/hints/ce6bb0e3-849f-487d-9274-38b8536b89cf-1603947885707-1.hints
Where does Cassandra keep metadata for the hints as system.hints folder didn't have any entry?
Cassandra Version is 3.0.12
There is a catalog of hints held in memory on each Cassandra node for tracking.
If you manually delete the contents of the hints directory on a node, the entries in the hints catalog become stale and you run into the NoSuchFileException you posted.
The correct way of deleting hints is with the nodetool truncatehints command. Cheers!
Related
We have cassandra 3.0.10 installed on centos. The developers made some coding mistakes on preparing statements. The result is that the prepared statement cache is overrunning and we always get evicted error message. The error is shown below:
INFO [ScheduledTasks:1] 2017-12-07 10:38:28,216 QueryProcessor.java:134 - 7 prepared statements discarded in the last minute because cache limit reached (8178944 bytes)
We have corrected the prepared statements and would like to flush the prepared statement cache to start from scratch. We have stopped and restarted the cassandra instance but the prepared statement count was not reset.
Cassandra 3.0.10 is installed on centos and we are using svcadm disable/enable cassandra to stop/start cassandra.
I noticed that in later version of cassandra, e.g. 3.11.1, there is a prepared_statements table under the system keyspace. Shutting down cassandra and deleting the file ${CASSANDRA_HOME}/data/data/system/prepared_statements-*, then restart cassandra actually resets the prepared_statement cache.
Appreciate any help on this.
Thanks.
Update: 2018-06-01
We are currently using a work-around to clear prepared statements associated with certain tables by dropping index then recreating the index on the table. This discards prepared statements that have dependencies on the defined index. For now, this is the most we can do. Problem is, if this won't work for tables that don't have index defined on them.
Still need a better way of doing this, e.g. some admin command to clear the cache.
After I truncated and dropped a table in Cassandra, I still see the sstables on disk plus the lot of open file handler pointing to these.
What is the proper way to get rid of them?
Is there a possibility without restarting the Cassandra nodes?
We're using Cassandra 3.7.
In Cassandra data does not get removed immediately instead marked as tombstoned. You can run nodetool repairto get rid of deleted data.
I'm encountering the same problem as Cassandra system.hints table is empty even when the one of the node is down:
I am learning Cassandra from academy.datastax.com. I am trying the Replication and Consistency demo on local machine. RF = 3 and Consistency = 1.
When my Node3 is down and I am updating my table using update command, the SYSTEM.HINTS table is expected to store hint for node3 but it is always empty.
#amalober pointed out that this was due to a difference the Cassandra version being used. From the Cassandra docs at DataStax:
In Cassandra 3.0 and later, the hint is stored in a local hints directory on each node for improved replay.
This same question was asked 3 years ago, How to access the local data of a Cassandra node, but the accepted solution was to
...Hack something together using the Cassandra source that reads SSTables and have that feed the local client you're hoping to build. A great starting point would be looking at the source of org.apache.cassandra.tools.SSTableExport which is used in the sstable2json tool.
Is there an easier way to access the local hints directory of a Cassandra node?
Is there an easier way to access the local hints directory of a Cassandra node?
The hint directory is defined in $CASSANDRA_HOME/conf/cassandra.yaml file (sometimes it is located under /etc/cassandra also, depending on how you install Cassandra)
Look for the property hints_directory
I guess you are using ccm. So, the hint file should be in $CASSANDRA_HOME/.ccm/yourcluster/yournode/hints directory
I haven't been able to reproduce your issue with not getting a hints file. Every attempt I had resulted in the hints file as expected. There is a way to view the hints easier now.
We added a dump for hints in sstable-tools that you can use to view the mutations in the HH files. We may in the future add ability to use the HH files like sstables in the shell (use mutations to build memtable and include in queries) but for now its pretty raw.
Its pretty simple (sans metadata setup) if you wanna do analysis of data yourself. You can see what we did here and change to your needs: https://github.com/tolbertam/sstable-tools/blob/master/src/main/java/org/apache/cassandra/hints/HintsTool.java#L39
Cassandra service on one of my nodes went down and we couldnt restart it because of some corruption in one of the tables. So we tried rebuilding it by deleting all the data files and then starting the service, once it shows up in the ring we ran nodetool repair multiple times but it got hung throwing the same error
Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/var/lib/cassandra/data/profile/AttributeKey/profile-AttributeKey-ib-1848-Data.db): corruption detected, chunk at 1177104 of length 11576.
This occurs after 6gb of data is recovered. Also my replication factor is 3 so the same data is fine on the other 2 nodes.
I am a little new to Cassandra and am not sure what I am missing, has anybody seen this issue with repair? I have also tried scrubbing but it failed because of the corruption.
Please help.
rm /var/lib/cassandra/data/profile/AttributeKey/profile-AttributeKey-ib-1848-* and restart.
Scrub should not fail, please open a ticket to fix that at https://issues.apache.org/jira/browse/CASSANDRA.
first use the nodetool scrub if it does not fix
then shut down the node and run sstablescrub [yourkeyspace] [table] you will be able to remove the corrupted tables which were not done at nodetool scrub utility and run a repair you will be able to figure out the issue.
I'm using cassandra 0.7.4 on centos5.5 x86_64 with jdk-1.6.0_24 64-Bit.
When I restart it , it throw out:
ERROR 11:37:32,009 Exception encountered during startup.
java.io.IOError: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:476)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
Caused by: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
at org.apache.cassandra.config.CFMetaData.map(CFMetaData.java:223)
at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:472)
... 3 more
I try to location the problem: when I delete the file of the system keyspace ,It can restart sucess!
So I think this problem is cause by system Keyspace,even at the CF Scheam.
Then I build a new test environment, I know this proble is cause by this opeartion
update keyspace system with replication_factor=3;
But now how can i repair it ?!
There are many data on this cluster,and I couldn't lose data.
I have already do update keyspace system with replication_factor=1; ,but the problem still exist.
I try to use nodetool to repair after or befor flush, all no effect.
How can I restart cassandra without lose data ? Who can help me?
You should never modify the system keyspace unless you really, really know what you are doing. (If you have to ask, you don't. :)
So, the answer is: don't do that.
To recover, you should set initial_token in cassandra.yaml to your node's current token (which you can see with "nodetool ring"), then delete the system keyspace and restart. Then you'll need to recreate your columnfamily definitions, but your data will not be affected.