I m facing a problem, Cassandra is not storing the data but other commit-log is working. I saw as per seeing the configuration .yaml file. I have checked the folder and Cassandra has created a folder in MYKEYSPACENAME, but the data is not been stored there.
Is there something that I need to store the data? I m using the Cassandra 1.0.7 version.
It sounds like everything is working normally. When Cassandra receives data to write, it first writes to a commit log and to an in memory data structure (called a Memtable). Once the Memtable is full Cassandra will flush it to an SSTable on disk. You can force Cassandra to flush its Memtables using nodetool:
nodetool flush [keyspace] [cfnames]
This is not something that you need to do for normal operation of a Cassandra ring. Cassandra will eventually flush the Memtables to disk. If for some reason one of your Cassandra machines goes down, when it restarts it will replay the commit log so you will not lose any previously received writes.
The Cassandra wiki has more information on Memtables and SSTables.
Related
Edited after reading nodetool tagged questions.
We take snapshots of our single node cassandra database daily. If I want to restore a snapshot either on that node, or on our staging server which is running a different instance of cassandra, my understanding is I have to:
nodetool disablegossip
nodetool disablebinary
nodetool drain
Copy the sstable files from the snapshot directories to the sstable directories under the keyspace directory.
Run nodetool refresh on each table.
Enable binary & gossip.
Is this sufficient to safely bring the snapshot sstable files in without cassandra overwriting them while I'm doing the refresh?
What is the opposite of nodetool drain?
Another edit: What about sstableloader? Should I use that instead? If so, how? I looked at the "documentation" and am none the wiser.
The steps you outlined isn't quite right. You don't shutdown Cassandra and you shouldn't just copy the files on top of the existing SSTables.
At a high level, the steps to restore table snapshots on a node are:
TRUNCATE the table you want to restore (will remove the SSTables from the data directories).
Copy the SSTables from data/ks_name/table-UUID/snapshots/snapshot_name subdirectory into the "live" data directory data/ks_name/table-UUID.
Run nodetool refresh -- ks_name table_name.
You will need to repeat these steps for each application table you want to restore. NOTE: Do NOT restore system tables, only application tables.
The detailed steps are documented in Restoring from a snapshot in Cassandra.
To restore a snapshot into another cluster, I prefer to refer to this as "cloning". The procedure for cloning snapshots to another cluster depends on whether the source and destination clusters have identical configuration.
If both source and destination clusters are identical, follow the steps I documented here -- https://community.datastax.com/questions/4534/. I've explained what identical configuration means in this post.
If they are not identical, follow the steps I documented here -- https://community.datastax.com/questions/4477/. Cheers!
I have a cassandra cluster running in the kubernetes environment, in a namespace, say test1, and I want to test the restore function. So I did a snapshot in the test1 cassandra, moved the snapshot to another node, from these data started a cassandra cluster in another namespace, say test2. The problem was, test2 cassandra cluster replaced test1 cluster totally, Customer's data that should write to the test1 cassandra cluster had written to the test2 cassandra cluster.
An hour later, I noticed this, stopped test2 cassandra cluster, and restarted test1 cassandra cluster, although it has come back to work shortly, but some data was lost.
After a while, I noticed there was some commitlog at that period in the test2 cassandra node, and want to recover these data. Can I just stop the test1 cassandra cluster, put these commitlog files into the commitlog directory of test1 cassandra node, then start cassandra, let cassandra to replay these commitlog ?
Commitlogs from one node can’t be played on a different node or
cluster. They are transactions specific to the node they came from.
source - read the notes ("Important note: A point-in-time restore requires a cluster restart for the commitlog replay to run" and "Some Helpful Notes for Planning")
Later update:
I'm not sure what you mean by "test2 cassandra cluster replaced test1 cluster totally". My assumption is that you restored everything, including system keyspaces.
If you did this, yes, then applying the commit logs might work since besides the IP's and the hostnames, the cluster is sort of the same.
If you look into the CommitLogReader code, you will see that a mutation is considered invalid if an UnknownTableException is thrown (basically if the id of the table is not the same between the commit log and the system keyspace).
I did a similar test on ccm and successfully replayed the commit logs after I changed the id of the table both on file system and in system_schema.tables.
From my perspective your cluster is pretty messed up. Although you could do this and it might work, I think you will always have a high risk of corrupt data.
So, since in the datastax documentation (which we could consider the base documentation for Cassandra) is stated that this operation is supported I am not recommending this.
I have below cassandra query ;
Few days ago i have developed application using c# and Single node Cassandra db. While the application in production, power failure occurred and cassandra commitlog got corrupt. Because of it cassandra node not starting, so i have shifted all commitlog files to another directory and started the cassandra node.
Recently i noticed the power failure day's data not available in database, I have all commitlog files with corrupted commitlog file name.
Can you please suggest, is there a way to recover data using commitlog files.
As well how to avoid commitlog file corruption issue, so that in production data loss can be avoid.
Thank you.
There is no way to restore back the node to the previous state if your commit logs have got corrupted and you have no SSTables with you.
If your commit logs are healthy (meaning it's not corrupted), then you just need to restart your node . It will be replayed,as a result will rebuild the memtable(s) and flush generation-1 SSTables on the disk.
What you can ideally do is to forcibly create SSTables.
You can do that under the apache-cassandra/bin directory by
nodetool flush
So if you are wary of losing commit logs .You can rebuild your node to previous states using SSTables so created above using
nodetool.bat refresh [keyspace] [columnfamily].
Alternatively you can also try creating snapshots.
nodetool snapshot
This command will take a snapshot of all keyspaces on the node.You also have the option of creating backups but this one will only keep record of the latest operations.
For more info try reading
https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsNodetool_r.html
I suggest you can also try having more nodes and thus increase the replication factor to avoid such scenarios in future.
Hope it helps!
I have a single-node Cassandra setup for my application. To reclaim disk space occupied by deleted records (tombstoned records), I triggered a nodetool compact for my keyspace. Unfortunately, this compaction process got interrupted. Now, when I try to re-start the service, it does not recognise the keyspace (from the data directory configured in cassandra.yaml) for which compaction was in progress when it got interrupted. Other keyspaces like system and system_traces are successfully initiated from the same data directory.
Has anybody encountered a similar issue before? Also, pointers to restore a keyspace only from data files would be of great help (for the lack of maintenance of snapshots).
PS: Upon analysing further it was found that an rm command on the cassandra data directory was issued but immediately cancelled. Most of the data seems to be in place, but there is a chance that the Data.db file of the system keyspace was lost. Is there a way to recover from this state?
Seems like you have corrupted your setup by deleting System keyspace files, hence Cassandra might not be checking the same at boot time.
Try this:
Download same version of cassandra again.
Create your keyspace & cf schemas
Move whatever old data is left to new data directory(cassandra will only load the non-corrupted data) -
sudo mv /data/cassandra_old/data/[keyspace]/[cf]-[md5-old]/* /data/cassandra_new/data/[keyspace]/[cf]-[md5-new]/
It should solve it if I understand the problem correctly.
If I make snapshots on every node in 10 node cluster, how to restore them into 5 node cluster where every node has stronger CPU and more storeage ?
The traditional way to restore sstable backups is by copying the sstable files to the data directory and calling 'refresh' to load the data into the running cluster. If the topology has changed, if you're unable to access the data directory, if you have filename collisions, or if you don't have sufficient room or time to deal with lots of nodes having a ton of data they don't own, then nodetool refresh may be less than ideal.
However, cassandra includes a bundled tool called 'sstableloader', which reads sstables from disk and writes them into a running cluster. sstableloader may be a good fit to load data from your sstables into the cluster, without worrying about the changed topology.
More info is available: http://www.pythian.com/blog/bulk-loading-options-for-cassandra/