I am new to cassandra and trying to figure out how to restore data from Snapshots+incremental backups onto a new node.
Below is my understanding :
1.take snapshot of the cassandra node
2.enable incremental backups
3.copy the snapshots / incremental backups to remote location
4.for restore data on new node :
- install cassandra
- copy the keyspace / table structure plus snapshot data on the new node from remote location
Now my real question is where do i place the incremental backup files ?
do they go in same backups folder on new node under /keyspace/table/backups or i need to copy these files to the main folder /keyspace/table ?
Also how to copy when I have multiple incremental backup files ?
Related
We have three node cluster for Cassandra/Elassandra and I needs to setup backup for this. I used "nodetool snapshot" command for taking backup, but as we are using elasticserach so do I need to take separate backups of Indices or taking backup from "nodetool snapshot" is enough for this.
if separate backup is required for indices then can you pls suggest me how to take backup/restore because there is no proper documentation for taking elassendra backup/restore
Thanks
Since
Elassandra = Elasticsearch + Cassandra, So you need backup from Cassandra on the same time of backup from Elasticsearch.
By design, Elassandra synchronously updates Elasticsearch indices on the Cassandra write path. Therefore, Elassandra can backup data by taking a snapshot of Cassandra SSTables and Elasticsearch Lucene files on the same time on each node, as follows :
For Cassandra SSTables use:
nodetool snapshot --tag <snapshot_name> <keyspace_name>
And for index files use copy them by:
cp -al $CASSANDRA_DATA/elasticsearch.data/<cluster_name>/nodes/0/indices/<index_name>/0/index/(_*|segment*) $CASSANDRA_DATA/elasticsearch.data/snapshots/<index_name>/<snapshot_name>/
However there is a documentation on Elassandra to Backup and Restore.
I am working on a cassandra backup and recovery strategy for our cassandra system and am trying to understand how the backup and sstable recovery works in cassandra. Here are of my observations and related questions (my need is to setup a standby/backup cluster which would become active if the primary cluster goes down.. so I want to keep them in sync in terms of data, so I want to take periodic backups at my active cluster and recover to the standby cluster)
Took a snapshot backup. Dropped a table in cassandra. Stopped cassandra, recovered from the snapshot backup (copied the sstables to the data/ folder), started cassandra. Ran cqlsh on the node, and I still do not see the table created. Should this work? Am I missing any step ?
In the above scenario, I then tried to re-setup the schema (I take backup of the schema in the snapshot) using the cql commant source . This created the table for me. However it creates a "new version" of table for me. When I recover the snapshot has the older version (different uuid labelled folders for table). After recovery, I still see no data in the table. Possibly because I created a new table?
I was finally able to recover data after running nodetool repair and using sstableloader to restore table data from another node in the cluster.
My question is
a. what is the right way to setup a new (blank- no schema) cluster from a snapshot? How do you setup the schema and recover data?
b. how do you restore a cluster from a backup with table alterations. How do you bring a cluster running an older version of schema to a newer version of schema when recovering from a backup (snapshot or incremental)?
(NOTE: cassandra newbie here)
So if you want to restore a snapshot, you need to copy the snapshot files back to the sstable directory and then run: nodetool refresh. You can read:
https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsBackupSnapshotRestore.html
for more information. Also, there are 3rd party tools that can back up your data and then restore it as it was at the time of the backup. We use a tool: Cohesity (formally Talena/Imanis). It has a lot of capabilities (refreshing A to B, restore/rename, etc.). There are other popular ones as well. All of them have costs associated with them.
Hopefully that helps?
-Jim
We have a regular backup of our cluster and we store schema and snapshot back up on aws s3 on daily basis.
Somehow we have lost all the data and while recovering the data from backup we are able to recover schema but while copying snapshots files to /var/lib/cassandra/data directory its not showing up the data in the tables.
After copying the data we have done nodetool refresh -- keyspace table but still nothing is working out.
could you please help on this ?
Im new at Apache Cassandra, but my first focus at this topic was the Backup.
If you want to restore from a Snapshot (on new node/cluster) you have to shut down Cassandra on any node and clear any existing data from these folders:
/var/lib/cassandra/data -> If you want to safe your System Keyspaces so delete only your Userkeyspaces folders
/var/lib/cassandra/commitlog
/var/lib/cassandra/hints
/var/lib/cassandra/saved_cashes
After this, you have to start Cassandra again (the whole Cluster). Create the Keyspace like the one you want to restore and the table you want to restore. In Your Snapshot folder you will find a schema.cql script for the creation of the table.
After Creating the Keyspaces an tables again, wait a moment (time depends on the ammount of nodes in your cluster and keypsaces you want to restore.)
Shut down the Cassandra Cluster again.
Copy the Files from the Snapshot folder to the new folders of the tables you want to restore. Do this on ALL NODES!
After copying the files, start the nodes one by one.
If all nodes are running, run the nodetool repair command.
If you try to check the data via CQLSH, so think of the CONSISTENCY LEVEL! (ALL/QUORUM)
Thats the way, wich work at my Cassandra cluster verry well.
The general steps to follow for restoring a snapshot is:
1.Shutdown Cassandra if still running.
2.Clear any existing data in commitlogs, data and saved caches directories
3.Copy snapshots to relevant data directories
4.Copy incremental backups to data directory (if incremental backups are enabled)
If required, set restore_point_in_time parameter in commitlog_archiving.properties to
restore point.
5.Start Cassandra.
6.Run repair
So try running repair after copying data.
I am trying to implement Cassandra(2.2.6) backup and restore using nodetool commands. Currently I have four nodes in Cassandra cluster out of which two are seed nodes with incremental_backup : enabled
I have followed below steps for backup and restore on one of the nodes:
Creating a snapshot of a keyspace with a sample table:
nodetool snapshot testkeyspace
Drop the test table from the given keyspace : cqlsh> Drop table test
Recreate the schema of the above dropped table
Stop the Cassandra service
Clear cassandra-commitlog directory
Copy the content of backup directory and latest snapshot directory from the old directory of test table into the new test table directory
Start Cassandra service
Run nodetool repair
Run nodetool refresh testkeyspace test
After the restore when I query the table, some of the records are missing. I am not sure what exactly is going wrong.
I am learning Cassandra now-a-days, i have successfully backup and restore tables or keyspace mentioned in this URL.
But i am looking for following options
1)Take complete backup of a keyspace at different location other then mentioned directory in cassandra.yaml. -t option create directory in snapshot folder not different HDD location.
2) Or backup/restore procedure same like mysql.
Thanks
You have a few options...for small amounts of data, you can use COPY to backup / restore from csv:
http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/copy_r.html
For larger stuff, you've got the right link. You essentially take a snapshot (which puts it in the folder you mention), and then use something like tar to zip the files and output to a different directory. This is what we're doing in production... we clear the previous snapshot, take a snapshot and tar the folder to a network backup.