What is the suggested method for migrating 500GB KeySpace from Cassandra v3 with 20 nodes to Cassandra v4 with 40 nodes? (copy/dsbulk/sstableloader/spark)
Thanks
copy/dsbulk/sstableloader/spark
Related
I have this case. We have 6 nodes DSE cluster and the task is to back it up, and restore all the keyspaces, tables and data into a new cluster. But this new cluster has only 4 nodes.
Is it possible to do this?
Yes, it is definitely possible to do this. This operation is more commonly referred to as "cloning" -- you are copying the data from one DataStax Enterprise (DSE) cluster to another.
There is a Cassandra utility called sstableloader which reads the SSTables and loads it to a cluster even when the destination cluster's topology is not identical to the source.
I have previously documented the procedure in How to migrate data in tables to a new Cassandra cluster which is also applicable to DSE clusters. Cheers!
I have running 21 node Cassandra cluster with 150+ schema and about 20 TB data.I need to shift the schema and data from Cassandra to 7 node Scylla in no downtime scenario.
Both Scylla and Cassandra support the same cqlsh version and almost same in distributing the data and gossiping.
To shift the data I am trying to create new Scylla datacenter in existing Cassandra cluster and update the Keyspace topology to have Scylla also in the DC list of replication then Bootstrap/Rebuild the Scylla node in the cluster.
To do this I am getting error of TCP connection failure when adding seed list in node.
Scylla Error:-
scylla: [shard 0] rpc - client 10.200.1.2:34236: server connection dropped: connection is closed
scylla: [shard 0] rpc - client 10.200.1.2:7000: fail to connect: Connection refused.
Cassandra Error:-
[MessagingService-Outgoing-/10.200.2.2-Gossip] OutboundTcpConnection.java:411 - Socket to /10.200.2.2 closed
[HANDSHAKE-/10.200.2.2] OutboundTcpConnection.java:570 - Cannot handshake version with /10.200.2.2
[HANDSHAKE-/10.200.2.2] OutboundTcpConnection.java:561 - Handshaking version with /10.200.2.2
Please help me if anyone has done this already or any better idea of shifting data without downtime, without data loss in less risk.
You can not have an heterogeneous cluster with C* and Scylla nodes on the same cluster.
Create a separate scylla cluster, create the schema, change the app to do double writes (to both clusters) and then migrate the C* historical data to Scylla.
There are multiple ways to migrate the data. This should help: https://youtu.be/CDOesdWDT9Y No downtime, no problem there are options for that too.
While Scylla is compatible with Cassandra across several axes (SSTables, CQL/Drivers, etc.), Scylla did need to make some changes to the gossip protocol which make it impossible to join a Cassandra cluster. There is no known way to join Scylla to a Cassandra cluster.
Scylla has published several suggested techniques for migration.
Blog describing the techniques: https://www.scylladb.com/2019/04/02/spark-file-transfer-and-more-strategies-for-migrating-data-to-and-from-a-cassandra-or-scylla-cluster/
Webinar walking through the migration techniques [requires registration]: https://go.scylladb.com/wbn-spark-scylla-migration-strategies-registration.html
Documentation: https://docs.scylladb.com/operating-scylla/procedures/cassandra_to_scylla_migration_process/
Community Slack for Q&A: http://slack.scylladb.com
We have currently using DSE 4.8 and 5.12. we want to migrate to apache cassandra .since we don't use spark or search thought save some bucks moving to apache. can this be achieved without down time. i see sstableloader works other way. can any one share me the steps to follow to migrate from dse to apache cassandra. something like this from dse to apache.
https://support.datastax.com/hc/en-us/articles/204226209-Clarification-for-the-use-of-SSTABLELOADER
Figure out what version of Apache Cassandra is being run by DSE. Based on the DSE documentation DSE 4.8.14 is using Apache Cassandra 2.1 and DSE 5.1 is using Apache Cassandra 3.11
Simplest way to do this is to build another DC (Logical DC per Cassandra) and add it to the existing cluster.
As usual, with a "Nodetool Rebuild {from-old-DC}" on to the new DC nodes, let Cassandra take care of streaming data to the new Apache Cassandra nodes naturally.
Once data streaming is completed, based on the LoadBalancingPolicy being used by applications, switch their local_dc to DC2 (the new DC). Once the new DC starts taking traffic, shutdown nodes in old DC say DC1 one by one.
alter keyspace dse_system and dse_security not using everywhere
on non-seed nodes, cleanup cassandra data directory
turn on replace in cassandra-env.sh
start instance
monitoring streaming process using command 'nodetool netstats|grep Receiving'
change seeds node definition and rolling restart before finally migrate previous seeds nodes.
Is it at all possible to have two different DSE versions in the same cluster? In my case, I have a a cluster of two DSE 5 nodes and another one of two DSE 4.8.9 nodes. Can I connect them such that data is replicated from DSE 4.8.9 to DSE 5 in real time?
No. If you were to try this, you'd be in an "Upgrade State." And clusters in an upgrade state are bound by these restrictions:
Do not enable new features.
Do not run nodetool repair.
Do not issue
these types of CQL queries during a rolling restart: DDL and
TRUNCATE.
During upgrades, the nodes on different versions might show
a schema disagreement.
Failure to upgrade SSTables when required
results in a significant performance impact and increased disk usage.
Upgrading is not complete until the SSTables are upgraded.
Trying something like this would be further exacerbated by the fact that 4.8.9 is based on Cassandra 2.1 and 5.0 is based on Cassandra 3.0. There were some significant changes between the two, so you would undoubtedly run into problems.
The best way to go about this would be to upgrade your 4.8.9 nodes to 5.0 first, and then add your new 5.0 cluster nodes afterward.
What would be the minimal and the highest setup for a 90TB cassandra cluster. Kindly include the spec of processor, switch, hard disks and RAM. The no. of nodes is 5. Datastax's cassandra is gonna be used, so I guess in-memory function requires more amount of RAM.
I found a Document for determining the configurations for the Datastax cassandra nodes. Found here.
http://www.datastax.com/documentation/cassandra/2.0/pdf/cassandra20.pdf