I'm trying to build two 3-node Cassandra clusters in separate data centers. I want to have NetworkToplogyStrategy replication between them, with a replication factor of 3 in each. Thus, I want each node in each data center to have the same records.
Question, what should my token assignment look like for each node? (since i'm not actually partitioning, just replicating).
Thank you!
If you're using Cassandra 1.2 use virtual nodes with automatic assignment.
If you're using 1.1 or earlier, use for one DC the evenly distributed tokens:
0
56713727820156410577229101238628035242
113427455640312821154458202477256070484
(0, 1 and 2 times 2**127/3)
For the other DC, you can choose anything as long as it is also evenly distributed. Offsetting by 1 works:
1
56713727820156410577229101238628035243
113427455640312821154458202477256070485
Although for now the tokens don't matter since all nodes hold the same data, if you want to scale in the future it will help to have them already balanced.
Related
Highly appreciate if someone can help with below questions.
*RF= Replication Factor
*CL= Consistency Level
We have requirement of strong Consistency and higher Availability. So, I have been testing RF and CL for 7 nodes ScyllaDB cluster , by keeping RF=7 (100% data on each node) and CL=QUORUM.
What will happen to data copy / replication if 2 nodes goes down ? Does it replicate 2 down nodes data (6th & 7th copy) on to remaining 5 nodes?
or will it simply discard these copies ? What will be effect of RF=7 when there are only 5 active nodes ?
I could not find anything in logs. Do we have any document/link reference for this case ? Or how can I verify and prove this behaviour? Please explain?
With RF=7, the data is always replicated to 7 nodes.
When a node (or two) goes down, the rest of the five nodes already have a copy, and no additional streaming is required.
Using CL=QUORUM, even three nodes down, will not hurt your HA or consistency.
When the fail nodes come back to life, they will be sync, either automatically using Hinted Handoff (for a short failure) or with Repair (for longer failure)[1]
If you replace a dead node[2], the other replicas will stream the data to it till it is up to speed with the
[1] https://docs.scylladb.com/architecture/anti-entropy/
[2] https://docs.scylladb.com/operating-scylla/procedures/cluster-management/replace_dead_node/
Data will always replicate to all nodes cause you have set RF=7 if 2 nodes down then remaining nodes will store hints for those nodes once, nodes come up remaining nodes will replicate the data automatically based on hint period.If hint period(default 3 hours) expired then you need to run manual repair to get data sync in the cluster.
If there is a 4 node Cassandra cluster, is it possible to configure Cassandra in a way to have half of the nodes down (two in this case) without affecting the applications?
Also how long can nodes be down without Cassandra cancelling the write queue?
This depends on the client CL and DC replication factor.
Let's assume the RF is 4 (all), if the client has a CL=ONE or LOCAL_ONE, the application would not notice any issues. Any other client CL would have problems (e.g. cl=local_quorum of 4 is 3, allowing only a single node to be down).
Let's assume the RF=1 or 2. If CL=ONE or LOCAL_ONE, the application would be unaffected by queries that only manipulate data on the available nodes. However, any access to rows that only exist on the unavailable nodes would be impacted. In other words, CL=ONE or LOCAL_ONE only works if you're manipulating data that has at least one node available to return the response (You only need ONE to respond in this scenario). If the rows you're querying are on both of the unavailable nodes, you'll get an error stating something like: Expected response of 1, received 0.
Many applications configure CL to be some sort of quorum (local or not) - so in that case, the application would certainly fail unless you had RF=5 (so at least 5 nodes). Quorum of 5 is 3, allowing for 2 nodes to fail.
Hopefully that makes sense.
Yes, assuming you are talking about all four nodes in one data centre, if you set your replication factor to 3 or greater and your read and write consistency level to ONE.
For writes the nodes that are up will store hints for the nodes that are down, so when they come back up they can write the data. How long the nodes store these hints can be set in cassandra.yaml.
When creating a new namespace in Cassandra, we need to give a number for a replication factor.
Ex:
Does the number, that we are giving as the replication factor, determine the number of nodes that initially create to store the replicate data?
Can anybody give a clear clarification about what that replication factor does?
It will not create the number of nodes specified. It just means the number of copies of data. For instance if your cluster is having 5 nodes, your write will be replicated(written) to 3 different nodes depending on the token range it falls. Coming to SimpleStrategy its asn implementation where it does not consider rack or dc's into consideration when replicating.
The explanation #Praneeth Gudumasu given for replication_factor is true. The number of nodes in a Cassandra cluster is not something you "give", you can actually connect as many number of nodes as you wish: https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddNodeToCluster.html
and each time you connect a new node it is assigned a token range as per Cassandra's architecture. If you don't know how many nodes you need for your application I suggest running a performance test with data size approaching the size you would be inserting in your real application, then try to execute some queries (concurrently) and see with how many nodes you would get a reasonable response time for your queries.
In Mongo we can go for any of the below model
Simple replication(without shard where one node will be working as master and other as slaves) or
Shard(where data will be distributed on different shard based on partition key)
Both 1 and 2
My question - Can't we have Cassandra just with replication without partitioning just like model_1 in mongo ?
From Cassandra vs MongoDB in respect of Secondary Index?
In case of Cassandra, the data is distributed into multiple nodes based on the partition key.
From above it looks like it is mandatory to distribute the data based on some p[artition key when we have more than one node ?
In Cassandra, replication factor defines how many copies of data you have. Partition key is responsible for distributing of data between nodes. But this distribution may depend on the amount of nodes that you have. For example, if you have 3 nodes cluster & replication factor equal to 3, then all nodes will get data anyway...
Basically your intuition is right: The data is always distributed based on the partition key. The partition key is also called row key or primary key, so you can see: you have one anyway. The 1. case of your mongo example is not doable in cassandra, mainly because cassandra does not know the concept of masters and slaves. If you have a 2 node cluster and a replication factor of 2, then the data will be held on 2 nodes, like Alex Ott already pointed out. When you query (read or write), your client will decide to which to connect and perform the operation. To my knowledge, the default here would be a round robin load balancing between the two nodes, so either of them will receive somewhat the same load. If you have 3 nodes and a replication factor of 2, it becomes a little more tricky. The nice part is though, that you can determine the set of nodes which hold your data in the client code, thus you don't lose any performance by connecting to a "wrong" node.
One more thing about partitions: you can configure some of this, but this would be per server and not per table. I've never used this, and personally i wouldn't recommend to do so. Just stick to the default mechanism of cassandra.
And one word about the secondary index thing. Use materialized views
I am new to Cassandra and I would like to learn more about Cassandra's racks and structure.
Suppose I have around 70 column families in Cassandra and two AWS2 instances.
How many Data Centres will be used?
How many nodes will each rack have?
Is it possible to divide a column family in multiple keyspaces?
The intent of making Cassandra aware of logical racks and data centers is to provide additional levels of fault tolerance. The idea (as described in this document, under the "Network Topology Strategy") is that the application should still be able to function if one rack or data center goes dark. Essentially, Cassandra...
places replicas in the same data center by walking the ring clockwise
until reaching the first node in another rack. NetworkTopologyStrategy
attempts to place replicas on distinct racks because nodes in the same
rack (or similar physical grouping) often fail at the same time due to
power, cooling, or network issues.
In this way, you can also query your data by LOCAL_QUORUM, in which QUORUM ((replication_factor / 2) + 1) is only computed from the nodes present in the same data center as the coordinator node. This reduces the effects of inter-data center latency.
As for your questions:
How many data centers are used are entirely up to you. If you only have two AWS instances, putting them in different logical data centers is possible, but only makes sense if you are planning to use consistency level ONE. As-in, if one instance goes down, your application only needs to worry about finding one other replica. But even then, the snitch can only find data on one instance, or the other.
Again, you can define the number of nodes that you wish to have for each rack. But as I indicated with #1, if you only have two instances, there isn't much to be gained by splitting them into different data centers or racks.
I do not believe it is possible to divide a column family over multiple keyspaces. But I think I know what you're getting at. Each keyspace will be created on each instance. As you have 2 instances, you will be able to specify a replication factor of 1 or 2. If you had 3 instances, you could set a replication factor of 2, and then if you lost 1 instance you would still have access to all the data. As you only have 2 instances, you need to be able to handle one going dark, so you will want to make sure both instances have a copy of every row (replication factor of 2).
Really, the logical datacenter/rack structure becomes more-useful as the number of nodes in your cluster increases. With only two, there is little to be gained by splitting them with additional logical barriers. For more information, read through the two docs I linked above:
Apache Cassandra 2.0: Data Replication
Apache Cassandra 2.0: Snitches