I recently started working with an Elassandra cluster with two data centers which have been configured using NetworkTopologyStrategy.
Cluster details : Elassandra 6.2.3.15 = Elasticsearch 6.2.3 + Cassandra 3.11.4
Datacenter: DC1
=================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN <ip1> 50 GiB 256 ? 6cab1f4c-8937-437d-b010-0a5677443dc3 rack1
UN <ip2> 48 GiB 256 ? 6c9e7ad5-a642-4c0d-8b77-e78d821d904b rack1
UN <ip3> 50 GiB 256 ? 7e493bc6-c8a5-471e-8eee-3f3fe985b90a rack1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN <ip4> 47 GiB 256 ? c49c1203-cc38-41a2-b9c8-2b42bc907c17 rack1
UN <ip5> 67 GiB 256 ? 0d9f31bc-9690-49b6-9d88-4fb30c1b6c0d rack1
UN <ip6> 88 GiB 256 ? 80c4d60d-185f-457a-ae9b-2eb611735f07 rack1
schema info
CREATE KEYSPACE my_keyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3'} AND durable_writes = true;
The DC2 is kind of a Disaster Recovery site and in an ideal world, we should be able to use only that in case of a disaster.
With the very limited knowledge I have, I strongly suspect that we need
to modify the rack configuration to have a 'proper' D/R cluster (So
that data in DC1 gets replicated in DC2) or am I getting this
wrong? If so, is there a standard guideline to follow?
When there are multiple DCs, does Cassandra automatically replicate this regardless of rack configurations? (Are racks kind of additional fail proof?)
DC2 has more data than DC1. Is this purely related to hash function?
Is there any other things that can be rectified in this cluster?
Many thanks!
These replication settings mean that the data for your keyspace is replicated in real time between the 2 DCs with each DC having 3 replicas (copies):
CREATE KEYSPACE my_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'DC1': '3',
'DC2': '3'
}
Replication in Cassandra happens in real time -- any writes sent to one DC is sent to all other DCs at the same time. Unlike traditional RDBMS or configurations with primary/secondary or active/DR, Cassandra replication is instantaneous and immediate.
The logical Cassandra racks are for additional redundancy mechanism. If you have C* nodes deployed in different (a) physical racks, or (b) public cloud availability zones, Cassandra will distribute the replicas to separate racks so each rack has a full copy of the data. With a replication factor of 3 in the DC, if a rack goes down for whatever reason then there's still full copies of the data in the remaining 2 racks and read/write requests with a consistency of LOCAL_QUORUM (or lower) will not be affected.
I've explained this in a bit more detail in this post -- https://community.datastax.com/questions/1128/.
If you're new to Cassandra, we recommend https://www.datastax.com/dev which has links to short hands-on tutorials where you can quickly learn the basics of Cassandra -- all free. This tutorial is a good place to start -- https://www.datastax.com/try-it-out. Cheers!
Related
I am loading some graph data sing titan API and configured cassandra as the storage backend. My graph data has around 1 million vertices. I want this data to be distributed across N cassandra nodes.
So for this, I configured 3 nodes in same system with IPs for each node as 127.0.0.1 , 127.0.0.2 and 127.0.0.3. The output of nodetool status shows all 3 IPs and load shared equally.
I tried loading a graph but the whole data is replicated in all 3 nodes (1M vertices in node1, 1M vertices in node2 and 1M vertices in node3). I want the data to be distributed across all 3 nodes, like 1M/3 in node1, 1M/3 in node2 and 1M/3 in node3.
output of DESCRIBE KEYSPACE TITAN:
CREATE KEYSPACE titan WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;
output of nodetool status:
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 7.79 MB 1 ? f5a689f0-f4c1-4f68-ab81-58066e986cd4 rack1
UN 127.0.0.2 229.79 KB 1 ? b6940e7e-b6eb-4d1f-959e-b5bd0f5cea15 rack1
UN 127.0.0.3 7.11 MB 1 ? a3244b16-a73c-4801-868f-05de09615ed9 rack1
Can someone please share me the details on correct configurations to share the load. Please correct me in case anything is wrong.
Thanks,
Hari
Cassandra version: dsc-cassandra-2.1.9
Had 3 nodes, one of which was down for a long time. Brought it back up and decomissioned it. Then did a nodetool removenode.
When I try to make a cql query I get the above error.
Initially thought this might be because replication strategy was SimpleStrategy. So did a ALTER KEYSPACE history WITH REPLICATION =
{'class' : 'NetworkTopologyStrategy', 'dc1' : 2};
and changed the endpoint_snitch: GossipingPropertyFileSnitch instead of SimpleSnitch
did a nodetool repair on both nodes and restarted the cassandra services
But the problem is still there. What do I do?
EDIT 1: Nodetool status of machine A
-- Address Load Tokens Owns Host ID Rack
UN 192.168.99.xxx 19.8 GB 256 ? xxxxxxxx-xxxx-xxx-xxxx-xxxxx4ea RAC1
UN 192.168.99.xxx 18.79 GB 256 ? xxxxxxxx-xxxx-xxx-xxxx-xxxxxx15 RAC1
nodetool status output of machine B
-- Address Load Tokens Owns Host ID Rack
UN 192.168.99.xxx 19.8 GB 256 ? xxxxxxxx-xxxx-xxx-xxxx-xxxxxxxx4ea RAC1
UN 192.168.99.xxx 18.79 GB 256 ? xxxxxxxx-xxxx-xxx-xxxx-xxxxxxxxf15 RAC1
What is weird is that under the columns Owns you have no %, only the ?. This same issue occured to me in the past when I bootstrapped a new C* cluster and was using SimpleStrategy and SimpleSnitch. I did like you an ALTER KEYSPACE to switch to NetworkTopology and GossipingPropertyFileSnitch but it did not solve my issue so I rebuilt the cluster from scratch (fortunately I had no data inside)
If you have a backup of your data somewhere, just try to rebuild the 2 nodes from scratch.
Otherwise, consider backing up your sstable files on one node and rebuild the cluster and put the sstables back. Be careful because some file name/folder renaming may be necessary
After setting up a 3 node cassandra cluster (cassandra version - 2.1.9), I ran the "nodetool status" command. I realized that the effective ownership % sums up to 200%.
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN <IP> 105.35 KB 256 67.4% <HostID> rack1
UN <IP> 121.92 KB 256 63.3% <HostID> rack1
UN <IP3> 256.11 KB 256 69.3% <HostID> rack1
Does any one know why would we get a 200% ownership? Is it because of some replication factor? If so, how do I find out about that?
Thanks!
This is dependent on the replication factor of the keyspace you are displaying.
For example, if you create a keyspace like this:
CREATE KEYSPACE test_keyspace WITH replication = {'class':
'NetworkTopologyStrategy', 'datacenter1': 2 };
And then display the status of that keyspace:
nodetool status test_keyspace
Then the Owns column will sum to 200%.
If you used a replication factor of 3, it would sum to 300%, and if you used a replication factor of 1, it would sum to 100%.
To see how a keyspace is defined, go into cqlsh and enter desc keyspace test_keyspace
I am trying to create cassandra cluster. For that I have a single node data centres
One data center is named DC1 and the other is DC2. Hence there are 2 single node data center. I followed the steps given here
http://docs.datastax.com/en/cassandra/2.0/cassandra/initialize/initializeMultipleDS.html
Since I have single node data center, so my seed will be a single machine. I can do ndoetool -h status to both the machines. I created a keyspace like this
CREATE KEYSPACE sams WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '1', 'DC2': '1'} AND durable_writes = true;
and a table like
CREATE TABLE apikey ( appname text, appkey text, PRIMARY KEY ((appname), appkey) );
I create keyspace and table in one data centre. This should get replicated on the other machine but it does not replicate. There are no keyspaces shown on the other data center neither any tables.
What am I missing here?
Adding nodetool output
nodetool -h cassandra1 status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 100.117.100.107 178.28 KB 256 100.0% 0c5da294-2a86-472d-98ec-857ed5140417 RAC1
nodetool -h cassandra2 status
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 100.117.150.55 162.94 KB 256 ? 9f3e49f6-debc-4a9c-ba93-bc65b3851a48 RAC1
It looks like you have accidentally set up two independent clusters rather than a single cluster that spans two data centers.
Make sure in cassandra.yaml that both nodes have the same cluster_name and that both have the same list of ip's for "- seeds:". Since you only have two nodes, I'd list both ip's as the seeds rather than just one.
If the nodes have joined the same cluster, then it should show both of them when you run nodetool status.
I want to verify and test the 'replication_factor' and the consistency level ONE of Cassandra DB.
And I specified a Cluster: 'MyCluster01' with three nodes in two data center: DC1(node1, node2) in RAC1, DC2(node3) in RAC2.
Structure shown as below:
[root#localhost ~]# nodetool status
Datacenter: DC1
===============
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.0.62 409.11 KB 256 ? 59bf9a73-45cc-4f9b-a14a-a27de7b19246 RAC1
UN 10.0.0.61 408.93 KB 256 ? b0cdac31-ca73-452a-9cee-4ed9d9a20622 RAC1
Datacenter: DC2
===============
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.0.63 336.34 KB 256 ? 70537e0a-edff-4f48-b5db-44f623ec6066 RAC2
Then, I created a keyspace and table like following:
CREATE KEYSPACE my_check1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};
create table replica_test(id uuid PRIMARY KEY);
After I inserted one record into that table:
insert into replica_test(id) values (uuid());
select * from replica_test;
id
--------------------------------------
5e6050f1-8075-4bc9-a072-5ef24d5391e5
I got that record.
But when I stopped node1 and queried again in either node 2 and node 3,
none of the query succeeded.
select * from replica_test;
Traceback (most recent call last): File "/usr/bin/cqlsh", line 997,
in perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled) File
"/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.3.post.zip/cassandra-driver-2.1.3.post/cassandra/cluster.py",
line 1337, in execute
result = future.result(timeout) File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.3.post.zip/cassandra-driver-2.1.3.post/cassandra/cluster.py",
line 2861, in result
raise self._final_exception Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE"
info={'required_replicas': 1, 'alive_replicas': 0, 'consistency':
'ONE'}
While the 'nodetool status' command returned:
UN 10.0.0.62 409.11 KB 256 ? 59bf9a73-45cc-4f9b-a14a-a27de7b19246 RAC1
DN 10.0.0.61 408.93 KB 256 ? b0cdac31-ca73-452a-9cee-4ed9d9a20622 RAC1
UN 10.0.0.63 336.34 KB 256 ? 70537e0a-edff-4f48-b5db-44f623ec6066 RAC2
And when I tried stopping node 2, keeping node 1 and 3 alive; or stopping node 3, keeping node 1 and 2 alive; The error occurred as well.
Then what 's the problem, since I think I 've already satisfied the consistency level, and where exactly does this record exists?
What ultimately does 'replication_factor' controls?
To directly answer the question, replication factor (RF) controls the number of replicas of each data partition that exist in a cluster or data center (DC). In your case, you have 3 nodes and a RF of 1. That means that when a row is written to your cluster, that it is only stored on 1 node. This also means that your cluster cannot withstand the failure of a single node.
In contrast, consider a RF of 3 on a 3 node cluster. Such a cluster could withstand the failure of 1 or 2 nodes, and still be able to support queries for all of its data.
With all of your nodes up and running, try this command:
nodetool getendpoints my_check1 replica_test 5e6050f1-8075-4bc9-a072-5ef24d5391e5
That will tell you on which node the data for key 5e6050f1-8075-4bc9-a072-5ef24d5391e5 resides. My first thought, is that you are dropping the only node which has this key, and then trying to query it.
My second thought echoes what Carlo said in his answer. You are using 2 DCs, which is really not supported with the SimpleStrategy. Using SimpleStrategy with multiple DCs could produce unpredictable results. Also with multiple DCs, you need to be using the NetworkTopologyStrategy and something other than the default SimpleSnitch. Otherwise Cassandra may fail to find the proper node to complete an operation.
First of all, re-create your keyspace and table with the NetworkTopologyStrategy. Then change your snitch (in the cassandra.yaml) to a network-aware snitch, restart your nodes, and try this exercise again.
NetworkTopologyStrategy should be used when replicating accross multiple DC.