I have the following cluster consisting of two node in two different datacenters:
Datacenter: 168
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID Token Rack
UN 10.168.92.13 47.44 KB 100.0% 70e6fb88-60d3-4f19-b4a7-4eacc6790042 -9223372036854775808 92
Datacenter: 186
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID Token Rack
UN 10.186.163.119 73.33 KB 100.0% 19714869-3d7a-434b-9c41-e7d90f14205c 0 163
I created a keyspace using NetworkTopologyStrategy to spread the data among the nodes. Assuming this will make a replica of the data in every datacenter, like this:
Create KEYSPACE demo WITH replication = {'class':'NetworkTopologyStrategy','DC1':1,'DC2':1};
Then I made a simple table user (id,name,last_name) but when I try to import or insert data to it I get the following error:
Traceback (most recent call last):
File "./bin/cqlsh", line 1108, in perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled)
File "/home/ubuntu/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 1602, in execute
result = future.result()
File "/home/ubuntu/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 3347, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}
I also set in cassandra.yaml file endpoint_snitch: RackInferringSnitch
Does anyone know what's going on?
The names of the datacenters that appear when using nodetool status must match the names of the dcs when defining the keyspace strategy, for example:
Create KEYSPACE demo WITH replication = {'class':'NetworkTopologyStrategy','DC1':1,'DC2':1};
should be:
Create KEYSPACE demo WITH replication = {'class':'NetworkTopologyStrategy','168':1,'186':1};
Either that or renaming the datacenters to DC1 and DC2 and executing node repair.
Related
I am loading some graph data sing titan API and configured cassandra as the storage backend. My graph data has around 1 million vertices. I want this data to be distributed across N cassandra nodes.
So for this, I configured 3 nodes in same system with IPs for each node as 127.0.0.1 , 127.0.0.2 and 127.0.0.3. The output of nodetool status shows all 3 IPs and load shared equally.
I tried loading a graph but the whole data is replicated in all 3 nodes (1M vertices in node1, 1M vertices in node2 and 1M vertices in node3). I want the data to be distributed across all 3 nodes, like 1M/3 in node1, 1M/3 in node2 and 1M/3 in node3.
output of DESCRIBE KEYSPACE TITAN:
CREATE KEYSPACE titan WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;
output of nodetool status:
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 7.79 MB 1 ? f5a689f0-f4c1-4f68-ab81-58066e986cd4 rack1
UN 127.0.0.2 229.79 KB 1 ? b6940e7e-b6eb-4d1f-959e-b5bd0f5cea15 rack1
UN 127.0.0.3 7.11 MB 1 ? a3244b16-a73c-4801-868f-05de09615ed9 rack1
Can someone please share me the details on correct configurations to share the load. Please correct me in case anything is wrong.
Thanks,
Hari
After setting up a 3 node cassandra cluster (cassandra version - 2.1.9), I ran the "nodetool status" command. I realized that the effective ownership % sums up to 200%.
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN <IP> 105.35 KB 256 67.4% <HostID> rack1
UN <IP> 121.92 KB 256 63.3% <HostID> rack1
UN <IP3> 256.11 KB 256 69.3% <HostID> rack1
Does any one know why would we get a 200% ownership? Is it because of some replication factor? If so, how do I find out about that?
Thanks!
This is dependent on the replication factor of the keyspace you are displaying.
For example, if you create a keyspace like this:
CREATE KEYSPACE test_keyspace WITH replication = {'class':
'NetworkTopologyStrategy', 'datacenter1': 2 };
And then display the status of that keyspace:
nodetool status test_keyspace
Then the Owns column will sum to 200%.
If you used a replication factor of 3, it would sum to 300%, and if you used a replication factor of 1, it would sum to 100%.
To see how a keyspace is defined, go into cqlsh and enter desc keyspace test_keyspace
I am trying to create cassandra cluster. For that I have a single node data centres
One data center is named DC1 and the other is DC2. Hence there are 2 single node data center. I followed the steps given here
http://docs.datastax.com/en/cassandra/2.0/cassandra/initialize/initializeMultipleDS.html
Since I have single node data center, so my seed will be a single machine. I can do ndoetool -h status to both the machines. I created a keyspace like this
CREATE KEYSPACE sams WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '1', 'DC2': '1'} AND durable_writes = true;
and a table like
CREATE TABLE apikey ( appname text, appkey text, PRIMARY KEY ((appname), appkey) );
I create keyspace and table in one data centre. This should get replicated on the other machine but it does not replicate. There are no keyspaces shown on the other data center neither any tables.
What am I missing here?
Adding nodetool output
nodetool -h cassandra1 status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 100.117.100.107 178.28 KB 256 100.0% 0c5da294-2a86-472d-98ec-857ed5140417 RAC1
nodetool -h cassandra2 status
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 100.117.150.55 162.94 KB 256 ? 9f3e49f6-debc-4a9c-ba93-bc65b3851a48 RAC1
It looks like you have accidentally set up two independent clusters rather than a single cluster that spans two data centers.
Make sure in cassandra.yaml that both nodes have the same cluster_name and that both have the same list of ip's for "- seeds:". Since you only have two nodes, I'd list both ip's as the seeds rather than just one.
If the nodes have joined the same cluster, then it should show both of them when you run nodetool status.
I want to verify and test the 'replication_factor' and the consistency level ONE of Cassandra DB.
And I specified a Cluster: 'MyCluster01' with three nodes in two data center: DC1(node1, node2) in RAC1, DC2(node3) in RAC2.
Structure shown as below:
[root#localhost ~]# nodetool status
Datacenter: DC1
===============
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.0.62 409.11 KB 256 ? 59bf9a73-45cc-4f9b-a14a-a27de7b19246 RAC1
UN 10.0.0.61 408.93 KB 256 ? b0cdac31-ca73-452a-9cee-4ed9d9a20622 RAC1
Datacenter: DC2
===============
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.0.63 336.34 KB 256 ? 70537e0a-edff-4f48-b5db-44f623ec6066 RAC2
Then, I created a keyspace and table like following:
CREATE KEYSPACE my_check1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};
create table replica_test(id uuid PRIMARY KEY);
After I inserted one record into that table:
insert into replica_test(id) values (uuid());
select * from replica_test;
id
--------------------------------------
5e6050f1-8075-4bc9-a072-5ef24d5391e5
I got that record.
But when I stopped node1 and queried again in either node 2 and node 3,
none of the query succeeded.
select * from replica_test;
Traceback (most recent call last): File "/usr/bin/cqlsh", line 997,
in perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled) File
"/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.3.post.zip/cassandra-driver-2.1.3.post/cassandra/cluster.py",
line 1337, in execute
result = future.result(timeout) File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.3.post.zip/cassandra-driver-2.1.3.post/cassandra/cluster.py",
line 2861, in result
raise self._final_exception Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE"
info={'required_replicas': 1, 'alive_replicas': 0, 'consistency':
'ONE'}
While the 'nodetool status' command returned:
UN 10.0.0.62 409.11 KB 256 ? 59bf9a73-45cc-4f9b-a14a-a27de7b19246 RAC1
DN 10.0.0.61 408.93 KB 256 ? b0cdac31-ca73-452a-9cee-4ed9d9a20622 RAC1
UN 10.0.0.63 336.34 KB 256 ? 70537e0a-edff-4f48-b5db-44f623ec6066 RAC2
And when I tried stopping node 2, keeping node 1 and 3 alive; or stopping node 3, keeping node 1 and 2 alive; The error occurred as well.
Then what 's the problem, since I think I 've already satisfied the consistency level, and where exactly does this record exists?
What ultimately does 'replication_factor' controls?
To directly answer the question, replication factor (RF) controls the number of replicas of each data partition that exist in a cluster or data center (DC). In your case, you have 3 nodes and a RF of 1. That means that when a row is written to your cluster, that it is only stored on 1 node. This also means that your cluster cannot withstand the failure of a single node.
In contrast, consider a RF of 3 on a 3 node cluster. Such a cluster could withstand the failure of 1 or 2 nodes, and still be able to support queries for all of its data.
With all of your nodes up and running, try this command:
nodetool getendpoints my_check1 replica_test 5e6050f1-8075-4bc9-a072-5ef24d5391e5
That will tell you on which node the data for key 5e6050f1-8075-4bc9-a072-5ef24d5391e5 resides. My first thought, is that you are dropping the only node which has this key, and then trying to query it.
My second thought echoes what Carlo said in his answer. You are using 2 DCs, which is really not supported with the SimpleStrategy. Using SimpleStrategy with multiple DCs could produce unpredictable results. Also with multiple DCs, you need to be using the NetworkTopologyStrategy and something other than the default SimpleSnitch. Otherwise Cassandra may fail to find the proper node to complete an operation.
First of all, re-create your keyspace and table with the NetworkTopologyStrategy. Then change your snitch (in the cassandra.yaml) to a network-aware snitch, restart your nodes, and try this exercise again.
NetworkTopologyStrategy should be used when replicating accross multiple DC.
I was trying to set up the user authentication for cassandra when I had the following problem.
First I have updated the cassandra.yaml with:
authenticator: PasswordAuthenticator
then created the system_auth keyspace with a replication_factor of 3 as:
CREATE KEYSPACE system_auth WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3};
Then re-ran cassandra with the command:
bin/cassandra
Then after all that, I proceeded and made sure to repair my node(s) with (though I am not sure if I understand why this is important/necessary):
nodetool repair
Even after doing all of those steps cassandra/java yells at me with:
Traceback (most recent call last):
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/cqlsh", line 2044, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/cqlsh", line 2030, in main
display_float_precision=options.float_precision)
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/cqlsh", line 480, in __init__
cql_version=cqlver, transport=transport)
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/../lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/connection.py", line 143, in connect
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/../lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/connection.py", line 59, in __init__
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/../lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/thrifteries.py", line 157, in establish_connection
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/../lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py", line 465, in login
File "/home/tscobb/Documents/BM/apache-cassandra-2.0.5/bin/../lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py", line 486, in recv_login
cql.cassandra.ttypes.AuthenticationException: AuthenticationException(why='org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM')
Honestly, at this point I am not sure how to proceed. If anyone has any ideas, it would be greatly appreciated.
I am running cassandra 2.0.5 in a Ubuntu vm. Not sure if that helps at all.
Also, when I run:
nodetool status
The following comes up:
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 1.57 MB 256 100.0% 37ae1d4c-0df5-43d2-9593-4603ac68c34a rack1
'class' : 'SimpleStrategy', 'replication_factor' : 3
As you only have one node in your cluster, setting a replication factor of 3 would be your problem. Here is an article that describes replication in Cassandra. You should give it a quick read through. One part in particular applies here:
When replication factor exceeds the number of nodes, writes are
rejected, but reads are served as long as the desired consistency
level can be met.
Cannot achieve consistency level QUORUM
Cassandra computes "Quorum" as (replication_factor / 2) + 1. So if you have a replication factor of 3, then 2 nodes need to be running for your consistency level to be met.
Basically, set your replication factor back down to 1, and you should be fine.