I try to remove node from cassandra cluster v.2.2.4.
The 'nodetool decommission' was run on removed node.
Now I have following:
nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.0.0.158 30.83 GB 256 ? 8b49b920-30c6-4b99-b00e-f1667e12386a r1
UN 10.0.0.0.77 31.01 GB 256 ? 4c207524-8e8a-44a5-a428-ce5027406eb2 r1
UN 10.0.0.0.154 28.5 GB 256 ? 12ed1ee7-9425-4107-95e1-420067ef536d r1
UN 10.0.0.0.152 28.69 GB 256 ? 7cd58f7f-ba4c-455c-80a8-a8dca856ac38 r1
UN 10.0.0.0.150 28.22 GB 256 ? f59e8f67-1459-43ae-8469-7ac455b1d858 r1
UN 10.0.0.0.85 59.03 GB 256 ? c372b9e6-922b-4198-aa60-558ae2a181bd r1
UN 10.0.0.0.133 33.44 GB 256 ? 54fcb413-3a46-46ae-905d-53fb8c8e0534 r1
nodetool describecluster
Cluster Information:
Name: testCluster1
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
15c69d7a-ff7b-3f3a-8190-a4632cfebe6b: [10.0.0.0.158, 10.0.0.0.77, 10.0.0.0.154, 10.0.0.0.152, 10.0.0.0.150, 10.0.0.0.133, 10.0.0.0.85]
UNREACHABLE: [10.0.0.0.212]
Why is the removed node 10.0.0.0.212 marked 'UNREACHABLE'?
How can this issue be resolved?
Related
Node tool repair command is failing to repair form some of tables . Cassandra version is 3.11.6 version . I have following queries :
Is this really a problem . what is the impact if we ignore this error ?
How can we get rid of this token range for some keyspace ?
what is the possible reason that it complaining about this token range ?
here is the error trace :
[2020-11-12 16:33:46,506] Starting repair command #6 (d5fa7530-2504-11eb-ab07-59621b514775), repairing keyspace solutionkeyspace with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 256, pull repair: false, ignore unreplicated keyspaces: false)
[2020-11-12 16:33:46,507] Repair command #6 failed with error Nothing to repair for (1838038121817533879,1854751957995458118] in solutionkeyspace - aborting
[2020-11-12 16:33:46,507] Repair command #6 finished with error
error: Repair job has failed with the error message: [2020-11-12 16:33:46,507] Repair command #6 failed with error Nothing to repair for (1838038121817533879,1854751957995458118] in solutionkeyspace - aborting
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message: [2020-11-12 16:33:46,507] Repair command #6 failed with error Nothing to repair for (1838038121817533879,1854751957995458118] in solutionkeyspace - aborting
at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
Scheme definition
CREATE KEYSPACE solutionkeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '1', 'datacenter2': '1'} AND durable_writes = true;
CREATE TABLE solutionkeyspace.schemas (
namespace text PRIMARY KEY,
avpcontainer map<text, text>,
schemacreationcql text,
status text,
version text
) WITH bloom_filter_fp_chance = 0.001
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 10800
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.05
AND speculative_retry = '99PERCENTILE';
nodetool status output
bash-5.0# nodetool -p 7199 status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.16.0.68 5.71 MiB 256 ? 6a4d7b51-b57b-4918-be2f-3d62653b9509 rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
It looks like you are running Cassandra version 3.11.9+ where this new behaviour was introduced.
https://issues.apache.org/jira/browse/CASSANDRA-15160
You will not see this error if you add the following nodetool repair command line option;
-iuk
or
--ignore-unreplicated-keyspaces
This change would have broken people's repair scripts when they upgrade from an earlier version of Cassandra to 3.11.9+. We certainly noticed it.
I'm new to DSE and Cassandra.
I have a cluster consists of 3 DC: "analytic", "dc2" , "trans".
All use the same profile DSE, with DSEAuth.
So the problem is the LIST and CREATE command only work when I use cqlsh to DC trans.
The other will replay NoHostAvailabe.
[root#bigdata-142-116 ~]# nodetool status
Datacenter: analytic
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.142.120 50.78 MiB 256 ? 78f58a89-30b3-4646-8762-f8ed528301a9 rack1
UN 10.0.142.121 126.94 MiB 256 ? 7229079d-12c9-4ef6-8753-b79edbcec8cf rack1
UN 10.0.142.122 35.91 MiB 256 ? e3c1e9c3-4bd1-4cd0-8479-69ba1b28676e rack1
UN 10.0.142.123 52.7 MiB 256 ? 25c591fe-36cc-4923-82bc-c0944364b486 rack1
Datacenter: dc2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.142.116 2.33 GiB 256 ? 9aea87a8-be95-45ad-a9c8-91ad7e658dff rack2
UN 10.0.142.117 2.64 GiB 256 ? 51e078ec-2434-401c-9db8-4adaaf263ed4 rack1
UN 10.0.142.118 2.76 GiB 256 ? 1a0ccb08-c65d-40ce-ae99-acb30f6e9d9a rack1
UN 10.0.142.119 2.67 GiB 256 ? d29708ca-729d-4727-a816-129b4fc72e04 rack1
Datacenter: trans
=================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.142.113 164.09 MiB 256 ? 8fd44e60-9b84-44af-aebf-26b3c6a3ab17 rack1
UN 10.0.142.114 130.3 MiB 256 ? 8eb9a807-ff1f-47d7-92f0-e876f0fb56ee rack2
UN 10.0.142.115 113.3 MiB 256 ? e5f9a5d7-03b7-406e-ab0e-e429301af543 rack1
# cqlsh -u cassandra -p cassandra 10.0.142.120
Connected to GDT2 Cluster at 10.0.142.120:9042.
cassandra#cqlsh> list ROLES ;
NoHostAvailable:
# cqlsh -u cassandra -p cassandra 10.0.142.115
Connected to GDT2 Cluster at 10.0.142.115:9042.
cassandra#cqlsh> LIST ROLES ;
role | super | login | options
-----------+-------+-------+---------
cassandra | True | True | {}
sysadmin | True | True | {}
test | False | True | {}
(3 rows)
Describe your system_auth keyspace. Does it look like this?
> desc KEYSPACE system_auth ;
CREATE KEYSPACE system_auth WITH
replication = {'class': 'NetworkTopologyStrategy', 'trans': '1'}
AND durable_writes = true;
Or even worse, does it look like this?
CREATE KEYSPACE system_auth WITH
replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}
AND durable_writes = true;
The problem here, is that it seems that the trans DC is the only DC which has the replica(s) for the system_auth.roles table. To fix that, I recommend this:
ALTER KEYSPACE system_auth WITH
replication = {'class': 'NetworkTopologyStrategy', 'trans': '3',
'analytic': '3', 'dc2': '3'};
That will instruct all future writes to tables in the system_auth keyspace to write 3 replicas to each logical data center. Once that command is complete, run the following to ensure that all current replicas are moved to their proper places:
> consistency ALL
> SELECT * FROM resource_role_permissons_index ;
> SELECT * FROM role_permissions ;
> SELECT * FROM role_members ;
> SELECT * FROM roles;
The first command forces your query consistency level up to ALL, requiring a response from all replicas for success. That sounds like something you wouldn't want to do in Cassandra (and usually, you wouldn't). But reading at consistency ALL forces a read repair 100% of the time. The SELECT queries read all replicas and actually forces the repair. That will essentially trick Cassandra into repairing the replicas for you.
Also:
cqlsh -u cassandra -p cassandra
Don't use the default cassandra/cassandra user. Changing the password and/or disabling that account is one of the first things that should be done.
I tested throughput performance of Cassandra cluster with 2,3 and 4 nodes. There was significant improvement when I used 3 nodes(as compared to 2), however, the improvement wasn't so significant when I used 4 nodes, instead of 3.
Given below are specs of the 4 nodes:
N->No. of physical CPU cores, Ra->Total RAM, Rf->Free RAM
Node 1: N=16, Ra=189 GB, Rf=165 GB
Node 2: N=16, Ra=62 GB, Rf=44 GB
Node 3: N=12, Ra=24 GB, Rf=38 GB
Node 4: N=16, Ra=189 GB, Rf=24 GB
All nodes are on RHEL 6.5
Case 1(2 nodes in the cluster, Node 1 and Node 2)
Throughput: 12K ops/second
Case 2(3 nodes in the cluster, Node 1, Node 2 and Node 3)
Throughput: 20K ops/second
Case 3(All 4 nodes in the cluster)
Throughput: 23K ops/second
1 operation involved 1 read + 1 write(Read/write takes place on the same row)(Row cache can't be used). In all cases, Read consistency =2 and Write Consistency =1. Both read and write were asynchronous. The client application used Datastax's C++ driver and was being run with 10 threads.
Given below are the keyspace and table details:
CREATE KEYSPACE cass WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '2'} AND durable_writes = true;
CREATE TABLE cass.test_table (
pk text PRIMARY KEY,
data1_upd int,
id1 int,
portid blob,
im text,
isflag int,
ms text,
data2 int,
rtdata blob,
rtdynamic blob,
rtloc blob,
rttdd blob,
rtaddress blob,
status int,
time bigint
) WITH bloom_filter_fp_chance = 0.001
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
Some parameters from YAML are given below(All 4 nodes used similar YAML files):
commitlog_segment_size_in_mb: 32
concurrent_reads: 64
concurrent_writes: 256
concurrent_counter_writes: 32
memtable_offheap_space_in_mb: 20480
memtable_allocation_type: offheap_objects
memtable_flush_writers: 1
concurrent_compactors: 2
Some parameters from jvm.options are given below(all nodes used same values):
### CMS Settings
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=4
-XX:MaxTenuringThreshold=6
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSWaitDuration=10000
-XX:+CMSParallelInitialMarkEnabled
-XX:+CMSEdenChunksRecordAlways
-XX:+CMSClassUnloadingEnabled
Given below are some client's connection specific parameters:
cass_cluster_set_max_connections_per_host ( ms_cluster, 20 );
cass_cluster_set_queue_size_io ( ms_cluster, 102400*1024 );
cass_cluster_set_pending_requests_low_water_mark(ms_cluster, 50000);
cass_cluster_set_pending_requests_high_water_mark(ms_cluster, 100000);
cass_cluster_set_write_bytes_low_water_mark(ms_cluster, 100000 * 2024);
cass_cluster_set_write_bytes_high_water_mark(ms_cluster, 100000 * 2024);
cass_cluster_set_max_requests_per_flush(ms_cluster, 10000);
cass_cluster_set_request_timeout ( ms_cluster, 12000 );
cass_cluster_set_connect_timeout (ms_cluster, 60000);
cass_cluster_set_core_connections_per_host(ms_cluster,1);
cass_cluster_set_num_threads_io(ms_cluster,10);
cass_cluster_set_connection_heartbeat_interval(ms_cluster, 60);
cass_cluster_set_connection_idle_timeout(ms_cluster, 120);
Is there anything wrong with the configurations due to which Cassandra didn't scale much when number of nodes were increased from 3 to 4?
During a test, you may check ThreadPools using nodetool tpstats.
You will be able to see if some stages have too many pending (or blocked) tasks.
If there are no issues with ThreadPools, may be you cloud launch a benchmark using cassandra-stress in order to see if the limitation comes from your client?
I don't know if it is only for test purpose but as far as I know, Read before Write data is an antipattern with Cassandra.
I already created one keyspace on DC.
Create query for tradebees_dev keyspace :- (This keyspace is working fine.)
CREATE KEYSPACE tradebees_dev WITH replication = {'class': 'NetworkTopologyStrategy', 'solr': '3'} AND durable_writes = true;
Status is below :-
nodetool status tradebees_dev
Datacenter: Solr
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 1.09 GB 256 100.0% e754d239-8370-4e1d-82c8-dce3d401f972 rack1
UN 127.0.0.2 1.19 GB 256 100.0% 70b74025-1881-4756-a0c8-a1ec5e57d407 rack1
UN 127.0.0.3 1.53 GB 256 100.0% 3ba4bfe4-c894-4cd1-a684-f0f20edac78f rack1
After that I created another keyspace on same DC with same replication factor .
Create query for crawl_dev keyspace :-
CREATE KEYSPACE crawl_dev WITH replication = {'class': 'NetworkTopologyStrategy', 'solr': '3'} AND durable_writes = true;
nodetool status crawl_dev
Datacenter: Solr
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 1.09 GB 256 0.0% e754d239-8370-4e1d-82c8-dce3d401f972 rack1
UN 127.0.0.2 1.19 GB 256 0.0% 70b74025-1881-4756-a0c8-a1ec5e57d407 rack1
UN 127.0.0.3 1.53 GB 256 0.0% 3ba4bfe4-c894-4cd1-a684-f0f20edac78f rack1
As first keyspace is working fine but when I am trying to do select query on second keyspace i.e on crawl_dev , I am getting below error message.
Traceback (most recent call last):
File "/usr/share/dse/resources/cassandra/bin/cqlsh", line 1124, in perform_simple_statement
rows = self.session.execute(statement, trace=self.tracing_enabled)
File "/usr/share/dse/resources/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2-5d33cb4.zip/cassandra-driver-2.7.2-5d33cb4/cassandra/cluster.py", line 1602, in execute
result = future.result()
File "/usr/share/dse/resources/cassandra/bin/../lib/cassandra-driver-internal-only-2.7.2-5d33cb4.zip/cassandra-driver-2.7.2-5d33cb4/cassandra/cluster.py", line 3347, in result
raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}
Please suggest me how to resolve this issue. and also let me know can we create two keyspaces on same DC . YES or NO.
After some research , I got info then I checked
/etc/dse/cassandra/cassandra-rackdc.properties
in this file , dc=DC1 and rc=RACK1 is given.
Thanks.
The datacenter name in the "create keyspace" command is case-sensitive, so instead of:
CREATE KEYSPACE tradebees_dev WITH replication = {'class': 'NetworkTopologyStrategy', 'solr': '3'} AND durable_writes = true;
you want to capitalize Solr, for example:
CREATE KEYSPACE tradebees_dev WITH replication = {'class': 'NetworkTopologyStrategy', 'Solr': '3'} AND durable_writes = true;
You're on the right track for troubleshooting with your "nodetool status [keyspace]" commands. Notice in your result for tradebees_dev, each node reports 100% in the Owns column, which makes sense because you have RF 3 on a 3-node cluster. Then notice for crawl_dev it shows 0% which means no nodes own that data, and hence the error you received. In your example above I suspect you did create tradebees_dev with capital "Solr" in the replication factory and that's why it worked.
I don't see why you should not be able to create multiple keyspaces on the same DC. Indeed, you already have multiple keyspaces in the cluster:
cqlsh> DESCRIBE keyspaces;
system_traces system_schema system_auth system system_distributed
We had a two nodes cassandra cluster which we want to expand to four.
We followed the procedure described there: http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_add_node_to_cluster_t.html
But after adding two more nodes (at same time, with a 2 minutes interval as recommended in the documentation), we experienced some data loss. In some column families, there was missing elements.
Here is the nodetool netstats:
[centos#ip-10-11-11-187 ~]$ nodetool status
Datacenter: us-west-2
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.11.11.187 5.63 MB 256 ? 0e596912-d649-4eed-82a4-df800c422634 2c
UN 10.11.1.104 748.79 MB 256 ? d8b96739-0858-4926-9eb2-27c96ca0a1c4 2c
UN 10.11.11.24 7.11 MB 256 ? e3e76dcf-2c39-42e5-a34e-9e986d4a9f7c 2c
UN 10.11.1.231 878.91 MB 256 ? cc1b5cfd-c9d0-4ca9-bbb1-bce4b2deffc1 2c
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
I don't quite understand if the "Note" is bad or not.
When we added the nodes, we put the two first servers - already available in the cluster - in the seeds of the configuration of the first added node. For the second added node, we put the newly added node in the seeds as well.
We are using EC2Snitch, and the listen_address has been set to the above addresses on each server.
We didn't run the cleanup yet, but we tried to run a repair, and there was written that nothing were to be repaired in our keycap.
Here is how our cluster was created:
CREATE KEYSPACE keyspace_name WITH replication = {'class': 'NetworkTopologyStrategy', 'us-west-2': '1'} AND durable_writes = true;
And the options of all of our tables:
CREATE TABLE keyspace_name."CFName" (
// ...
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
The data reappears if I decommission the new nodes.
EDIT: It was actually an error when reading the documentation... A colleague did set auto_bootstrap to false instead of setting it to true...
You should perform nodetool rebuild on the new nodes after you add them with auto_bootstrap: false
http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsRebuild.html
HTH
Well you can specify the keyspace name to remove the Node which in this case is
nodetool status keyspace_name