My cluster of size 2 had entered into somewhat inconsistent state. On one node (call it node A) nodetool status was correctly showing 2 nodes. While on another node (call it B) it was showing only one i.e. itself. After several attempts I could not fixed the issue. So I decommissioned node B. But nodetool status on node A was still showing the node B that to in UN state. I had to restart cassandra on node A so that it forget node B.
But this has lead to another problem. I am making new node (call it C) to join the cluster of node A. But that node is taking hours. It's already six hours and I am wondering whether it will successfully join finally.
Looking at debug logs of node C suggest that node B (the decommissioned one) is causing trouble. Logs at node C are constantly showing:
DEBUG [GossipTasks:1] 2017-04-29 12:38:40,004 Gossiper.java:337 - Convicting /10.120.8.53 with status removed - alive false
Nodetool status on node A is showing the node C in joining state as expected.
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UJ 10.120.8.113 1006.97 MiB 256 ? f357d8d0-2379-43d8-8ae5-62224191fb6c rack1
UN 10.120.8.23 5.29 GiB 256 ? 596260a0-785a-435c-a3f3-632f56c5c882 rack1
Load for node C increases in fraction after couple of hours.
I checked whether system.peers contains node B. But the table contains zero rows.
I am using cassandra 3.7.
What's going wrong. What can I do to avoid losing data on node A and still scale the cluster?
Run nodetool netstats on node C and see if there's is a progress going on.
Also review nodetool compactionstats, see amount of compactions pending, and see if it goes down with time.
If the bootstraping failed, try restarting the node.
As an alternative, you can remove node C and add it once again, with auto_bootstrap setting set to false. After the node is up, run nodetool rebuild, and nodetool repair after the process - should be a faster alternative for regular bootstrap.
Related
I have a few questions about Cassandra, waiting for suggestions from experts. Thank you.
I'm using 3 nodes with replication factor 3. And all nodes own %100 of data.
[root#datab ~]# nodetool status mydatabase
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.1 11.83 GiB 8 100.0% c0fb9cfc-b20a-4c65-93b6-8e14107cc411 r1
UN 192.168.1.2 20.13 GiB 8 100.0% dd940c60-9645-45ca-894d-f0af49a7bd83 r1
UN 192.168.1.3 17.27 GiB 8 100.0% 3031587e-5354-4342-9ddc-e5696985fc8c r1
I want to remove node 192.168.1.3 and use this server as separated for testing purpose, I mean I want to keep %100 of data till removing the node.
I tried to decommission, but I couldn't use as separated.
For example, I have a table with 100gb data in it and select all queries returning slower. This 3 nodes running on separated hardware (servers). If I add 10 nodes for each server with docker, it makes queries run faster?
For 3 node, what is the difference between having replication factor 2 and 3? 3 node with replication factor keeps %100 data, but whenever I alter replication factor to 2, data percentages are getting down in few seconds, with factor 2 if I lose one of server, do I lost any data?
Whats the proper step to remove 1 node from the dc1 ?
changing factor to 2 ?
removenode ID ?
or first remove node than change factor ?
Thank you !!
I have added a new node into the cluster and was expecting the data on Cassandra to balance itself across nodes.
node status yields
$ nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
Load of node 2 is just 400KB, we have time series data and query on that. how can I rebalance the load between these clusters?
configuration for both nodes are
cluster_name: 'cluster1'
- seeds: "node1_ip, node2_ip"
num_tokens: 256
endpoint_snitch: GossipingPropertyFileSnitch
auto_bootstrap: false
thank you for your time :)
I have added a new node into the cluster and was expecting the data on Cassandra to balance itself across nodes.
Explicitly setting `auto_bootstrap: false' tells it not to do that.
how can I rebalance the load?
Set your keyspace to a RF of 2.
Run nodetool -h 10.128.0.14 repair.
-Or-
Take the 10.128.0.14 out of the cluster.
Set auto_bootstrap: true (or just remove it).
And start the node up. It should join and stream data.
Pro-tip: With a data footprint of 270GB, you should have been running with more than one node to begin with. It would have been much easier to start with 3 nodes (which is probably the minimum you should be running on).
In 5 node cassandra cluster, 1 node died and we recovered it after few days.
On the recovered node, I ran nodetool repair , but it never seems to end as it never returns to command prompt
Data distribution too doesn't seems to change when i run nodetool status from other nodes.
In System.log of cassandra, I see the below ERROR
ERROR [RepairJobTask:7] 2017-03-31 12:16:07,898
RepairRunnable.java:261 - Repair session
b3b72750-15dd-11e7-b158-5d4916117d35 for range [(-167561
1436141724195,-1654146357749205555] Validation failed in /x.x.x.x (IP
address of the cassandra node where repair is being executed)
I have a nine node Cassandra cluster and everything seems to be working fine, except for two of my servers show each other as DN. All other nodes in the cluster show all nodes as UN. These two show all nodes UN except for each other, where they show each other as DN. There are no errors in the system.log on either server that indicates a problem. All nodes are listed as seed nodes across the cluster. I am able to telnet between the servers on port 7001, so I don't think it is a network issue. We are using Internode Communication Encryption so I wonder if it might be an issue with that?
Related Nodetool Status Snippet on 64.6.220.249:
DN 64.6.220.251 106.19 GB 256 ? e008bc26-5d12-48b5-a381-6a175b085496 Rack1
Related Nodetool Status Snippet on 64.6.220.251:
DN 64.6.220.249 105.31 GB 256 ? 59709c2a-6270-40be-a444-042bdf18873e Rack1
Related Nodetol Status Snippet from another node in the cluster (all nodes show this, except for the two above):
UN 64.6.220.251 106.19 GB 256 ? e008bc26-5d12-48b5-a381-6a175b085496 Rack1
UN 64.6.220.249 105.31 GB 256 ? 59709c2a-6270-40be-a444-042bdf18873e Rack1
GossipInfo ran from 64.6.220.249:
/64.6.220.251
generation:1473238188
heartbeat:12693992
SCHEMA:a7b7f6f4-24ba-3153-90cc-dc8ad2754251
RACK:Rack1
SEVERITY:0.0
RPC_ADDRESS:64.6.220.251
HOST_ID:e008bc26-5d12-48b5-a381-6a175b085496
INTERNAL_IP:64.6.220.251
X_11_PADDING:{"workload":"Cassandra","active":"true"}
LOAD:1.14019618013E11
NET_VERSION:8
DC:Cassandra-ALPHA
RELEASE_VERSION:2.1.5.469
STATUS:NORMAL,-1122920019547920198
GossipInfo ran from 64.6.220.251:
/64.6.220.249
generation:1473237564
heartbeat:12696040
RACK:Rack1
DC:Cassandra-ALPHA
RPC_ADDRESS:64.6.220.249
SCHEMA:a7b7f6f4-24ba-3153-90cc-dc8ad2754251
INTERNAL_IP:64.6.220.249
SEVERITY:0.0
X_11_PADDING:{"workload":"Cassandra","active":"true"}
RELEASE_VERSION:2.1.5.469
NET_VERSION:8
LOAD:1.13072884091E11
HOST_ID:59709c2a-6270-40be-a444-042bdf18873e
STATUS:NORMAL,-1027844444513030305
Nodetool describecluster from 64.6.220.249:
Cluster Information:
Name: Fusion Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
a7b7f6f4-24ba-3153-90cc-dc8ad2754251: [64.6.220.254, 170.75.212.226, 170.75.212.225, 64.6.220.252, 170.75.212.224, 64.6.220.253, 64.6.220.250, 64.6.220.249]
UNREACHABLE: [64.6.220.251]
Nodetool describecluster from 64.6.220.251:
Cluster Information:
Name: Fusion Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
a7b7f6f4-24ba-3153-90cc-dc8ad2754251: [64.6.220.254, 170.75.212.226, 170.75.212.225, 64.6.220.252, 170.75.212.224, 64.6.220.253, 64.6.220.250, 64.6.220.251]
UNREACHABLE: [64.6.220.249]
Can anyone point me in the right direction as to why these two nodes show each other as "DN", even though all other nodes see them as "UN?
I have seen this "mixed" gossip state before. When this happens, typically bouncing the cassandra process on the nodes being reported as "DN" fixes it.
Also, when you see this it's also a good idea to run a nodetool describecluster. You should check the results to ensure that you only have one schema version. If you have multiple schema versions (known as "schema disagreement") it's best to bounce those affected nodes as well.
I'm not entirely sure as to why this happens, but a contributing factor is having too many nodes designated as "seed nodes." When you have too many seed nodes, their gossip states can take longer to get in-sync, and that may lead to this condition.
I have seen this issue too with Cassandra version 2.2.13 following a few nodes being restarted soon after each other (10-15 minutes apart).
What's interesting is that looking at all the nodes in the cluster, using nodetool status, it's clear that the bulk of the nodes had a consensus on which nodes are UP or DN, and only a few nodes didn't have consensus.
My solution was to nodetool status on all nodes, identify the ones with inconsistent views, restart those nodes and tail the cassandra logs on the consistent nodes until you see the restarted (rogue) node is online with an entry like INFO 02:15:31 InetAddress /18.136.19.11 is now UP, and then move onto the next rogue node until all nodes are consistent.
In my case I had 2 nodes on status DN but they were actually UP and functioning.
I refreshed the gossip on both nodes and it solved the problem by running:
nodetool disablegossip && nodetool enablegossip
hope this helps.
I had a 3 node cassandra cluster with replication factor of 2. The nodes were running either dsc1.2.3 or dsc1.2.4. Each node had num_token value of 256 and initial_token was commented. This 3 node cluster was perfectly balanced i.e. each owned around 30% of the data.
One of the nodes crashed so I started a new node and nodetool removed the node that had crashed. The new node got added to the cluster but the two older nodes have most of the data now (47.0% and 52.3%) and the new node has just 0.7% of the data.
The output of nodetool status is
Datacenter: xx-xxxx
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.xxx.xxx.xxx 649.78 MB 256 47.0% ba3534b3-3d9f-4db7-844d-39a8f98618f1 1c
UN 10.xxx.xxx.xxx 643.11 MB 256 52.3% 562f7c3f-986a-4ba6-bfda-22a10e384960 1a
UN 10.xxx.xxx.xxx 6.84 MB 256 0.7% 5ba6aff7-79d2-4d62-b5b0-c5c67f1e1791 1c
How do i balance this cluster?
You didn't mention running a repair on the new node, if indeed you haven't yet done that it's likely the cause of your lack of data on the new node.
Until you run a nodetool repair the new node will only hold the new data that gets written to it or the data that read-repair pulls in. With vnodes you generally shouldn't need to re-balance, if I'm understanding vnodes correctly, but I haven't personally yet moved to using vnodes so I may be wrong about that.
It looks like your new node hasn't bootstrapped. Did you add auto_bootstrap=true to your cassandra.yaml?
If you don't want to bootstrap, you can run nodetool repair on the new node and then nodetool cleanup on the two others until the distribution is fair.