Below mentioned is my Cassandra set up:
192.168.80.115 (Seed)
192..168.80.116 (Node1)
192..168.80.117 (Node2)
192..168.80.118 (Node3)
I have installed the Cassandra in all the nodes
below mentioned is the configuration of the cassandra.yaml
cassandra.yaml - 192.168.80.115 (Seed)
======================================
cluster_name: 'Test Cluster'
initial_token: 0
seed_provider:
- seeds: "192.168.80.115"
listen_address: 192.168.80.115
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
cassandra.yaml - 192.168.80.116 (Node1)
========================================
cluster_name: 'Test Cluster'
initial_token: -4611686018427387904
seed_provider:
- seeds: "192.168.80.115"
listen_address: 192.168.80.116
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
cassandra.yaml - 192.168.80.117 (Node2)
========================================
cluster_name: 'Test Cluster'
initial_token: 4611686018427387904
seed_provider:
- seeds: "192.168.80.115"
listen_address: 192.168.80.117
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
cassandra.yaml - 192.168.80.118 (Node3)
========================================
cluster_name: 'Test Cluster'
initial_token: -9223372036854775808
seed_provider:
- seeds: "192.168.80.115"
listen_address: 192.168.80.118
rpc_address: 0.0.0.0
endpoint_snitch: RackInferringSnitch
When I start Cassandra and check the node status in the seed I am able to see only my localhost and not other nodes
192.168.80.115 Seed
Datacenter: datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 96.98 KB 256 ? deb3f882-1886-4e09-be08-21ebd7e99c8d rack1
And status in the nodes are as bellow
192.168.80.116 Node 1
Datacenter: datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 116.33 KB 256 ? 21a310ee-5652-4179-8b59-d1c7dcd538ee rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
192.168.80.117 Node 2
Datacenter: datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 156.95 KB 256 ? b64a5a17-21d0-4cf6-bbd6-1eff8d8feb93 rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
192.168.80.118 Node 3
Datacenter: datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 157.04 KB 256 ? 43a24dea-26cc-4094-9da7-2b26b8c5f7e5 rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
Why I am not able to see worker nodes in seed node status
Related
I have a 3 node cassandra (3.0.15) cluster, where nodes seem to have partitioned. nodetool status and nodetool describecluster outputs are inconsistent when seen from different nodes. Gossip information (nodetool gossipinfo) seems to be up to date on all nodes. No errors in the logs other than read timeouts which seems to be due from the partitioning.
Attempts to resolve this that didn't work:
Rolling restart
Full cluster restart
disable/enable gossip on each node
Node1 (192.168.2.247):
$ nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
59ffe9aa-aca7-34b3-8c5e-b736d221b922: [192.168.2.248, 192.168.2.247]
UNREACHABLE: [192.168.2.249]
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.2.248 210.23 GB 256 67.7% f6593c49-6739-4b8c-a8d9-8321915a660d rack1
DN 192.168.2.249 190.9 GB 256 68.6% 6ac211c2-bab1-4e2b-bc84-18d911e005d0 rack1
UN 192.168.2.247 157.82 GB 256 63.7% eb462857-cbfb-4e41-8be9-5d241d273b81 rack1
Node2 (192.168.2.248):
$ nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
59ffe9aa-aca7-34b3-8c5e-b736d221b922: [192.168.2.248, 192.168.2.249, 192.168.2.247]
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.2.248 210.24 GB 256 67.7% f6593c49-6739-4b8c-a8d9-8321915a660d rack1
UN 192.168.2.249 190.9 GB 256 68.6% 6ac211c2-bab1-4e2b-bc84-18d911e005d0 rack1
UN 192.168.2.247 157.82 GB 256 63.7% eb462857-cbfb-4e41-8be9-5d241d273b81 rack1
Node3 (192.168.2.249):
$ nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
59ffe9aa-aca7-34b3-8c5e-b736d221b922: [192.168.2.248]
UNREACHABLE: [192.168.2.249, 192.168.2.247]
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.2.248 210.24 GB 256 67.7% f6593c49-6739-4b8c-a8d9-8321915a660d rack1
UN 192.168.2.249 190.92 GB 256 68.6% 6ac211c2-bab1-4e2b-bc84-18d911e005d0 rack1
UN 192.168.2.247 157.84 GB 256 63.7% eb462857-cbfb-4e41-8be9-5d241d273b81 rack1
I am keep getting below error when I tried to run single node or multinode Cassandra cluster.
Single node cluster with default config works fine, however nodetool staus shows 127.0.0.1 as IP Address.
After changing listen_address: 192.168.1.143 (this is my ip address) on cassandra.yaml file I am getting below error.
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers
java.lang.RuntimeException: Unable to gossip with any peers
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1443)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:547)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:804)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:664)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:379)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
Well, after trying different approaches finally I was able to resolve it and able to run single and 3 node cluster.
Below are the configuration changes you need to make on cassandra.yaml file
First Node
--------------
listen_address: 192.168.1.143 (This should be your server/node IP)
seeds: "192.168.1.143" (For your first node please mention your node IP address)
Second Node
---------------
listen_address: 192.168.1.144 (This should be your server/node IP)
seeds: "192.168.1.143" (specify your first node IP, additionally, you can also mention current IP address ,192.168.1.144)
Third Node
---------------
listen_address: 192.168.1.145 (This should be your server/node IP)
seeds: "192.168.1.143" (specify your first/second node IP, additionally, you can also mention current IP address ,192.168.1.145)
After starting cassandra on all 3 servers, nodetool status returned the following
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.143 258.83 KiB 256 100.0% 7b3a0644-c8dd-4a47-9186-0237f3725941 rack1
UN 192.168.1.144 309.71 KiB 256 100.0% e7a11a60-d795-47ee-8d21-7cc21b4cbdca rack1
UN 192.168.1.145 309.71 KiB 256 100.0% b2a4545a-f279-r5h7-2fy6-23dk8fg5c8kq rack1
Hope this helps!!
Yes, for joining cassandra cluster first time. you should start seed node first then other nodes.
I have deployed Cassandra cluster(Datastax) in google cloud in multiple regions. When I deployed i checked OpsCenter, it was connected and able to see information. I have restarted (Shutdown and started) the cassandra VM's and OpsCenter, then I checked the OpsCenter. It is showing it is not connected to Cluster.
Check the OpsCenter log file and found the following error message.
Error Message
[opscenterd] ERROR: Unhandled error in Deferred: There are no clusters with name or ID 'Test_Cluster'
nodetool status
Datacenter: asia-east1-a
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.140.0.2 31.07 MB 64 ? d52bfd45-26e3-4c22-9369-caf5f7aa567a asia-east1-a
UN 10.140.0.3 31.05 MB 64 ? 35da9f1a-b897-4236-a735-366a6f4e5fa2 asia-east1-a
Datacenter: europe-west1-b
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.132.0.2 212.07 KB 64 ? 0e29e8a5-a94e-4843-8515-4ed1ad751009 europe-west1-b
UN 10.132.0.3 195.63 KB 64 ? e1b130ad-52e2-4450-88ef-aa7032d93979 europe-west1-b
Datacenter: us-east1-b
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.142.0.2 295.4 KB 64 ? 032df12d-f45a-4d57-b6fc-aa4c3f516212 us-east1-b
UN 10.142.0.3 214.28 KB 64 ? 714242e5-4c66-4e90-8816-9328c0c311ff us-east1-b
Checked Datastax agent
sudo service datastax-agent status
DataStax Agent datastax-agent is running
Faced same issue when I tried to deploy new cluster and shutdown and restarted.
I have deployed a datastax cassandra cluster on google cloud. The cluster size is 4 nodes on two different datacenters.
Servers are running fine and able to view opscenter but no data on it and getting the following errors.
This is a fresh deployment and all nodes are up and running. No tables were created yet.
Error Message from opscenter log:
[opscenterd] ERROR: Unhandled error in Deferred: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)
address.yaml from one of the node:
stomp_interface: 10.128.x.x [ ops center ip ]
agent_rpc_interface: 10.132.x.x [ one of the node ip]
agent_rpc_broadcast_address: 10.132.x.x
**cassandra.yaml from one of the node:**
cluster_name: 'Test Cluster'
num_tokens: 64
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /var/lib/cassandra/hints
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: com.datastax.bdp.cassandra.auth.DseRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /mnt/data
commitlog_directory: /mnt/commitlog
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /mnt/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- **seeds: "XX.XXX.X.X" [ The node IP from different Data Center]**
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: true
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: **SAME NODE IP**
broadcast_address: **SAME NODE IP**
start_native_transport: true
native_transport_port: 9042
start_rpc: true
rpc_address: 0.0.0.0
rpc_port: 9160
broadcast_rpc_address: SAME NODE IP
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
Opscenter.conf from opscenter:
[webserver]
port = 8888
interface = 0.0.0.0
[authentication]
enabled = False
[stat_reporter]
nodetool status:
Datacenter: europe-west1-b
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.XXX.0.2 220 KB 64 ? 5381a371-c532-4be7-a7f5-80162bef4541 europe-west1-b
UN 10.XXX.0.3 148.6 KB 64 ? e5b3dfec-f9bf-4952-b48f-b5a82749a9fe europe-west1-b
Datacenter: us-east1-b
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.XXX.0.2 158.72 KB 64 ? b93980e0-3c28-4bb0-9e49-220cb009d44c us-east1-b
UN 10.XXX.0.3 192.85 KB 64 ? af242b71-b7b4-4345-9888-5b89eb5e6199 us-east1-b
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
You have a DC in your replication factor ($dc) that doesnt exist, so when OpsCenter sends a request to read data the coordinator responds with an UnavailableException.
Try updating your keyspace like ALTER KEYSPACE "OpsCenter" WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east1-b': '2'} and doing a full repair if you want to keep the data thats there.
If it hasnt run yet so you dont have data you want to save, just delete the opscenter keyspace (drop keyspace "OpsCenter") and restart opscenterd and the agents (after opscenter service has started back up).
I am having trouble with three nodes in Cassandra, each of them in an individual computer, as I am trying to set up my first Cassandra structure. I have set up everything as in the Datastax documentation, and I have the same configuration in the different cassandra.yaml of each machine (changing the relative ips). The thing is that after configuring everything, each computer sees each other as DN, and each machine (localhost) appears as UN, with the difference that in the .101 computer I can see two different datacenters, while in the other computers only one datacenter appears.
So in my 192.168.1.101 machine when I type
sudo nodetool status
I get this output:
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
DN 192.168.1.200 ? 256 ? 968d5d1e-a113-40ce-9521-e392a927ea5e r1
DN 192.168.1.102 ? 256 ? fc5c2dbe-8834-4040-9e77-c3d8199b6767 r1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 446.13 KB 256 ? 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
While when I type "nodetool status" in one of the other two machines, I get this output:
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
DN 192.168.1.200 ? 256 ? 968d5d1e-a113-40ce-9521-e392a927ea5e rack1
UN 127.0.0.1 506,04 KB 256 ? fc5c2dbe-8834-4040-9e77-c3d8199b6767 rack1
DN 192.168.1.101 ? 256 ? 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
In OpsCenter I can only see my 192.168.1.101 machine:
... Which makes me think that something's odd in the yaml file of this machine and the others, but I have checked several times and it seems that the configuration is the same in the other computers. Enpoint_snitch is set to "GossipingPropertyFileSnitch".
Any tips on how to solve the reason why all the other nodes appear as Down Normal and why I am getting two datacenters would be highly appreaciated. It's driving me crazy!
Thanks for reading.
Any tips on how to solve the reason why all the other nodes appear as Down Normal and why I am getting two datacenters would be highly appreaciated. It's driving me crazy!
On each machine, edit the $CASSANDRA_HOME/conf/cassandra-rackdc.properties file to set:
dc=dc1
rack=rack1
nodetool status shows that you set the wrong DC name for 2 nodes (DC1 instead of dc1). It's case sensitive
It looks like some of the installed nodes were dead, so I deleted the nodes that were not the local machine in each of the nodes, ie:
nodetool removenode 968d5d1e-a113-40ce-9521-e392a927ea5e
nodetool removenode fc5c2dbe-8834-4040-9e77-c3d8199b6767
and after that I got the right output when I executed nodetools status
[machine1]~$ sudo nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 286.93 KB 256 ? 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
[machine2]~$ sudo nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 268.45 KB 256 ? fc5c2dbe-8834-4040-9e77-c3d8199b6767 rack1
And made sure that the parameters cluster_name, seeds, listen_address and rpc_address were right.
cluster_name: 'Test Cluster'
seeds: "192.168.1.101, 192.168.1.102"
listen_address: 192.168.1.101
rpc_address: 192.168.1.101
Changing listen_address and rpc_address to the corresponding ip of each machine in their corresponding cassandra.yaml file.
After that I got the right output (I am using only 2 machines for the nodes now):
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.101 309.13 KB 256 51.6% 6d28d540-2b44-4522-8612-b5f70a3d7d52 rack1
UN 192.168.1.102 257.15 KB 256 48.4% fc5c2dbe-8834-4040-9e77-c3d8199b6767 rack1