[DSE6.8, Opscenter6.7] When building a DSE cluster in LCM, only one node is recognized by Opscenter if require_endpoint_verification: is set to true.
I suspect that this is due to the IP address not being added correctly.
Do you know why?
(The only other setting was DSE Unified Authentication)
server_encryption_options:
internode_encryption: all
keystore_type: JKS
keystore: path_to_keystore.jks
keystore_password: keystore_password
require_client_auth: true
require_endpoint_verification: true
truststore_type: JKS
truststore: path_to_truststore.jks
truststore_password: truststore_password
Related
We currently run a multi region cassandra cluster in AWS. It runs in four regions, 12 nodes per region. It runs without node to node encryption (or client encryption either). We are trying to enable inter datacenter node to node encryption. However, when we flip encryption over we get an exception that nodes are unable to gossip with any peers.
It could possibly be that we didn't build our jks keystore/truststores correctly (more on how we built these files below). But, we additionally do not see intra datacenter communication working (which should be set to unencrypted communication). Additionally, cqlsh cannot connect to the node either; even though we have (by default) client_auth_required set to false.
ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO [main] 2019-08-15 18:47:07,384 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
Something to note is that this error message occurs after a few minutes of the node being up. (i.e. there is a delay between start up before this exception is thrown).
Information about our cassandra setup
cassandra version: 3.11.4
JDK version: openjdk-8.
Linux: Ubuntu 18.04 (bionic).
cassandra.yaml
endpoint_snitch: Ec2MultiRegionSnitch
server_encryption_options:
internode_encryption: dc
keystore: <omitted>
keystore_password: <omitted>
truststore: <omitted>
truststore_password: <omitted>
client_encryption_options:
enabled: false
cassandra-rackdc.properties
prefer_local=true
No obvious errors with SSH output
When starting cassandra with JVM_OPTS="$JVM_OPTS -Djavax.net.debug=ssl" added to cassandra-env.sh we see SSL logs printed to stdout (Note: Subject and Issuer were omitted on purpose).
found key for : cassy-us-west-2
adding as trusted cert:
Subject: ...
Issuer: ...
Algorithm: RSA; Serial number: 0xdad28d843fc73325d4c1a75207d4e74
Valid from Fri May 27 00:00:00 UTC 2016 until Tue May 26 23:59:59 UTC 2026
...
trigger seeding of SecureRandom
done seeding SecureRandom
Looking at Java SE SSL/TLS connection debugging, this looks correct. But to note, we see this series of messages (along with the RSA key signature output) repeated several times in rapid fire. We never observe any messages about the trust store being added; however that might be something that occurs only on client initiation (?)
Additionally, we do see cassandra report that the Encrypted Messaging service has been started.
INFO [main] 2019-08-15 18:45:31,022 MessagingService.java:704 - Starting Encrypted Messaging Service on SSL port 7001
Doesn't appear to be a cassandra.yaml configuration problem
We can bring the node back online by simply configuring internode_encryption: none. This action seems to rule out a broadcast_address or rpc_address configuration problem.
How we built our keystore/truststores
We followed the basic template datastax docs for preparing SSL certificates. One minor difference was that our private key and CSRs were generated using openssl. One per each region (we plan to share key/signed certs across nodes in regions). This was created using a command template as:
openssl req -new -newkey rsa:2048 -out cassy-<region>.csr -keyout cassy-<region>.key -config cassy-<region>.conf -subj "..." -nodes -sha256
The generated CSR was then signed by an internal root CA. Because we generated our files using openssl, we had to build our jks files by importing our certs into them.
Commands to generate truststore
We distribute this one file to all nodes.
keytool -importcert
-keystore generic-server-truststore.jks
-alias rootCa
-file rootCa.crt
-noprompt
-keypass omitted
-storepass omitted
Commands to generate keystore
This was done one per region; but essentially we created a keystore with keytool, then deleted the key entry and then imported our key entry using keytool from a pkcs12 file.
keytool -genkeypair -keyalg RSA -alias cassy-${region} -keystore cassy-${region}.jks -storepass omitted -keypass omitted -validity 365 -keysize 2048 -dname "..."
keytool -delete -alias cassy-${region} -keystore cassy-${region}.jks -storepass omitted
openssl pkcs12 -export -in signed_certs/${region}.pem -inkey keys/cassandra.${region}.key -name cassy-${region} -out ${region}.p12
keytool -importkeystore -deststorepass omitted -destkeystore cassy-${region}.jks -srckeystore ${region}.p12 -srcstoretype PKCS12
keytool -importcert -keystore cassy-${region}.jks -alias rootCa -file ca.crt -noprompt -keypass omitted -storepass omitted
Looking back at this, I don't remember why we used keytool to generate a keypair/keystore, then deleted and imported. I think it was because the keytool importkeystore command refused to run if the keystore didn't already exist.
ca.crt and pem file
The ca.crt file contains the root certificate and the intermediate certificate that was used to sign the CSR. The pem file contains the signed CSR returned to us, the intermediate cert, and the root CA (in that order).
openssl verify ca.crt and pem
openssl verify -CAfile ca.crt us-west-2.pem
signed_certs/us-west-2.pem: OK
Command output after enabling encryption
nodetool status (output truncated)
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N 52.44.11.221 ? 256 25.4% null 1c
...
?N 52.204.232.195 ? 256 23.2% null 1d
Datacenter: us-west-2
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N 34.209.2.144 ? 256 26.5% null 2c
UN 52.40.32.177 105.99 GiB 256 23.7% null 2c
?N 34.210.109.203 ? 256 24.7% null 2a
...
With the online node being the node with encryption set.
cqlsh to localhost
cassy-node6:~$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
cqlsh to remote node
Remote node is a node with encryption enabled
cassy-node6:~$ cqlsh 10.0.2.7
Connection error: ('Unable to connect to any servers', {'10.0.2.7': error(111, "Tried connecting to [('10.0.2.7', 9042)]. Last error: Connection refused")})
Behavior we expected
We expected for the node to report that the other regions were all down, as they need to be handled over encryption. So that works as expected; however, cqlsh and intra datacenter peers being reported as unreachable is unexpected.
Specifically, we expected the node to still show peer nodes within the same datacenter as up and normal, regardless if there is a cert issue/error. We also expected cqlsh to continue to work.
Lastly, we are also trying to figure out if we have a jks certificate problem.
I have deployed a datastax cassandra cluster on google cloud. The cluster size is 4 nodes on two different datacenters.
Servers are running fine and able to view opscenter but no data on it and getting the following errors.
This is a fresh deployment and all nodes are up and running. No tables were created yet.
Error Message from opscenter log:
[opscenterd] ERROR: Unhandled error in Deferred: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)
address.yaml from one of the node:
stomp_interface: 10.128.x.x [ ops center ip ]
agent_rpc_interface: 10.132.x.x [ one of the node ip]
agent_rpc_broadcast_address: 10.132.x.x
**cassandra.yaml from one of the node:**
cluster_name: 'Test Cluster'
num_tokens: 64
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /var/lib/cassandra/hints
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: com.datastax.bdp.cassandra.auth.DseRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /mnt/data
commitlog_directory: /mnt/commitlog
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /mnt/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- **seeds: "XX.XXX.X.X" [ The node IP from different Data Center]**
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: true
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: **SAME NODE IP**
broadcast_address: **SAME NODE IP**
start_native_transport: true
native_transport_port: 9042
start_rpc: true
rpc_address: 0.0.0.0
rpc_port: 9160
broadcast_rpc_address: SAME NODE IP
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
Opscenter.conf from opscenter:
[webserver]
port = 8888
interface = 0.0.0.0
[authentication]
enabled = False
[stat_reporter]
nodetool status:
Datacenter: europe-west1-b
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.XXX.0.2 220 KB 64 ? 5381a371-c532-4be7-a7f5-80162bef4541 europe-west1-b
UN 10.XXX.0.3 148.6 KB 64 ? e5b3dfec-f9bf-4952-b48f-b5a82749a9fe europe-west1-b
Datacenter: us-east1-b
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.XXX.0.2 158.72 KB 64 ? b93980e0-3c28-4bb0-9e49-220cb009d44c us-east1-b
UN 10.XXX.0.3 192.85 KB 64 ? af242b71-b7b4-4345-9888-5b89eb5e6199 us-east1-b
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
You have a DC in your replication factor ($dc) that doesnt exist, so when OpsCenter sends a request to read data the coordinator responds with an UnavailableException.
Try updating your keyspace like ALTER KEYSPACE "OpsCenter" WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east1-b': '2'} and doing a full repair if you want to keep the data thats there.
If it hasnt run yet so you dont have data you want to save, just delete the opscenter keyspace (drop keyspace "OpsCenter") and restart opscenterd and the agents (after opscenter service has started back up).
I am using mac osx. i created 3 virtual box by virtualbox. I've installed centos7 minimal version on each of the virtual box.
Then i installed cassandra on each of the box. After installation it was starting by cqlsh and nodetool status command.
But after then when i was trying to link each other and edit cassandra.yaml file its started showing
('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
i've edited the cassandra.yaml file as follows:
cluster_name: 'Home Cluster'
num_tokens: 256
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
- seeds: "192.168.56.102,192.168.56.103"
storage_port: 7000
listen_address: 192.168.56.102
rpc_address: 192.168.56.102
rpc_port: 9160
endpoint_snitch: SimpleSnitch
my /etc/hosts file contains:
192.168.56.102 node01
192.168.56.103 node02
192.168.56.104 node03
Please tell me whats wrong i'm doing? My cassandra cluster not working.
solution: I got the solution from AKKI. The problem was enpoint_snitch. I made the endpoint_snitch=GossipingPropertyFileSnitch and it fixed. My now output is as follows:
[root#dbnode2 ~]# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.56.101 107.38 KB 256 62.5% 0526a2e1-e6ce-4bb4-abeb-b9e33f72510a rack1
UN 192.168.56.102 106.85 KB 256 73.0% 0b7b76c2-27e8-490f-8274-571d00e60c20 rack1
UN 192.168.56.103 83.1 KB 256 64.5% 6c8d80ec-adbb-4be1-b255-f7a0b63e95c2 rack1
I had faced similar problem,
I tried the following solution:
In Cassandra.yaml file check if you have,
start_rpc = true
Changed my endpoint snitch to
endpoint_snitch: GossipingFilePropertySnitch
Opened all ports Cassandra uses on my CentOS
Cassandra inter-node ports
Port number Description
7000 Cassandra inter-node cluster communication.
7001 Cassandra SSL inter-node cluster communication.
7199 Cassandra JMX monitoring port.
Cassandra client port
Port number Description
9042 Cassandra client port.
9160 Cassandra client port (Thrift).
Command to open ports on CentOs 7(Find it according to your OS):
>sudo firewall-cmd --zone=public --add-port=9042/tcp --permanent
>sudo firewall-cmd –reload
Then Restart your systems
Also it seems that you are changing the Cassandra.Yaml file after starting cassandra.
Make sure you edit your Cassandra.yaml file on all nodes before starting Cassandra
Also remember to start the seed node first.
I have installed Datastax enterprise 2.8 on my dev VM (Centos 7). The install went through smoothly and the single node cluster is working great. But when I try to connect to the cluster using beeline or hive2 jdbc driver I get an error as shown below. My main aim is to connect Tableau using the Datastax Enterprise driver or Spark Sql driver.
Error observed is:
ERROR 2016-04-14 17:57:56,915
org.apache.thrift.server.TThreadPoolServer: Error occurred during
processing of message. java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException: Invalid status -128
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
~[libthrift-0.9.3.jar:0.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_99]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_99]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_99] Caused by: org.apache.thrift.transport.TTransportException: Invalid status
-128
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
~[libthrift-0.9.3.jar:0.9.3]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
~[libthrift-0.9.3.jar:0.9.3]
... 4 common frames omitted ERROR 2016-04-14 17:58:59,140 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend:
Application has been killed. Reason: Master removed our application:
KILLED
My cassandra.yml config:
cluster_name: 'Cluster1'
num_tokens: 256
hinted_handoff_enabled: true hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /var/lib/cassandra/saved_caches
commitlog_sync: periodic commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "10.33.1.124"
concurrent_reads: 32 concurrent_writes: 32 concurrent_counter_writes:
32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 10.33.1.124
start_native_transport: true native_transport_port: 9042
start_rpc: true
rpc_address: 10.33.1.124
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000 tombstone_failure_threshold: 100000
column_index_size_in_kb: 64
batch_size_warn_threshold_in_kb: 64
compaction_throughput_mb_per_sec: 16
compaction_large_partition_warning_threshold_mb: 100
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000 range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000 counter_write_request_timeout_in_ms:
5000 cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000 request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: com.datastax.bdp.snitch.DseSimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
keystore: resources/dse/conf/.keystore
keystore_password: cassandra
truststore: resources/dse/conf/.truststore
truststore_password: cassandra
client_encryption_options:
enabled: false
optional: false
keystore: resources/dse/conf/.keystore
keystore_password: cassandra
internode_compression: dc
inter_dc_tcp_nodelay: false
concurrent_counter_writes: 32
counter_cache_size_in_mb:
counter_cache_save_period: 7200
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
sstable_preemptive_open_interval_in_mb: 50
counter_write_request_timeout_in_ms: 5000
When connecting with beeline, I get the error:
dse beeline Beeline version 0.12.0.11 by Apache Hive beeline> !connect
jdbc:hive2://10.33.1.124:10000 scan complete in 10ms Connecting to
jdbc:hive2://10.33.1.124:10000 Enter username for
jdbc:hive2://10.33.1.124:10000: cassandra Enter password for
jdbc:hive2://10.33.1.124:10000: ********* Error: Invalid URL:
jdbc:hive2://10.33.1.124:10000 (state=08S01,code=0) 0:
jdbc:hive2://10.33.1.124:10000> !connect
jdbc:hive2://10.33.1.124:10000 Connecting to
jdbc:hive2://10.33.1.124:10000 Enter username for
jdbc:hive2://10.33.1.124:10000: Enter password for
jdbc:hive2://10.33.1.124:10000: Error: Invalid URL:
jdbc:hive2://10.33.1.124:10000 (state=08S01,code=0) 1:
jdbc:hive2://10.33.1.124:10000>
I see similar errors when connecting through Tableau as well.
The JDBC driver connects to the SparkSql Thrift server. If you do not start it, you cannot connect to it.
dse spark-sql-thriftserver
/Users/russellspitzer/dse/bin/dse:
usage: dse spark-sql-thriftserver <command> [Spark SQL Thriftserver Options]
Available commands:
start Start Spark SQL Thriftserver
stop Stops Spark SQL Thriftserver
I am having trouble with a 3-node Cassandra cluster on AWS.
There is one seed node and two data nodes. The nodes are crashing
when they are launched and when I am trying to start them manually.
The error message appears in all three nodes.
Cassandra's version is 2.0.9
I have tried the following settings:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "<seed.node.public.IP>"
rpc_address: <node.public.IP>
rpc_port: 9160
listen_address: (or with the node's public IP)
storage_port: 7000
endpoint_snitch: SimpleSnitch (and RackInferringSnitch as well).
The error message is
ERROR [main] 2014-09-29 08:59:45,241 CassandraDaemon.java (line 513) Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision (StorageService.java:446)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:657)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:504)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
Ports 7000, 7001, 7199, 8080, 9042, 9160, 61620 and 61621 are open within the Cluster's security group.
I have also read and tried the solutions given on the following links:
Cassandra Not Starting Up
Starting cassandra as a service does not work for 2.0.5, sudo cassandra -f works
Apache Cassandra: Unable to gossip with any seeds
Datastax Enterprise is crashing with Unable to gossip with any seeds error
https://github.com/Netflix/Priam/issues/313
Cassandra can not bind to the public IP address in EC2.
Replacing it with the public DNS or the private IP address
in listen_address, rpc_address and seeds.
The public DNS is resolving to the private IP address which is
the eth0 intherface on EC2 instances, where Cassandra is binding.
The working configuration is:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "<seed.node.public.DNS>"
rpc_address: <node.public.DNS>
rpc_port: 9160
listen_address: (or with the node's public DNS)
storage_port: 7000
endpoint_snitch: SimpleSnitch (and RackInferringSnitch as well).