sstableloader does not transmit the data, and refer to the weirf ports - cassandra

I want to bulkload my cassandra data from node A to node B.
when I set the 'listen_address' of each cassandra.yaml file to localhost,
they do not show error on console but the data is never transmitted.
when I set each node's listen address to their own local network[eth1 ipv4]address (192.168....), I get the following error.
I can read from this error log that the application is trying to access to port 1..4
and I do not have no idea what on earth is going on.
each node is on the virtual machine on the Virtual Box Hypervisor. Both OS is centOS.
[vagrant#localhost conf]$ ../bin/sstableloader -v -d 192.168.33.12 -p 9160 /db/data/m
oomin/hoahoa2/
Streaming revelant part of /db/data/moomin/hoahoa2/moomin-hoahoa2-hf-69-Data.db to [/192.168.33.12]
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:55:42,655 Failed attempt 1 to connect to /192.168.33.12 to stream /db/data/moomin/hoahoa2/moomin-hoahoa2-h
f-69-Data.db sections=1 progress=0/378000000 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:55:46,658 Failed attempt 2 to connect to /192.168.33.12 to stream /db/data/moomin/hoahoa2/moomin-hoahoa2-h
f-69-Data.db sections=1 progress=0/378000000 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:55:54,666 Failed attempt 3 to connect to /192.168.33.12 to stream /db/data/moomin/hoahoa2/moomin-hoahoa2-h
f-69-Data.db sections=1 progress=0/378000000 - 0%. Retrying in 16000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
progress: [/192.168.33.12 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)]
Here is my cassandra.yaml (the cassandra.yaml of target file is also configured the same way)
# communicate!
#
# Leaving it blank leaves it up to InetAddress.getLocalHost(). This
# will always do the Right Thing *if* the node is properly configured
# (hostname, name resolution, etc), and the Right Thing is to use the
# address associated with the hostname (it might not be).
#
# Setting this to 0.0.0.0 is always wrong.
listen_address: 192.168.33.12
#listen_address: localhost
rpc_address: 0.0.0.0
# port for Thrift to listen for clients on
rpc_port: 9160
# enable or disable keepalive on rpc connections
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compaction: false
compaction_throughput_mb_per_sec: 16
compaction_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.PropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
emory usage without a impact on performance.
index_interval: 128
Can anybody give me advice? I am really suffering as hell.

Related

why is cassandra not getting connected to server?

I have installed cassandra and worked on it. It worked properly. Now, it is showing as-
localhost/<> is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
Fatal configuration error; unable to start server. See log for stacktrace.
INFO 09:17:02 Announcing shutdown
INFO 09:17:02 Compacted 4 sstables to [./../data/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-33,]. 6,485 bytes to 5,751 (~88% of original) in 223ms = 0.024595MB/s. 4 total partitions merged to 1. Partition merge counts were {4:1, }
INFO 09:17:04 Waiting for messaging service to quiesce
user#inblrlt-user:~/dev/Cassandra/apache-cassandra-2.1.7/bin$ ./cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': OperationTimedOut('errors=None, last_host=None',)})
How to change my server address so that the issue is cleared?
Your localhost is already in use. Follow the following steps-
$ jps
You see some processes running. For example:
9107 Jps
1112 CassandraDaemon
Then kill the CassandraDaemon process by the process id you see after executing jps. In my example, here process id 1112 for CassandraDaemon.
$ kill -9 1112
Then check processes again after a while-
$ jps
You will see CassandraDaemon will no longer be available.
9170 Jps
Then remove your saved_caches and commilog and start cassandra again.
If you want to change the listen_address from localhost to any private ip or public ip, you need to make the following changes:
change seeds: at cassandra.yaml
change listen_address: at cassandra.yaml
change rpc_address: at cassandra.yaml
set JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<place_your_ip_here>" at cassandra-env.sh

Connecting opscenter but no data

I have deployed a datastax cassandra cluster on google cloud. The cluster size is 4 nodes on two different datacenters.
Servers are running fine and able to view opscenter but no data on it and getting the following errors.
This is a fresh deployment and all nodes are up and running. No tables were created yet.
Error Message from opscenter log:
[opscenterd] ERROR: Unhandled error in Deferred: com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)
address.yaml from one of the node:
stomp_interface: 10.128.x.x [ ops center ip ]
agent_rpc_interface: 10.132.x.x [ one of the node ip]
agent_rpc_broadcast_address: 10.132.x.x
**cassandra.yaml from one of the node:**
cluster_name: 'Test Cluster'
num_tokens: 64
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /var/lib/cassandra/hints
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: com.datastax.bdp.cassandra.auth.DseRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /mnt/data
commitlog_directory: /mnt/commitlog
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /mnt/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- **seeds: "XX.XXX.X.X" [ The node IP from different Data Center]**
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: true
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: **SAME NODE IP**
broadcast_address: **SAME NODE IP**
start_native_transport: true
native_transport_port: 9042
start_rpc: true
rpc_address: 0.0.0.0
rpc_port: 9160
broadcast_rpc_address: SAME NODE IP
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
Opscenter.conf from opscenter:
[webserver]
port = 8888
interface = 0.0.0.0
[authentication]
enabled = False
[stat_reporter]
nodetool status:
Datacenter: europe-west1-b
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.XXX.0.2 220 KB 64 ? 5381a371-c532-4be7-a7f5-80162bef4541 europe-west1-b
UN 10.XXX.0.3 148.6 KB 64 ? e5b3dfec-f9bf-4952-b48f-b5a82749a9fe europe-west1-b
Datacenter: us-east1-b
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.XXX.0.2 158.72 KB 64 ? b93980e0-3c28-4bb0-9e49-220cb009d44c us-east1-b
UN 10.XXX.0.3 192.85 KB 64 ? af242b71-b7b4-4345-9888-5b89eb5e6199 us-east1-b
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
You have a DC in your replication factor ($dc) that doesnt exist, so when OpsCenter sends a request to read data the coordinator responds with an UnavailableException.
Try updating your keyspace like ALTER KEYSPACE "OpsCenter" WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east1-b': '2'} and doing a full repair if you want to keep the data thats there.
If it hasnt run yet so you dont have data you want to save, just delete the opscenter keyspace (drop keyspace "OpsCenter") and restart opscenterd and the agents (after opscenter service has started back up).

Unable to gossip with any seeds cassandra

I installed Data Stax 3.7 on my Windows machine(IP:10.175.12.249) and made following changes in my cassandra.yaml file:
cluster_name: 'Test_cluster'
listen_address: "10.175.12.249"
start_rpc: true
rpc_address: "0.0.0.0"
broadcast_rpc_address: "10.175.12.249"
seeds: "10.175.12.249"
endpoint_snitch: SimpleSnitch
Now, I started the service and cassandra is running fine on seed node.
I tried adding another node to my cluster. So I installed Data Stax 3.7 on another Windows machine(IP:192.168.158.78) and made following changes in cassandra.yaml file:
cluster_name: 'Test_cluster'
listen_address: "192.168.158.78"
start_rpc: true
rpc_address: "0.0.0.0"
broadcast_rpc_address: "192.168.158.78"
seeds: "10.175.12.249"
endpoint_snitch: SimpleSnitch
Now when I started the cassandra service on my 2nd machine, I am getting the following error:
INFO 09:41:27 Cassandra version: 3.7.0
INFO 09:41:27 Thrift API version: 20.1.0
INFO 09:41:27 CQL supported versions: 3.4.2 (default: 3.4.2)
INFO 09:41:27 Initializing index summary manager with a memory pool size of 100 MB and a resize interval of 60 minutes
INFO 09:41:27 Starting Messaging Service on /192.168.158.78:7000 (Intel(R) Centrino(R) Advanced-N 6235)
INFO 09:41:27 Scheduling approximate time-check task with a precision of 10 milliseconds
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any seeds
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1386)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:561)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:855)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714)
ERROR 09:41:58 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1386) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:561) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:855) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370) [apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.0.jar:3.7.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.0.jar:3.7.0]
WARN 09:41:58 No local state or state is in silent shutdown, not announcing shutdown
INFO 09:41:58 Waiting for messaging service to quiesce
Below is the output of nodetool status on seed node(IP:10.175.12.249):
C:\Program Files\DataStax-DDC\apache-cassandra\bin>nodetool status
Datacenter: datacenter1
========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
DN 192.168.158.78 ? 256 68.1% 6bc4e927-3def-4dfc-b5e7-31f5882ce475 rack1
UN 10.175.12.249 257.76 KiB 256 65.7% 300d731e-a27c-4922-aacc-6d42e8e49151 rack1
Thanks!!!
The - seeds: in conf/cassandra.yaml should have the same value (same IP or the hostname) as listen_address: in the same conf file.
I came across this error when the IP addresses were not matching. Try keeping the same and restart the cluster. Hope this helps...

cassandra-stress "Failed to connect over JMX; not collecting these stats"

I’m trying to use the cassandra-stress tool for the first time today. Although I'm able to run the tool, a lot of "Failed to connect over JMX; not collecting these stats" messages are displayed in the output
Command
cassandra-stress user \
profile=./stress_write.yaml ops\(insert=1\) \
n=1000000 \
-log file=./stress_write.log \
-node node1,node2,node3,node4,node5,node6
Output
WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node5) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node1) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node2) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node4) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node3) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node5) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node1) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node2) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node4) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node3) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side.
INFO 19:44:26 Using data-center name 'DC2' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
INFO 19:44:26 New Cassandra host /node2:9042 added
INFO 19:44:26 New Cassandra host /node5:9042 added
Connected to cluster: MyCluster
INFO 19:44:26 New Cassandra host /node4:9042 added
INFO 19:44:26 New Cassandra host /node1:9042 added
INFO 19:44:26 New Cassandra host /node6:9042 added
Datatacenter: DC2; Host: /node4; Rack: rack1
Datatacenter: DC2; Host: /node3; Rack: rack1
Datatacenter: DC2; Host: /node6; Rack: rack1
Datatacenter: DC2; Host: /node5; Rack: rack1
Datatacenter: DC2; Host: /node1; Rack: rack1
Datatacenter: DC2; Host: /node2; Rack: rack1
INFO 19:44:26 New Cassandra host /node3:9042 added
Created schema. Sleeping 6s for propagation.
Failed to connect over JMX; not collecting these stats
Generating batches with [1..1] partitions and [1..1] rows (of [1..1] total rows in the partitions)
Failed to connect over JMX; not collecting these stats
Failed to connect over JMX; not collecting these stats
Improvement over 4 threadCount: 36%
Failed to connect over JMX; not collecting these stats
Improvement over 8 threadCount: 138%
Failed to connect over JMX; not collecting these stats
Improvement over 16 threadCount: 48%
Failed to connect over JMX; not collecting these stats
Improvement over 24 threadCount: 33%
Failed to connect over JMX; not collecting these stats
Improvement over 36 threadCount: 27%
Failed to connect over JMX; not collecting these stats
Improvement over 54 threadCount: 39%
Failed to connect over JMX; not collecting these stats
Improvement over 81 threadCount: 37%
Failed to connect over JMX; not collecting these stats
Improvement over 121 threadCount: 16%
Failed to connect over JMX; not collecting these stats
Improvement over 181 threadCount: 1%
Failed to connect over JMX; not collecting these stats
Improvement over 271 threadCount: 15%
Failed to connect over JMX; not collecting these stats
Improvement over 406 threadCount: 3%
Failed to connect over JMX; not collecting these stats
Improvement over 609 threadCount: -3%
Is there any command-line or file-based configuration parameter that I need to specify for JMX? I have tested and confirmed that connectivity between the stress machine and my nodes is not the issue, because I was able to establish a connection between them via jmxsh.
Another issue with the output, which may or may not be related to the JMX error, is that is it missing some key parts. I'm quoting the sample output from this Datastax documentation page to show the parts that are missing from what I got:
WARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level analysis only
Running with 4 threadCount
Running WRITE with 4 threads until stderr of mean < 0.02
total ops , adj row/s, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, gc: #, max ms, sum ms, sdv ms, mb
2552 , 2553, 2553, 2553, 2553, 1.5, 1.4, 2.5, 6.0, 12.6, 18.0, 1.0, 0.00000, 0, 0, 0, 0, 0
5173 , 2634, 2613, 2613, 2613, 1.5, 1.5, 1.8, 2.6, 8.6, 9.2, 2.0, 0.00000, 0, 0, 0, 0, 0
...
Results:
op rate : 3954
partition rate : 3954
row rate : 3954
latency mean : 1.0
latency median : 0.8
latency 95th percentile : 1.5
latency 99th percentile : 1.8
latency 99.9th percentile : 2.2
latency max : 73.6
total gc count : 25
total gc mb : 1826
total gc time (s) : 1
avg gc time(ms) : 37
stdev gc time(ms) : 10
Total operation time : 00:00:59
Sleeping for 15s
Running with 4 threadCount
Notes
My cluster is running DSE 4.6.1 (Cassandra 2.0.12)
I'm running the stress tool from a different machine
The stress tool version is from DSC 2.1 (Cassandra 2.1)
I have the same setup (Cassandra version is 2.0.12) and the stress tool is from 2.1 and saw similar issues.
Finally I had some time to investigate.
I downloaded the source code and ran it in the debugger. What I saw is that this error message is misleading. The tool connects to JMX but has problem with one of the mBeans (org.apache.cassandra.service:type=GCInspector).
I saw the same exeption when I ran the stress test with the option: -log level=verbose and saw the following Exception:
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy11.getAndResetStats(Unknown Source)
at org.apache.cassandra.tools.NodeProbe.getAndResetGCStats(NodeProbe.java:385)
at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:86)
at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:64)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:187)
at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:97)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:61)
at org.apache.cassandra.stress.Stress.main(Stress.java:109)
Caused by: javax.management.InstanceNotFoundException: org.apache.cassandra.service:type=GCInspector
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source)
at ....
I connected to Cassandra using jConsole and version 2.0.12 does not have this mBean.
But my output has most of the data cited in the sample (except of the garbage collection statistics).
Have you tried running cassandra-stress with default configuration? Also try setting verbose for logging, may be it will give you some ideas.
I was also facing the same issue(Cassandra 3.7), I ran my Cassandra-stress client with -log level=verbose and saw below exception :
java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exce4; nested exception is:
java.net.ConnectException: Connection timed out]
at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99)
at org.apache.cassandra.stress.util.JmxCollector.(JmxCollector.java:85)
at org.apache.cassandra.stress.StressMetrics.(StressMetrics.java:62)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:211)
at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:107)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:60)
at org.apache.cassandra.stress.Stress.run(Stress.java:133)
at org.apache.cassandra.stress.Stress.main(Stress.java:61)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmion is:
java.net.ConnectException: Connection timed out]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:188)
at org.apache.cassandra.tools.NodeProbe.(NodeProbe.java:155)
at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:95)
... 7 more
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 1.2.3.4;
java.net.ConnectException: Connection timed out]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 11 more
Caused by: java.rmi.ConnectException: Connection refused to host: 1.2.3.4; nested exception is:
java.net.ConnectException: Connection timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
... 16 more
So to resolve this issue, I have set my rpc_address property from Cassandra.yaml file to <host_ip> and commented the broadcast_rpc_address property.
This works for me and I am not getting that error anymore.
edit conf/cassandra.yaml
change rpc_address: localhost to
rpc_address: 0.0.0.0
restart the database
Open JMX Port to the world in cassandra-env.sh file then restart the Cassandra service. After the stress test is completed you can revert the JMX port changes.

run two cassandra versions in the same machine

I have a scenario to run cassandra of two different versions in the same machine but at different ports.
I started one cluster with following cassandra config at port 9161,
# TCP port, for commands and data
storage_port: 7000
# SSL port, for encrypted communication. Unused unless enabled in
# encryption_options
ssl_storage_port: 7004
port for the CQL native transport to listen for clients on
native_transport_port: 9043
port for Thrift to listen for clients on
rpc_port: 9161
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "127.0.0.1"
It runs well,
$ /usr/local/apache-cassandra-2.1.1/bin/cassandra -f
...
INFO 05:08:42 Loading settings from file:/usr/local/apache-cassandra-2.1.1/conf/cassandra.yaml
...
INFO 05:09:29 Starting listening for CQL clients on localhost/127.0.0.1:9043...
INFO 05:09:29 Binding thrift service to localhost/127.0.0.1:9161
INFO 05:09:29 Listening for thrift clients...
INFO 05:19:25 No files to compact for user defined compaction
$ jps
5866 CassandraDaemon
8848 Jps
However while starting another cassandra cluster configured to run at port 9160 with config,
# TCP port, for commands and data
storage_port: 7000
# SSL port, for encrypted communication. Unused unless enabled in
# encryption_options
ssl_storage_port: 7004
port for the CQL native transport to listen for clients on
native_transport_port: 9042
port for Thrift to listen for clients on
rpc_port: 9160
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "127.0.0.1"
it fails with message
$ /usr/local/apache-cassandra-2.0.11/bin/cassandra -f
Unable to bind JMX, is Cassandra already running?
How can I make it run two different version of cassandra in the same machine?
The problem is that I have no permission to stop the previous version. Neither can I use https://github.com/pcmanus/ccm
The problem is your new version cassandra is also trying to use port 7199 for the JMX monitoring. Change the JMX port to some other unused port, then it will start. The JMX port can be change in the file called cassandraFolder/bin/cassandra.bat. There will be a line
-Dcom.sun.management.jmxremote.port=7199^
Change the above port to some other unused port.
If you are using cassandra in linux environment the JMX configuration will be located in the file called cassandraFolder/conf/cassandra-env.sh. There will be a line
JMX_PORT="7199"
Change this to some other unused port.
But I was unclear with your question.
Are you trying to run new cassandra to join in the existing cluster?
If yes, changing the JMX port will be sufficient.
Are you trying to run new cassandra in a stand alone mode?
If yes, change following configuration in yaml file.
seeds: "127.0.0.2"
listen_address: 127.0.0.2
rpc_address: 127.0.0.2
add the following entry,
127.0.0.1 127.0.0.2
in the /etc/hosts file if your you are running in linux. If you are running in windows add the above entry in C:\Windows\System32\drivers\etc\hosts file. If your intension is to run in stand alone mode then be careful in your configuration. If you made anything wrong then your new cassandra will join into the existing cluster.
This link helps you to run cassandra cluster in a single windows machine
Well I fixed it with changing few more configurations which are storage_port/ssl_storage_port in conf/cassandra.yaml and JMX_PORT in conf/cassandra-env.sh,
conf/cassandra.yaml
# TCP port, for commands and data
storage_port: 7004
# SSL port, for encrypted communication. Unused unless enabled in
# encryption_options
ssl_storage_port: 7005
port for the CQL native transport to listen for clients on
native_transport_port: 9043
port for Thrift to listen for clients on
rpc_port: 9161
seed_provider:
# Addresses of hosts that are deemed contact points.
# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring. You must change this if you are running
# multiple nodes!
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "127.0.0.1"
conf/cassandra-env.sh
# Specifies the default port over which Cassandra will be available for
# JMX connections.
JMX_PORT="7200"

Resources