Unable to start Cassandra DSC on Mac - Error starting local jmx server - cassandra

Am trying to run Cassandra on my Mac OSX 10.10. I've installed dsc-cassandra-2.1.9.
Some of the cassandra.yaml settings are:
listen_address: 127.0.0.1
rpc_address: 127.0.0.1
rpc_port: 9160
However when I start cassandra, it fails:
$ ./bin/cassandra
...
WARN [main] 2015-10-08 23:01:43,226 CLibrary.java:70 - JNA link failure, one or more native method will be unavailable.
WARN [main] 2015-10-08 23:01:43,227 CassandraDaemon.java:81 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR [main] 2015-10-08 23:02:58,707 CassandraDaemon.java:116 - Error starting local jmx server:
java.io.IOException: Cannot bind to URL [rmi://localhost:7199/jmxrmi]: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Operation timed out]
at javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:826) ~[na:1.7.0_51]
at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:431) ~[na:1.7.0_51]
at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:112) [apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216) [apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) [apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:626) [apache-cassandra-2.1.9.jar:2.1.9]
Caused by: javax.naming.ServiceUnavailableException: null
Could anyone please help with fixing this issue?
updates to above problem --
after setting the JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1" as suggested in the answer below, cassandra seems to be running, however the nodetool doesn't seem to be reporting correct status:
$ ps -ef|grep cass
0 49292 1 0 8:53PM ttys001 0:13.04 /usr/bin/java -ea -javaagent:./bin/../lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4096M -Xmx4096M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:CompileCommandFile=./bin/../conf/hotspot_compiler -XX:CMSWaitDuration=10000 -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=127.0.0.1 -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=./bin/../logs -Dcassandra.storagedir=./bin/../data -cp ./bin/../conf:./bin/../build/classes/main:./bin/../build/classes/thrift:./bin/../lib/ST4-4.0.8.jar:./bin/../lib/airline-0.6.jar:./bin/../lib/antlr-runtime-3.5.2.jar:./bin/../lib/apache-cassandra-2.1.9.jar:./bin/../lib/apache-cassandra-clientutil-2.1.9.jar:./bin/../lib/apache-cassandra-thrift-2.1.9.jar:./bin/../lib/commons-cli-1.1.jar:./bin/../lib/commons-codec-1.2.jar:./bin/../lib/commons-lang3-3.1.jar:./bin/../lib/commons-math3-3.2.jar:./bin/../lib/compress-lzf-0.8.4.jar:./bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:./bin/../lib/disruptor-3.0.1.jar:./bin/../lib/guava-16.0.jar:./bin/../lib/high-scale-lib-1.0.6.jar:./bin/../lib/jackson-core-asl-1.9.2.jar:./bin/../lib/jackson-mapper-asl-1.9.2.jar:./bin/../lib/jamm-0.3.0.jar:./bin/../lib/javax.inject.jar:./bin/../lib/jbcrypt-0.3m.jar:./bin/../lib/jline-1.0.jar:./bin/../lib/jna-4.0.0.jar:./bin/../lib/json-simple-1.1.jar:./bin/../lib/libthrift-0.9.2.jar:./bin/../lib/logback-classic-1.1.2.jar:./bin/../lib/logback-core-1.1.2.jar:./bin/../lib/lz4-1.2.0.jar:./bin/../lib/metrics-core-2.2.0.jar:./bin/../lib/netty-all-4.0.23.Final.jar:./bin/../lib/reporter-config-2.1.0.jar:./bin/../lib/slf4j-api-1.7.2.jar:./bin/../lib/snakeyaml-1.11.jar:./bin/../lib/snappy-java-1.0.5.2.jar:./bin/../lib/stream-2.5.2.jar:./bin/../lib/super-csv-2.1.0.jar:./bin/../lib/thrift-server-0.3.7.jar org.apache.cassandra.service.CassandraDaemon
$ ./bin/nodetool status
objc[52026]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
nodetool: Failed to connect to '127.0.0.1:7199' - NameNotFoundException: 'jmxrmi'.
And cqlsh seems to be working:
$ ./bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.9 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh>

Faced the same error and below is the solution.
Stop cassandra
ps -ef | grep cassandra | grep -v grep | awk '{print $2}' | xargs kill -9
In cassandra-env.sh modify the following:-
Uncomment JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=" and change to
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1"
Change LOCAL_JMX=yes to LOCAL_JMX=no
Change true to false
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
If still you see any issues check if the port specified by JMX_PORT="XXX" is occupied or not. OR Simply you can change it to JMX_PORT="9909" and restart cassandra.

Verify that nothing else is running on port 7199. You can e.g. also use a different port for JMX by modifying JMX_PORT in cassandra-env.sh.
Look into cassandra-env.sh and try modifying the following line:
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"

Related

datastax cassandra java NoHostAvailableException

Had a fresh installation of Apache Cassandra version 3.7
Installation was successful and i could access cqlsh with out any issues.
[dpmuser#LOGDPM01 bin]$ cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 3.4.2 | Native protocol v4]
But when i try to connect via DataStax Cassandra client , I am hitting NoHostAvailableException .
Here is the sample code i tried
import com.datastax.driver.core.*;
public class DBHitTEst {
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
Session session = null;
Cluster cluster = Cluster.builder()
.addContactPoints("127.0.0.1").withPort(9042)
.build();
System.out.println("Connecting to cluster ...");
session = cluster.connect();
System.out.print("Connected !"+session);
}}
Exception in console shown below.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Connecting to cluster ...
Aug 10, 2017 7:58:23 AM io.netty.channel.ChannelInitializer channelRegistered
WARNING: Failed to initialize a channel. Closing: [id: 0x7d2724c5]
java.lang.NoSuchMethodError: io.netty.util.AttributeKey.valueOf(Ljava/lang/String;)Lio/netty/util/AttributeKey;
at com.datastax.driver.core.Message.(Message.java:39)
at com.datastax.driver.core.Connection$Initializer.initChannel(Connection.java:1425)
.
.
.
Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [/127.0.0.1:9042] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:232)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1600)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1518)
at com.datastax.driver.core.Cluster.init(Cluster.java:159)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:330)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:305)
at com.datastax.driver.core.Cluster.connect(Cluster.java:247)
at com.cassandra.db.executables.DBHitTEst.main(DBHitTEst.java:14)
Libraries References
cassandra-driver-core-3.3.0.jar
guava-23.0.jar
log4j-api-2.8.2.jar
metrics-core-3.0.2.jar
netty-all-4.0.9.Final.jar
slf4j-api-1.7.25.jar
Important configs is cassandra.yaml
start_rpc: false
start_native_transport: true
native_transport_port: 9042
rpc_address: localhost
rpc_port: 9160
listen_address: localhost
seeds: "127.0.0.1"
Other system particulars
[dpmuser#LOGDPM01 conf]$ hostname
LOGDPM01
[dpmuser#LOGDPM01 conf]$ more /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
[dpmuser#LOGDPM01 conf]$ ping LOGDPM01
PING LOGDPM01 (10.0.2.15) 56(84) bytes of data.
64 bytes from LOGDPM01 (10.0.2.15): icmp_seq=1 ttl=64 time=0.055 ms
Running centos VM.Kindly let me know where am i wrong ?
Library mismatch was throwing the warning I observed in console .
WARNING: Failed to initialize a channel. Closing: [id: 0x7d2724c5] java.lang.NoSuchMethodError: io.netty.util.AttributeKey.valueOf(Ljava/lang/String;)Lio/netty/util/AttributeKey; at com.datastax.driver.core.Message.(Message.java:39) at com.datastax.driver.core.Connection$Initializer.initChannel(Connection.java:1425) . . .
On choosing the correct libraries from the lib folder of Cassandra installation directly sorted the issue .

Exception encountered during startup: Unknown commitlog version 6

Cannot start cassandra in DataStax Enterprise (single node scenario)
runing $ sudo service dse start I get:
Starting DSE daemon : dse
DSE daemon starting with just Cassandra enabled (edit /etc/default/dse to enable)
running $ nodetool status
Failed to connect to '127.0.0.1:7199': Connection refused (Connection refused)
from output.log:
ERROR 08:03:24,900 Exception encountered during startup
java.lang.IllegalStateException: Unknown commitlog version 6
at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:80)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:182)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:95)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:151)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:131)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:336)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:394)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:574)
java.lang.IllegalStateException: Unknown commitlog version 6
at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:80)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:182)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:95)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:151)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:131)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:336)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:394)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:574)
Exception encountered during startup: Unknown commitlog version 6
INFO 08:03:24,915 DSE shutting down...
What can I do ? any suggestion are wellcome

Datastax Spark worker is always looking for master at 127.0.0.1

I am trying to bring up datastax cassandra in analytics mode by using "dse cassandra -k -s". I am using DSE 5.0 sandbox on a single node setup.
I have configured the spark-env.sh with SPARK_MASTER_IP as well as SPARK_LOCAL_IP to point to my LAN IP.
export SPARK_LOCAL_IP="172.40.9.79"
export SPARK_MASTER_HOST="172.40.9.79"
export SPARK_WORKER_HOST="172.40.9.79"
export SPARK_MASTER_IP="172.40.9.79"
All above variables are setup in spark-env.sh.
Despite these, the worker will not come up. It is always looking for a master at 127.0.0.1.This is the error i am seeing in /var/log/cassandra/system.log
WARN [worker-register-master-threadpool-8] 2016-10-04 08:02:45,832 SPARK-WORKER Logging.scala:91 - Failed to connect to master 127.0.0.1:7077
java.io.IOException: Failed to connect to /127.0.0.1:7077
Result from dse client-tool shows 127.0.0.1
$ dse client-tool -u cassandra -p cassandra spark master-address
spark://127.0.0.1:7077
However i am able to access the spark web UI from the LAN IP 172.40.9.79
Spark Web UI screenshot
Any help is greatly appreciated
Try add in file spark-defaults.conf this parameter: spark.master local[*] or spark.master 172.40.9.79. Maybe this solves your problem

Error when running cassandra-stress on Centos 6.7

I am trying to build a 3 node Cassandra cluster on Centos 6.7. I had no issues installing it and I can successfully run the cqlshell commands. However whenever I try to run the stress test as very simple testing case as:
cassandra-stress write n=10
I get this error
java.lang.RuntimeException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1] Cannot connect))
at org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:222)
at org.apache.cassandra.stress.settings.SettingsSchema.createKeySpacesNative(SettingsSchema.java:79)
at org.apache.cassandra.stress.settings.SettingsSchema.createKeySpaces(SettingsSchema.java:69)
at org.apache.cassandra.stress.settings.StressSettings.maybeCreateKeyspaces(StressSettings.java:230)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:54)
at org.apache.cassandra.stress.Stress.run(Stress.java:133)
at org.apache.cassandra.stress.Stress.main(Stress.java:61)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:231)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1414)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:393)
at org.apache.cassandra.stress.util.JavaDriverClient.connect(JavaDriverClient.java:154)
at org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:213)
... 6 more
Both the seeds, listen_address and rpc_address have the correct addresses and I opened 9042, 7000, 7001 and 7199 ports in the firewall. So I am not entirely sure what might be the problem.
run nodetool status
Take one of the printed IP addresses and try: cassandra-stress write n=10 -node <ip address>

connecting to cassandra using datastax' connector

When I'm trying to connect to the cassandra seed node using the datastax connector, I can't.
I have four spark nodes: one master and three workers. This works well on its own. The same machines have cassandra installed on them with the one being the spark master as a seed node. This works on its own as well (I successfully wrote and read from it).
Now, I'm trying to do
val info = spark_context.cassandraTable("files", "metainfo")
println( info.count )
Before, I specify the spark context as follows:
val confStandalone = new SparkConf()
.set("spark.cassandra.connection.host", "10.14.56.156")
.setMaster("spark://10.14.56.156:7077")
.setAppName("Test")
.set("spark.executor.memory", "1g")
.set("spark.eventLog.enabled", "true")
.set("spark.driver.host", "10.14.56.156")
.set("spark.broadcast.factory", "org.apache.spark.broadcast.HttpBroadcastFactory")
val spark_context = new SparkContext( confStandalone )
spark_context.addJar("SOME_PATH/spark-cassandra-connector_2.10-1.2.0-alpha1.jar")
In the cassandra.yaml file I set the rpc_address to 10.14.56.156 and used the standard ports (9160, 9042). Now when I do
sbt run
I get the following error:
15/03/18 16:38:43 INFO LocalNodeFirstLoadBalancingPolicy: Adding host 127.0.0.1 (datacenter1)
15/03/18 16:38:43 INFO LocalNodeFirstLoadBalancingPolicy: Adding host 10.14.56.156 (datacenter1)
15/03/18 16:38:43 INFO LocalNodeFirstLoadBalancingPolicy: Adding host 127.0.0.1 (datacenter1)
15/03/18 16:38:43 ERROR Session: Error creating pool to /127.0.0.1:9042 com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot connect
at com.datastax.driver.core.Connection.<init>(Connection.java:106)
at com.datastax.driver.core.PooledConnection.<init>(PooledConnection.java:35)
at com.datastax.driver.core.Connection$Factory.open(Connection.java:528)
...
Caused by: java.net.ConnectException: Connection refused: /127.0.0.1:9042 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
...
Now, when I change the rpc_address to 0.0.0.0 as id sometimes advised, I get the same error but with 10.14.56.156 instead of 127.0.0.1 and only the line:
15/03/18 16:38:43 INFO LocalNodeFirstLoadBalancingPolicy: Adding host 10.14.56.156 (datacenter1)
with the one above and the one below (referring to 127.0.0.1) removed.
I didn't set any firewall rules in the iptables, so I don't think that would be an issue. Help appreciated!
Have you looked at what broadcast_rpc_address is set to? The java-driver will derive the ip to connect to from the 'peer' column of system.peers. If rpc_address is set to 0.0.0.0, broadcast_rpc_address must be set.
My guess is that with your rpc_address set to 0.0.0.0, the driver is connecting from the broadcast_rpc_address even though it says [/10.14.56.156:9042] Cannot connect (you may see Connection refused: /127.0.0.1:9042 further in the stack trace).

Resources