When i am using Cassandra's nodetool to see the ring of a remote host (use IP address), it gives following error, how to make this work?
BTW - I can ping that host using IP address.
root#ServerA:~/cassandra# bin/nodetool -h 172.24.0.10 ring
Error connecting to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2343)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:296)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:106)
at org.apache.cassandra.tools.NodeProbe.(NodeProbe.java:82)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:405)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at java.net.Socket.(Socket.java:392)
at java.net.Socket.(Socket.java:206)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 10 more
You need to enable remote JMX:
To enable monitoring and management
from remote systems, set this system
property when you start the JVM:
com.sun.management.jmxremote.port=portNum
Related
I am trying to run a Spark job on a cluster that creates a JanusGraph.
I have an instance of JanusGraph server, Cassandra, ES running on a single machine, only the Spark computation occurs on the cluster. (Basically, I did a janusgraph.sh start on the machine
My configuration is as follows (x is the IP of the machine I am running the above instances on):
def getGraph(): JanusGraph = {
val config = JanusGraphFactory.build()
config.set("storage.backend", "cassandrathrift")
config.set("storage.cassandrathrift.keyspace", "jgex")
config.set("storage.hostname", "x")
config.set("index.jgex.backend", "elasticsearch")
config.set("index.jgex.index-name", "jgex")
config.set("jgex.hostname", "x")
config.open()
}
But when I do a spark-submit of the fat jar on the cluster, I get this:
java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:123)
at org.janusgraph.core.JanusGraphFactory$Builder.open(JanusGraphFactory.java:264)
at janus_create$.getGraph(janus_create.scala:66)
at janus_create$.makePropertiesandIndexes(janus_create.scala:830)
at janus_create$.main(janus_create.scala:921)
at janus_create.main(janus_create.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 16 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:219)
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.<init>(CassandraThriftStoreManager.java:198)
... 21 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.janusgraph.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeRawConnection(CTConnectionFactory.java:110)
at org.janusgraph.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:74)
at org.janusgraph.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:43)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:216)
... 22 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 28 more
I tried switching between cassandra and cassandrathrift but both did not work. Also, where do I specify where my gremlin is running. Is that relevant?
The prepackaged distribution assumes a single, localhost node of each Cassandra, Elasticsearch, and Gremlin Server. Notice the java.net.ConnectException: Connection refused at the bottom of your stack trace. If you have Spark running on a remote cluster, you need to ensure that the servers are available on a non-localhost address.
Stop the distribution first with bin/janusgraph.sh stop
Update the listen_address and rpc_address in $JANUSGRAPH_HOME/conf/cassandra/cassandra.yaml using the IP address of the machine (Cassandra docs)
Add network.host to $JANUSGRAPH_HOME/elasticsearch/config/elasticsearch.yml using the IP address of the machine (Elasticsearch docs)
Update the host in $JANUSGRAPH_HOME/conf/gremlin-server/gremlin-server.yaml using the IP address of the machine (JanusGraph docs)
Assuming you are using the default Gremlin Server configuration gremlin-server.yaml, you need to update the properties file in $JANUSGRAPH_HOME/conf/gremlin-server/janusgraph-cassandra-es-server.properties using the IP address of the machine. Update storage.hostname and index.search.hostname using the IP address of the machine, matching the server settings above.
Update: It seems that there are errors in your graph connection properties that might be contributing to the problem:
config.set("storage.cassandrathrift.keyspace", "jgex") -- should be "storage.cassandra.keyspace"
config.set("jgex.hostname", "x") -- should be "index.jgex.hostname"
I'm able to connect to cassandra docker instance using external ip with CQLSH but it is failing to connect from java client.
Server logs:
Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
Error from application:
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /14.24.34.44:9042 (com.datastax.driver.core.exceptions.TransportException: [/14.24.34.44:9042] Cannot connect))
Cassandra version used: 3.11.1
I have datastax-cassandra 1.2.5 and I have following settings in .yaml file
storage_port: 7000
ssl_storage_port: 7001
listen_address: localhost
rpc_port: 9160
I keep getting this. I tried to change the storage port once and it worked but than after same thing. I am not able to restart cassandra again
INFO 16:33:02,714 Completed flushing /var/lib/cassandra/data/system/local/system-local-ic-17-Data.db (241 bytes) for commitlog position ReplayPosition(segmentId=1371684781848, position=50142)
ERROR 16:33:02,793 Exception encountered during startup
java.lang.RuntimeException: java.net.BindException: Can't assign requested address
at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:446)
at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:389)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:583)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:548)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:445)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:325)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:413)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:456)
Caused by: java.net.BindException: Can't assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:344)
at sun.nio.ch.Net.bind(Net.java:336)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:436)
... 7 more
java.lang.RuntimeException: java.net.BindException: Can't assign requested address
at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:446)
at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:389)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:583)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:548)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:445)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:325)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:413)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:456)
Caused by: java.net.BindException: Can't assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:344)
at sun.nio.ch.Net.bind(Net.java:336)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.cassandra.net.MessagingService.getServerSocket(MessagingService.java:436)
... 7 more
Exception encountered during startup: java.net.BindException: Can't assign requested address
ERROR 16:33:02,798 Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:321)
at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:362)
at org.apache.cassandra.service.StorageService.access$000(StorageService.java:88)
at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:513)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:722)
Most likely there's something wrong with network configuration: localhost resolves either to invalid hostname assigned by DHCP (something like 192-168-1-10.local) or to local IPv6 address (::1) and IPv6 is disabled in java.
Check /etc/hosts
Check output of hostname command
Try setting listen_address to 127.0.0.1 or to valid IP address.
Check rpc_address setting in cassandra.yaml. Try setting it to 127.0.0.1
I'm getting exception while bulk loads with sstableloader. I'm using JDK 1.6.0_25 64bit, Ubuntu 12.04 server. Ipv6 is turned off. Network communication between hosts works correctly. I'm going crazy ;-(
Exception in thread "Streaming to /192.168.219.36:1" java.lang.RuntimeException: java.net.SocketException: Invalid argument or cannot assign requested address
at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Invalid argument or cannot assign requested address
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:276)
at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
All hosts runs Cassandra 1.1 (datastax edition). Ports 7000,7199,9160 opened. Any ideas ??
Why are you streaming to port 1? It should be the RPC_PORT and the default for that is 9160.
/192.168.219.36: 1
What params are you passing to the bulkloader?
I only try to add domain but get connection refused exception:
The command line i tried, and below the exceptions:
sudo ./james-cli.sh -h localhost -p 9999 adddomain mydomain.tld
Output:
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:357)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at org.apache.james.cli.probe.impl.JmxServerProbe.connect(JmxServerProbe.java:93)
at org.apache.james.cli.probe.impl.JmxServerProbe.<init>(JmxServerProbe.java:68)
at org.apache.james.cli.ServerCmd.main(ServerCmd.java:104)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
at javax.naming.InitialContext.lookup(InitialContext.java:411)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1924)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1891)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:274)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 14 more
I had this same problem until I actually issued the command in root. Change to root on the command line by typing su and it will ask for a password. Then you should be able to change the james settings.
Something strange is going on there. I have this problem also from time to time and I am not really sure how it was solved.
Last time i had it disappeared after I started james server which was not started. Before that I solved it with sudo, but with absolute path to the james-cli.sh
I have strange feeling that there is something not working properly in james' RMI.