JanusGraph, Spark cluster failing to connect to Cassandra - apache-spark

Question

I am trying to run a Spark job on a cluster that creates a JanusGraph.
I have an instance of JanusGraph server, Cassandra, ES running on a single machine, only the Spark computation occurs on the cluster. (Basically, I did a janusgraph.sh start on the machine
My configuration is as follows (x is the IP of the machine I am running the above instances on):
def getGraph(): JanusGraph = {
val config = JanusGraphFactory.build()
config.set("storage.backend", "cassandrathrift")
config.set("storage.cassandrathrift.keyspace", "jgex")
config.set("storage.hostname", "x")
config.set("index.jgex.backend", "elasticsearch")
config.set("index.jgex.index-name", "jgex")
config.set("jgex.hostname", "x")
config.open()
}
But when I do a spark-submit of the fat jar on the cluster, I get this:
java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:123)
at org.janusgraph.core.JanusGraphFactory$Builder.open(JanusGraphFactory.java:264)
at janus_create$.getGraph(janus_create.scala:66)
at janus_create$.makePropertiesandIndexes(janus_create.scala:830)
at janus_create$.main(janus_create.scala:921)
at janus_create.main(janus_create.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 16 more
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:219)
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.<init>(CassandraThriftStoreManager.java:198)
... 21 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.janusgraph.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeRawConnection(CTConnectionFactory.java:110)
at org.janusgraph.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:74)
at org.janusgraph.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:43)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
at org.janusgraph.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:216)
... 22 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 28 more
I tried switching between cassandra and cassandrathrift but both did not work. Also, where do I specify where my gremlin is running. Is that relevant?

Answer 1

The prepackaged distribution assumes a single, localhost node of each Cassandra, Elasticsearch, and Gremlin Server. Notice the java.net.ConnectException: Connection refused at the bottom of your stack trace. If you have Spark running on a remote cluster, you need to ensure that the servers are available on a non-localhost address.
Stop the distribution first with bin/janusgraph.sh stop
Update the listen_address and rpc_address in $JANUSGRAPH_HOME/conf/cassandra/cassandra.yaml using the IP address of the machine (Cassandra docs)
Add network.host to $JANUSGRAPH_HOME/elasticsearch/config/elasticsearch.yml using the IP address of the machine (Elasticsearch docs)
Update the host in $JANUSGRAPH_HOME/conf/gremlin-server/gremlin-server.yaml using the IP address of the machine (JanusGraph docs)
Assuming you are using the default Gremlin Server configuration gremlin-server.yaml, you need to update the properties file in $JANUSGRAPH_HOME/conf/gremlin-server/janusgraph-cassandra-es-server.properties using the IP address of the machine. Update storage.hostname and index.search.hostname using the IP address of the machine, matching the server settings above.
Update: It seems that there are errors in your graph connection properties that might be contributing to the problem:
config.set("storage.cassandrathrift.keyspace", "jgex") -- should be "storage.cassandra.keyspace"
config.set("jgex.hostname", "x") -- should be "index.jgex.hostname"