Titan error when one of the connected Cassandra nodes comes down - cassandra

We have Titan setup with Cassandra as it's storage.backend. We have a cluster of 3 Cassandra nodes. We are seeing the following error when one of the Cassandra nodes fail/shutdown.
This is the error that gets logged. Any help on this will be really appreciated.
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.7
ERROR ConnectionPoolMBeanManager:74 - com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
ERROR ConnectionPoolMBeanManager:74 - com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterTitanConnectionPool,ServiceType=connectionpool
INFO GraphDatabaseConfiguration:1518 - Generated unique-instance-id=3217005b14357-titan-m21e
INFO ConnectionPoolMBeanManager:53 - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterTitanConnectionPool,ServiceType=connectionpool
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.44
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.57
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.7
INFO ConnectionPoolMBeanManager:53 - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.44
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.57
INFO CountingConnectionPoolMonitor:194 - AddHost: 10.76.54.7
INFO Backend:459 - Configuring index [search]
INFO plugins:151 - [Donald Ritter] loaded [], sites []
INFO ElasticSearchIndex:354 - Configured remote host: 10.76.54.15 : 9300
INFO ElasticSearchIndex:354 - Configured remote host: 10.76.54.24 : 9300
INFO ElasticSearchIndex:354 - Configured remote host: 10.77.170.171 : 9300
INFO Backend:176 - Initiated backend operations thread pool of size 16
INFO KCVSLog:730 - Loaded unidentified ReadMarker start time 2016-02-16T23:45:27.539Z into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#49f59d47

Related

Add a Cassandra OSS 4.0 RC1 node into a cluster with DSE 6.0.14 nodes

Every nodes of the cluster are in Version DSE 6.0.14, they set in ssl mode (listen on port 7001).
We're trying to add a node in version open Sources 4.0 RC1.
We force the port communication on this node:
storage_port: 7001
else the node try to communicate on the 7000 port that is closed.
We encountered the following error, when I try to start the service of the new node :
INFO [main] 2021-05-10 16:22:00,985 StorageService.java:528 - Gathering node replacement information for /10.135.66.204:7001
DEBUG [main] 2021-05-10 16:22:00,986 YamlConfigurationLoader.java:112 - Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
DEBUG [main] 2021-05-10 16:22:00,996 YamlConfigurationLoader.java:112 - Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO [Messaging-EventLoop-3-1] 2021-05-10 16:22:01,138 InboundConnectionInitiator.java:281 - peer /10.137.65.201:54916 only supports messaging versions lower (2) than this node supports (10)
ERROR [Messaging-EventLoop-3-2] 2021-05-10 16:22:01,237 NoSpamLogger.java:98 - /xx.xxx.xx.xxx:7001->/xx.xxx.xx.xxx:7001-URGENT_MESSAGES-[no-channel] failed to connect
java.nio.channels.ClosedChannelException: null
[...]
INFO [ScheduledTasks:1] 2021-05-10 16:22:02,398 TokenMetadata.java:525 - Updating topology for all endpoints that have changed
ERROR [Messaging-EventLoop-3-1] 2021-05-10 16:22:09,467 InboundConnectionInitiator.java:360 - Failed to properly handshake with peer /xx.xxx.xx.xxx:54922. Closing the channel.
java.lang.AssertionError: null
[...]
I don't know if the error come from a mistake in the config of the node oss 4.0 or if there is an incompatibility between the new node version and the version of the existing node in the cluster.

Setting up a Selenium node on Linux

I am currently trying to set up a Selenium node on a Linux VM (headless). The hub is being run from a Windows device on port 5786 (4444 was in use).
I initiate the hub using the following code:
java -jar selenium-server-standalone-
2.53.0.jar -role node –hub http://<myIP>:5786/grid/register -port 5558
When i execute this, it returns the following error messages:
13:42:12.765 INFO - Launching a Selenium Grid node
13:42:13.435 WARN - error getting the parameters from the hub. The node may
end up with wrong timeouts.Connect to 10.146.48.80:4444 [/10.146.48.80]
failed: Connection refused
13:42:13.445 INFO - Java: Oracle Corporation 24.60-b09
13:42:13.445 INFO - OS: Linux 3.10.0-514.21.1.el7.x86_64 amd64
13:42:13.450 INFO - v2.53.0, with Core v2.53.0. Built from revision 35ae25b
13:42:13.521 INFO - Driver provider
org.openqa.selenium.ie.InternetExplorerDriver registration is skipped:
registration capabilities Capabilities [{platform=WINDOWS,
ensureCleanSession=true, browserName=internet explorer, version=}] does not
match the current platform LINUX
13:42:13.521 INFO - Driver provider org.openqa.selenium.edge.EdgeDriver
registration is skipped:
registration capabilities Capabilities [{platform=WINDOWS,
browserName=MicrosoftEdge, version=}] does not match the current platform
LINUX
13:42:13.521 INFO - Driver class not found:
com.opera.core.systems.OperaDriver
13:42:13.521 INFO - Driver provider com.opera.core.systems.OperaDriver is not
registered
13:42:13.522 INFO - Driver provider org.openqa.selenium.safari.SafariDriver
registration is skipped:
registration capabilities Capabilities [{platform=MAC, browserName=safari,
version=}] does not match the current platform LINUX
13:42:13.523 INFO - Driver class not found:
org.openqa.selenium.htmlunit.HtmlUnitDriver
13:42:13.523 INFO - Driver provider
org.openqa.selenium.htmlunit.HtmlUnitDriver is not registered
13:42:13.552 INFO - Version Jetty/5.1.x
13:42:13.553 INFO - Started HttpContext[/selenium-server/driver,/selenium-
server/driver]
13:42:13.553 INFO - Started HttpContext[/selenium-server,/selenium-server]
13:42:13.553 INFO - Started HttpContext[/,/]
13:44:03.596 INFO - Started
org.openqa.jetty.jetty.servlet.ServletHandler#188fd321
13:44:03.596 INFO - Started HttpContext[/wd,/wd]
13:44:03.599 INFO - Started SocketListener on 0.0.0.0:5555
13:44:03.599 INFO - Started org.openqa.jetty.jetty.Server#a01330f
13:44:03.599 INFO - Selenium Grid node is up and ready to register to the hub
13:44:03.630 INFO - Starting auto registration thread. Will try to register
every 5000 ms.
13:44:03.630 INFO - Registering the node to the hub:
http://10.146.48.80:4444/grid/register
13:44:03.638 INFO - Couldn't register this node: Error sending the
registration request: Connect to 10.146.48.80:4444 [/10.146.48.80] failed:
Connection refused
In the command that you shared viz.,
java -jar selenium-server-standalone-2.53.0.jar -role node –hub http://<myIP>:5786/grid/register -port 5558
You have a copy paste problem. The - before the parameter hub doesn't seem to be a hyphen character.
I was able to deduce this as a possible root cause because the value of -hub is completely being ignored and the default hub port of 4444 is being attempted to use. The stacktrace that you shared re-iterates this
13:42:13.435 WARN - error getting the parameters from the hub. The node may
end up with wrong timeouts.Connect to 10.146.48.80:4444 [/10.146.48.80]
failed: Connection refused
As you can see, even though you are providing your Hub's registration URL to include a port number of 5786, its still picking up 4444 and complaining Connection refused.
Please try typing out the entire command for starting the hub and try again. That should work.

How to execute gremlin query with mogwai

Im trying to query a titan db 0.5.4 via mogwai, but when I run the following script i get the error: rexpro.exceptions.RexProScriptException: transaction is not open
and I found the same question here
P.S there is no tag for mogwai
script:
#!/usr/bin/env python3
from mogwai.connection import execute_query, setup
con = setup('127.0.0.1', graph_name="bio4j", username="re", password="re")
results = execute_query("2 * a",params={"a":2}, connection= con)
print(results)
results = execute_query("bio4j.E",params={}, connection= con)
print(results)
log:
$ ./bin/rexster.sh --start
0 [main] INFO com.tinkerpop.rexster.Application - .:Welcome to Rexster:.
93 [main] INFO com.tinkerpop.rexster.server.RexsterProperties - Using [/Users/Phoenix/Dropbox/Graph4Bio/Titan/rexhome/config/rexster.xml] as configuration source.
102 [main] INFO com.tinkerpop.rexster.Application - Rexster is watching [/Users/Phoenix/Dropbox/Graph4Bio/Titan/rexhome/config/rexster.xml] for change.
730 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration - Generated unique-instance-id=0a69045d1736-AngryMac-local1
804 [main] INFO com.thinkaurelius.titan.diskstorage.Backend - Initiated backend operations thread pool of size 8
905 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog - Loaded unidentified ReadMarker start time Timepoint[1455128079919000 μs] into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#302c971f
908 [main] INFO com.tinkerpop.rexster.RexsterApplicationGraph - Graph [bio4j] - configured with allowable namespace [tp:gremlin]
932 [main] INFO com.tinkerpop.rexster.config.GraphConfigurationContainer - Graph bio4j - titangraph[berkeleyje:/Users/Phoenix/Dropbox/Graph4Bio/Bio4j/bio4j] loaded
939 [main] INFO com.tinkerpop.rexster.server.metrics.HttpReporterConfig - Configured HTTP Metric Reporter.
941 [main] INFO com.tinkerpop.rexster.server.metrics.ConsoleReporterConfig - Configured Console Metric Reporter.
2058 [main] INFO com.tinkerpop.rexster.server.HttpRexsterServer - HTTP/REST thread pool configuration: kernal[4 / 4] worker[8 / 8]
2060 [main] INFO com.tinkerpop.rexster.server.HttpRexsterServer - Using org.glassfish.grizzly.strategies.LeaderFollowerNIOStrategy IOStrategy for HTTP/REST.
2160 [main] INFO com.tinkerpop.rexster.server.HttpRexsterServer - Rexster Server running on: [http://localhost:8182]
2160 [main] INFO com.tinkerpop.rexster.server.RexProRexsterServer - Using org.glassfish.grizzly.strategies.LeaderFollowerNIOStrategy IOStrategy for RexPro.
2160 [main] INFO com.tinkerpop.rexster.server.RexProRexsterServer - RexPro thread pool configuration: kernal[4 / 4] worker[8 / 8]
2162 [main] INFO com.tinkerpop.rexster.server.RexProRexsterServer - Rexster configured with [DefaultSecurity].
2163 [main] INFO com.tinkerpop.rexster.server.RexProRexsterServer - RexPro Server bound to [0.0.0.0:8184]
2177 [main] INFO com.tinkerpop.rexster.server.ShutdownManager - Bound shutdown socket to /127.0.0.1:8183. Starting listener thread for shutdown requests.
152568 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.EngineController - ScriptEngineManager has factory for: ECMAScript
152568 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.EngineController - ScriptEngineManager has factory for: gremlin-groovy
152568 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.EngineController - Registered ScriptEngine for: gremlin-groovy
152569 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.EngineHolder - Initializing gremlin-groovy engine with additional imports.
153259 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.EngineHolder - ScriptEngine initializing with a custom script
154074 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.EngineController - ScriptEngineManager has factory for: Groovy
154076 [Grizzly(2) SelectorRunner] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - RexPro Session created: a2b416ce-75ea-4ecb-9835-b287162c90cb
154354 [Grizzly(4)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - Try to destroy RexPro Session: a2b416ce-75ea-4ecb-9835-b287162c90cb
154355 [Grizzly(4)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - RexPro Session destroyed or doesn't otherwise exist: a2b416ce-75ea-4ecb-9835-b287162c90cb
154356 [Grizzly(5)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - RexPro Session created: 5b8a669f-615d-4f84-9d1e-2d10624347f0
154525 [Grizzly(7)] WARN com.tinkerpop.rexster.protocol.server.ScriptServer - Could not process script [bio4j.E] for language [groovy] on session [[B#6634722f] and request [[B#68f38099]
154527 [Grizzly(8)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - Try to destroy RexPro Session: 5b8a669f-615d-4f84-9d1e-2d10624347f0
154527 [Grizzly(8)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - RexPro Session destroyed or doesn't otherwise exist: 5b8a669f-615d-4f84-9d1e-2d10624347f0
154529 [Grizzly(1)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - Try to destroy RexPro Session: 00000000-0000-0000-0000-000000000000
154529 [Grizzly(1)] INFO com.tinkerpop.rexster.protocol.session.RexProSessions - RexPro Session destroyed or doesn't otherwise exist: 00000000-0000-0000-0000-000000000000
Maintainer of mogwai here.
What version of mogwai are you using? in 0.7.7 there is no return value for setup method and the connection object should not be passed around. In fact when you call setup it creates a connection pool (a synchronous rexpro connection pool since there was no concurrency option specified). So in general, just call setup once for the life of your app and you can use execute query without any references.
Also this message in particular stands out:
154525 [Grizzly(7)] WARN com.tinkerpop.rexster.protocol.server.ScriptServer - Could not process script [bio4j.E] for language [groovy] on session [[B#6634722f] and request [[B#68f38099]
Is your graph configured with a graph name of "bio4j"? The default titan graph name is "graph" and the default graph object name mogwai uses is "g". If you have a graph name of "bio4j" you wouldn't reference this directly, you'd use the graph object name associated to the transaction. You can think of a graph-name as a database name in a SQL database, and the graph object being the transactional reference to said database. This is configured in the xml configuration file when starting titan. Particularly:
<graphs>
<graph>
<graph-name>graph</graph-name>
....
</graph>
</graphs>
So assuming you changed that from "graph" to "bio4j" and left the default graph_obj_name in the setup function as "g", then your query should read "g.E".

Selenium test execution via jenkins on linux machine without GUI (CLI-only) - HEADLESS MODE

This is regarding Selenium Automation Testing. I have a Jenkins job setup for some test executions.Jenkins is setup on a Ubuntu machine without GUI (CLI Only).
So when I run the scripts seems like it can't find the web browser obviously.
This job is working perfectly fine in windows. In Windows I get like this result.
Window Successful Result
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
06/08/2015 00:04:47,996 INFO [main] (BasicTestObject.java:251) - ======BEGIN Test workflow============
06/08/2015 00:04:48,002 INFO [main] (BasicTestObject.java:252) - BEGIN Test: MlpBvt
06/08/2015 00:04:48,002 INFO [main] (BasicTestObject.java:253) - ======BEGIN Test workflow============
06/08/2015 00:04:58,862 DEBUG [main] (DefaultUIDriver.java:300) - Opened url: http://mlpdemo.qaprod.ecollege.com/
06/08/2015 00:04:58,912 INFO [main] (BasicTestObject.java:296) - -------------BEGIN Test Method-------------------
06/08/2015 00:04:58,913 INFO [main] (BasicTestObject.java:297) - BEGIN Test Method: verifyAdminLogin
06/08/2015 00:04:58,913 INFO [main] (BasicTestObject.java:298) - -------------BEGIN Test Method-------------------
06/08/2015 00:04:58,969 DEBUG [main] (DefaultUIElement.java:980) - Waiting 60000ms for element to be displayed [Locator = {By.xpath: //input[#id='clientname']}]
06/08/2015 00:04:59,058 DEBUG [main] (DefaultUIElement.java:538) - Element is displayed [Locator = {By.xpath: //input[#id='clientname']}]
06/08/2015 00:04:59,059 DEBUG [main] (DefaultUIElement.java:992) - After 89ms, element is displayed [Locator = {By.xpath: //input[#id='clientname']}]
On Linux I get like this
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
06/08/2015 00:18:46,834 INFO [main] (BasicTestObject.java:251) - ======BEGIN Test workflow============
06/08/2015 00:18:46,839 INFO [main] (BasicTestObject.java:252) - BEGIN Test: MlpBvt
06/08/2015 00:18:46,839 INFO [main] (BasicTestObject.java:253) - ======BEGIN Test workflow============
06/08/2015 00:18:46,998 DEBUG [main] (CapturePageOnFailureListener.java:186) - CapturePageOnFailure found 2 parameters
06/08/2015 00:18:47,002 WARN [main] (DebugUIDriver.java:311) - Called quit() on debugDriver containing null uiDriver
06/08/2015 00:18:47,025 INFO [main] (BasicTestObject.java:304) - -------------END Test Method-------------------
06/08/2015 00:18:47,026 INFO [main] (BasicTestObject.java:305) - END Test Method: verifyAdminLogin
06/08/2015 00:18:47,026 INFO [main] (BasicTestObject.java:306) - -------------END Test Method-------------------
06/08/2015 00:18:47,031 INFO [main] (BasicTestObject.java:304) - -------------END Test Method-------------------
06/08/2015 00:18:47,032 INFO [main] (BasicTestObject.java:305) - END Test Method: VerifyProfessorLogin
06/08/2015 00:18:47,032 INFO [main] (BasicTestObject.java:306) - -------------END Test Method-------------------
06/08/2015 00:18:47,036 INFO [main] (BasicTestObject.java:304) - -------------END Test Method-------------------
06/08/2015 00:18:47,036 INFO [main] (BasicTestObject.java:305) - END Test Method: VerifyStudentLogin
06/08/2015 00:18:47,037 INFO [main] (BasicTestObject.java:306) - -------------END Test Method-------------------
06/08/2015 00:18:47,038 INFO [main] (BasicTestObject.java:283) - ======END Test workflow============
06/08/2015 00:18:47,038 INFO [main] (BasicTestObject.java:284) - END Test: MlpBvt
06/08/2015 00:18:47,040 INFO [main] (BasicTestObject.java:285) - ======END Test workflow============
06/08/2015 00:18:47,100 DEBUG [main] (ProcessTool.java:36) - Getting current tool for LINUX
06/08/2015 00:18:47,100 WARN [main] (ProcessTool.java:40) - Could not find ProcessTool for LINUX
06/08/2015 00:18:47,101 WARN [main] (ProcessTool.java:88) - There was no ProcessTool for LINUX
06/08/2015 00:18:47,101 DEBUG [main] (ProcessTool.java:115) - process count for There was no ProcessTool for LINUX:1
Tests run: 12, Failures: 1, Errors: 0, Skipped: 11, Time elapsed: 1.976 sec <<< FAILURE!
Results :
Failed tests:
It is mentioned as
06/08/2015 00:18:47,002 WARN [main] (DebugUIDriver.java:311) - Called quit() on debugDriver containing null uiDriver
Please provide me some technical specialities regarding this matter. Can I run this job in linux ? Please help me out
Well for unix systems you have to use Xvfb to run tests in headless mode, for jenkins you can use xvfb plugin
Simple example how to open firefox in headless mode
from xvfbwrapper import Xvfb
from selenium import webdriver
xf = Xvfb() # xf = Xvfb(1920, 1080) - will create virtual display with 1920x1080 size
xf.start()
# browser won't appear
driver = webdriver.Firefox()
driver.get("http://google.com")

Rexster refuses to start with extension but does not display errors

I have a small Rexster/Titan cluster using Cassandra. A Rexster extension is used to query the graph. I did some benchmarking and did start and stop Rexster/Titan many times. But now I run into a strange issue: Rexster refuses to start but does not display any error message.
I tried to figure out what is causing this and reduced the cluster to a single node 192.168.0.4.
If I remove my extension Rexster manages to start up.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)...... OK
(connected to 127.0.0.1:8184).
Run rexster-console.sh to connect.
but when I place my extension uber JAR in the ext folder Rexster refuses to start.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)............................
timeout exceeded (60 seconds): could not connect to 127.0.0.1:8184
See /var/lib/titan/bin/../log/rexstitan.log for Rexster log output.
If I now check rexstitan.log, as suggested by the console output, I can not find any error message.
# rexstitan.log
0 [main] INFO com.tinkerpop.rexster.Application - .:Welcome to Rexster:.
73 [main] INFO com.tinkerpop.rexster.server.RexsterProperties -
Using [/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml]
as configuration source.
78 [main] INFO com.tinkerpop.rexster.Application - Rexster is watching
[/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml] for change.
244 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
252 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
537 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
538 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
1951 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set cluster.partition=false from store features
1971 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set default timestamp provider MICRO
2019 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Generated unique-instance-id=7f0000012902-node1
2045 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
2046 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2053 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
2054 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2228 [main] INFO com.thinkaurelius.titan.diskstorage.Backend -
Initiated backend operations thread pool of size 4
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
6625 [main] INFO com.tinkerpop.rexster.RexsterApplicationGraph -
Graph [graph] - configured with allowable namespace [*:*]
The only entry that looks strange to me is the one concerning the log:
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
My exception uses the logger for debugging. You can see the instantiation an usage on github: https://github.com/sebschlicht/titan-graphity-kribble/blob/master/src/main/java/de/uniko/sebschlicht/titan/extensions/GraphityExtension.java#L22
Though Rexster failed to start there is a process with the PID displayed in the console but curl fails to connect to Rexster:
$ curl 192.168.0.4:8182
curl: (7) Failed to connect to 192.168.0.4 port 8182: Connection refused
Why doesn't Rexster throw an exception? How can I debug this situation?
edit:
I removed any log messages in my code. I removed all exceptions that may be thrown during startup. Still Rexster refuses to start with my extension and the only hint in the log files is the unidentified read marker. I have to clue what prevents Rexster from starting.
The log message is nothing to worry about.
After rebuilding the application in another project step-by-step Rexster is now able to start with the extension. During this rebuild I noticed two situations, that can cause the behaviour described:
Missing dependency
If your project depends on a second project you might use Maven to inject it as a dependency. However, if you use
mvn clean package
to build the extension's JAR file it does not contain this dependency by default. You need to use a Maven plugin (e.g. maven-shade-plugin) to create a shaded JAR that contains all the dependencies your extension needs. Set the dependency scope to provided for all Titan/Rexster/Blueprints related dependencies. Use the shaded uber-JAR to deploy the extension to Rexster.
However, this was not new to me and should not have caused the problem in my case. There might be more situations that cause this problem or maybe there was a problem with Maven that messed up the shaded JAR. Feel free to browse the commit on github to catch this voodoo.
Missing extension
Another cause of this behaviour is a missing extension.
If you specify an extension in the com.tinkerpop.rexster.extension.RexsterExtension resource file, that is not present on startup, Rexster does neither log nor throw an exception, but refuses to start.

Resources