Unable to set up my own Stanford CoreNLP server with error "Could not delete shutdown key file" - nlp

I try to set up my own Stanford CoreNLP server following the official guide. However, I am not able to start the server using the following command:
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
I paste the error messages below:
my_server_name$ java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - setting default constituency parser
[main] INFO CoreNLP - warning: cannot find edu/stanford/nlp/models/srparser/englishSR.ser.gz
[main] INFO CoreNLP - using: edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz instead
[main] INFO CoreNLP - to use shift reduce parser download English models jar from:
[main] INFO CoreNLP - http://stanfordnlp.github.io/CoreNLP/download.html
Exception in thread "main" java.lang.IllegalStateException: Could not delete shutdown key file
at edu.stanford.nlp.pipeline.StanfordCoreNLPServer.<init>(StanfordCoreNLPServer.java:195)
at edu.stanford.nlp.pipeline.StanfordCoreNLPServer.main(StanfordCoreNLPServer.java:1323)
[Thread-0] INFO CoreNLP - CoreNLP Server is shutting down.
The main problem is the IllegalSstateException: Could not delete shutdown key file. I just wonder whether the cause of this problem is the sudo access. The official guide doesn't explicitly state this command needs sudo access.
I want to ask 1) whether the above command requires the sudo access and 2) if that command doesn't need sudo access, what could be the potential error for my IllegalSstateException.
Thanks.
PS: I am running on a server with Ubuntu 16.04.3 LTS.

This error happens when the shutdown key file already exists on your filesystem, you're starting a new CoreNLP server instance, and it can't delete the old shutdown key file. Are you running the server as two different users?
More generally, do you have permissions to the directory stored in java property java.io.tmpdir? This is, traditionally, /tmp on Linux machines. The shutdown key is stored in:
System.getProperty("java.io.tmpdir") + File.separator + "corenlp.shutdown"
So, for a Linux system:
/tmp/corenlp.shutdown
The error says that this file exists, and cannot be deleted by Java. You should check your permissions on this file, and that should help you debug what's wrong.
An easy workaround, in the worst case, is to set the tmpdir yourself when starting the server. For example:
java -Djava.io.tmpdir=/path/to/tmp -mx4g edu.stanford.nlp.pipeline.StanfordCoreNLPServer 9000

Related

arangodb starter mode does not start

I have d/l'd arangodb3-linux-3.9.2 from GIT on Centos 7. I created a database dir and ran the README instructions for a standalone start. The first time it runs, I get 100 failures, the key INFO log lines seem to be
... [INFO] server started component=arangodb pid=49827 type=single
... [INFO] Wait on 49827 returned component=arangodb exit-status=1 trap-cause=-1
It creates the log file, setup.json and a single8529 dir in the database dir I sped'd. Is it just taking too long to start? The whole 100 fails take about 1 or 2 seconds.
If I try to run it again with the same README instructions, the next time I get this error:
... [FATAL] Failed to run service error="open /.../single8529/data/ENGINE: no such file"
I have also tried with --starter.host 127.0.0.1 -- to simplify
Also I and can confirm that port 8529 is open
I couldn't get arangodb 'starter' according to their README to work, but this does start the server:
arangod --database.directory MYDIR --rocksdb.max-background-jobs 4

SPARK YARN: cannot send job from client (org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032)

I'm trying to send spark job to yarn (without HDFS) in HA mode.
For submitting I'm using org.apache.spark.deploy.SparkSubmit.
When I send request from machine with active Resource Manager, it works well. But if I' trying to send from machine with standby Resource Manager, job fails with error:
DEBUG org.apache.hadoop.ipc.Client - Connecting to spark2-node-dev/10.10.10.167:8032
DEBUG org.apache.hadoop.ipc.Client - Connecting to /0.0.0.0:8032
org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep
However, when I send request via command line (spark-submit), it works well through both active and standby machine.
What can cause the problem?
P.S. Use the same parameters for both type of sending job: org.apache.spark.deploy.SparkSubmit and spark-submit command line request. And properties yarn.resourcemanager.hostname.rm_id defined for all rm hosts
The problem was with absence of yarn-site.xml within class path for spark-submitter jar. Actually spark submitter jar does not take to account YARN_CONF_DIR or HADOOP_CONF_DIR env var, so cannot see yarn-site.
One solution that I found was to put yarn-site into classpath of jar.

HDInsight Emulator not running on Windows / connection exception

I'm trying to setup a HDInsight emulator on a Windows 8.1 PC following these instructions: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-emulator-get-started/
When trying to run a MapReduce job, I get a connection error.
How can I solve or further investigate this issue?
Details below.
Prerequisites:
Installed Azure Powershell and Azure SDK for VS 2015
Installed HDInsight Emulator for Azure incl. Hortonworks Data Platform
Started local hdp services (13 services running)
Connected Visual Studio to Emulator (had to follow troubleshooting point 2: replacing IP addresses in core-site.xml with '*' due to dynamic IP)
Created directories and copied text files as suggested
Problem:
When trying to run the first example, I get the following error:
16/01/11 10:36:39 INFO mapreduce.Job: Job job_1452503376359_0003 failed with state FAILED due to: Application application_1452503376359_0003 failed 2 times due to AM Container for appattempt_1452503376359_0003_000002 exited with exitCode: -1000 due to: Call From EH3HOST/192.168.56.1 to EH3HOST:8020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
.Failing this attempt.. Failing the application.
16/01/11 10:36:39 INFO mapreduce.Job: Counters: 0
The following worked for me:
Search for XML files containing <your own host name>:8020 inside the c:\hdp\hdp-<Version Number>\etc\hadoop\ folder. (e.g. EH3HOST:8020)
You should find at least
mapred-site.xml
core-site.xml
yarn-site.xml
Replace all occurrences within these files with 127.0.0.1:8020.

How to connect master and slaves in Apache-Spark? (Standalone Mode)

I'm using Spark Standalone Mode tutorial page to install Spark in Standalone mode.
1- I have started a master by:
./sbin/start-master.sh
2- I have started a worker by:
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://ubuntu:7077
Note: spark://ubuntu:7077 is my master name, which I can see it in Master-WebUI.
Problem: By second command, a worker started successfully. But it couldn't associate with master. It tries repeatedly and then give this message:
15/02/08 11:30:04 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster#ubuntu:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: ubuntu/127.0.1.1:7077
15/02/08 11:30:04 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$RegisterWorker] from Actor[akka://sparkWorker/user/Worker#-1296628173] to Actor[akka://sparkWorker/deadLetters] was not delivered. [20] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
15/02/08 11:31:15 ERROR Worker: All masters are unresponsive! Giving up.
What is the problem?
Thanks
I usually start from spark-env.sh template. And I set, properties that I need. For simple cluster you need:
SPARK_MASTER_IP
Then, create a file called "slaves" in the same directory as spark-env.sh and slaves ip's (one per line). Assure you reach all slaves through ssh.
Finally, copy this configuration in every machine of your cluster. Then start the entire cluster executing start-all.sh script and try spark-shell to check your configuration.
> sbin/start-all.sh
> bin/spark-shell
You can set export SPARK_LOCAL_IP="You-IP" #to set the IP address Spark binds to on this node in $SPARK_HOME/conf/spark-env.sh
In my case, using spark 2.4.7 in standalone mode, I've created a passwordless ssh key using ssh-keygen, but still got asked for worker password when starting the cluster.
What I did was follow the instructions here
https://www.cyberciti.biz/faq/how-to-set-up-ssh-keys-on-linux-unix/
This line solved the problem:
ssh-copy-id -i $HOME/.ssh/id_rsa.pub user#server-ip

Rexster refuses to start with extension but does not display errors

I have a small Rexster/Titan cluster using Cassandra. A Rexster extension is used to query the graph. I did some benchmarking and did start and stop Rexster/Titan many times. But now I run into a strange issue: Rexster refuses to start but does not display any error message.
I tried to figure out what is causing this and reduced the cluster to a single node 192.168.0.4.
If I remove my extension Rexster manages to start up.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)...... OK
(connected to 127.0.0.1:8184).
Run rexster-console.sh to connect.
but when I place my extension uber JAR in the ext folder Rexster refuses to start.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)............................
timeout exceeded (60 seconds): could not connect to 127.0.0.1:8184
See /var/lib/titan/bin/../log/rexstitan.log for Rexster log output.
If I now check rexstitan.log, as suggested by the console output, I can not find any error message.
# rexstitan.log
0 [main] INFO com.tinkerpop.rexster.Application - .:Welcome to Rexster:.
73 [main] INFO com.tinkerpop.rexster.server.RexsterProperties -
Using [/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml]
as configuration source.
78 [main] INFO com.tinkerpop.rexster.Application - Rexster is watching
[/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml] for change.
244 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
252 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
537 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
538 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
1951 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set cluster.partition=false from store features
1971 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set default timestamp provider MICRO
2019 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Generated unique-instance-id=7f0000012902-node1
2045 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
2046 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2053 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
2054 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2228 [main] INFO com.thinkaurelius.titan.diskstorage.Backend -
Initiated backend operations thread pool of size 4
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
6625 [main] INFO com.tinkerpop.rexster.RexsterApplicationGraph -
Graph [graph] - configured with allowable namespace [*:*]
The only entry that looks strange to me is the one concerning the log:
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
My exception uses the logger for debugging. You can see the instantiation an usage on github: https://github.com/sebschlicht/titan-graphity-kribble/blob/master/src/main/java/de/uniko/sebschlicht/titan/extensions/GraphityExtension.java#L22
Though Rexster failed to start there is a process with the PID displayed in the console but curl fails to connect to Rexster:
$ curl 192.168.0.4:8182
curl: (7) Failed to connect to 192.168.0.4 port 8182: Connection refused
Why doesn't Rexster throw an exception? How can I debug this situation?
edit:
I removed any log messages in my code. I removed all exceptions that may be thrown during startup. Still Rexster refuses to start with my extension and the only hint in the log files is the unidentified read marker. I have to clue what prevents Rexster from starting.
The log message is nothing to worry about.
After rebuilding the application in another project step-by-step Rexster is now able to start with the extension. During this rebuild I noticed two situations, that can cause the behaviour described:
Missing dependency
If your project depends on a second project you might use Maven to inject it as a dependency. However, if you use
mvn clean package
to build the extension's JAR file it does not contain this dependency by default. You need to use a Maven plugin (e.g. maven-shade-plugin) to create a shaded JAR that contains all the dependencies your extension needs. Set the dependency scope to provided for all Titan/Rexster/Blueprints related dependencies. Use the shaded uber-JAR to deploy the extension to Rexster.
However, this was not new to me and should not have caused the problem in my case. There might be more situations that cause this problem or maybe there was a problem with Maven that messed up the shaded JAR. Feel free to browse the commit on github to catch this voodoo.
Missing extension
Another cause of this behaviour is a missing extension.
If you specify an extension in the com.tinkerpop.rexster.extension.RexsterExtension resource file, that is not present on startup, Rexster does neither log nor throw an exception, but refuses to start.

Resources