I am learning how to integrate pubsub with elasticsearch. There are various options like pubsubbeat, Google_pubsub input plugin, Google Cloud Pub/Sub Output Plugin.
I am currently trying to use pubsubbeat and stucked after running the command " ./pubsubbeat -c pubsubbeat.yml -e -d "*" " as suggested. Log of console is as follows
2019-05-23T14:42:19.949+0100 INFO instance/beat.go:468 Home path: [/home/amishra/pubsubbeat-linux-amd64] Config path: [/home/amishra/pubsubbeat-linux-amd64] Data path: [/home/amishra/pubsubbeat-linux-amd64/data] Logs path: [/home/amishra/pubsubbeat-linux-amd64/logs]
2019-05-23T14:42:19.949+0100 DEBUG [beat] instance/beat.go:495 Beat metadata path: /home/amishra/pubsubbeat-linux-amd64/data/meta.json
2019-05-23T14:42:19.949+0100 INFO instance/beat.go:475 Beat UUID: 4bd6119e-603a-426c-9d5b-6ac588bb000e
2019-05-23T14:42:19.949+0100 INFO instance/beat.go:213 Setup Beat: pubsubbeat; Version: 6.2.2
2019-05-23T14:42:19.949+0100 DEBUG [beat] instance/beat.go:230 Initializing output plugins
2019-05-23T14:42:19.949+0100 DEBUG [processors] processors/processor.go:49 Processors:
2019-05-23T14:42:19.952+0100 INFO pipeline/module.go:76 Beat name: allspark
2019-05-23T14:42:19.952+0100 INFO [PubSub: dev/elk-logstash-poc/logstash-poc] beater/pubsubbeat.go:54 config retrieved: &{Project:dev Topic:elk-logstash-poc CredentialsFile:/home/amishra/key/key.json Subscription:{Name:logstash-poc RetainAckedMessages:false RetentionDuration:5h0m0s} Json:{Enabled:false AddErrorKey:false}}
On second thought, I tried solution 2 but was getting below error and haven't able to resolve yet
io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
WARNING: [io.grpc.internal.ManagedChannelImpl-1] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host pubsub.googleapis.com, cause=java.net.UnknownHostException: pubsub.googleapis.com
Any lead on how to make thing working will be great help
The issue got resolved in unexpected way i.e. installing and deploying on fresh machine. Root cause is still unknown.
Related
I have a Linux VM hosted on Azure. On this VM I "installed" the standalone version of Keycloak 19.0.2. When I connect to the VM with SSH, I can simply start the server with bin/kc.sh start-dev. This works without any problems.
Now I want to start the Keycloak server automatically on the VM startup. I tried this with Crontab. This did not work because the VM resets the Crontab on each restart.
Then I tried to directly start it from Azure with a Custom Script for Linux. Since I had trouble setting it up, I started playing around with the Run command feature in the Azure portal. When I run the command sudo /home/my_user/keycloak/bin/kc.sh start-dev I get the following output:
2022-09-14 14:19:32,839 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: false
2022-09-14 14:19:34,670 INFO [org.keycloak.common.crypto.CryptoIntegration] (main) Detected crypto provider: org.keycloak.crypto.def.DefaultCryptoProvider
2022-09-14 14:19:36,475 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-09-14 14:19:36,483 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-09-14 14:19:36,553 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-09-14 14:19:37,251 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.9.Final
2022-09-14 14:19:37,886 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: node_884187, Site name: null
2022-09-14 14:19:41,657 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (development) mode
2022-09-14 14:19:41,657 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Unable to start HTTP server
2022-09-14 14:19:41,658 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: io.quarkus.runtime.QuarkusBindException
2022-09-14 14:19:41,658 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
TLDR: The server starts perfectly fine, when I manually start it via SSH. When I try to start the server with the Azure portal (Run command) it does not work.
Issue:
Database failed to start, exited with status 'KILLED", Logs end with: Caused by: ... bind(..) failed: Address already in use
Very similar to these:
Neo4j Desktop DB failed to start with with status 'KILLED'
https://community.neo4j.com/t/database-failed-to-start/5612
https://community.neo4j.com/t/database-failed-to-start-db-database-ee8e1c21-23d9-40fd-8a4e-02fd55a0d324-v4-1-0-exited-with-status-killed-check-the-logs/21045
Logs end at:
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
Things I've tried:
Killing all processes associated with neo4j
searching for default port lsof :11002/3
reinstalling
some config files you don't seem to be able to delete in .config
Os and Version Data
OS: Linux pop-os 5.4.0-7634-generic #38~1596560323~20.04~7719dbd-Ubuntu SMP Tue Aug 4 19:12:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
neo4j version: 4.1.0
desktop version: 1.3.4
neo4j.log
Directories in use:
home: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
config: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
logs: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
plugins: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
import: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
data: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
run: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
Directories in use:
home: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
config: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
logs: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
plugins: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
import: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
data: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
run: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2020-08-21 19:04:24.399+0000 INFO Note that since you did not explicitly set the port in dbms.connector.bolt.advertised_address Neo4j automatically set it to 11002 to match dbms.connector.bolt.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO Note that since you did not explicitly set the port in dbms.connector.http.advertised_address Neo4j automatically set it to 11003 to match dbms.connector.http.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO Starting...
Directories in use:
home: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
config: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
logs: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
plugins: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
import: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
data: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
run: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2020-08-21 19:04:24.399+0000 INFO Note that since you did not explicitly set the port in dbms.connector.bolt.advertised_address Neo4j automatically set it to 11002 to match dbms.connector.bolt.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO Note that since you did not explicitly set the port in dbms.connector.http.advertised_address Neo4j automatically set it to 11003 to match dbms.connector.http.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO Starting...
2020-08-21 19:04:26.747+0000 INFO ======== Neo4j 4.1.0 ========
Directories in use:
home: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
config: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
logs: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
plugins: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
import: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
data: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
run: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2020-08-21 19:04:24.399+0000 INFO Note that since you did not explicitly set the port in dbms.connector.bolt.advertised_address Neo4j automatically set it to 11002 to match dbms.connector.bolt.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO Note that since you did not explicitly set the port in dbms.connector.http.advertised_address Neo4j automatically set it to 11003 to match dbms.connector.http.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO Starting...
2020-08-21 19:04:26.747+0000 INFO ======== Neo4j 4.1.0 ========
2020-08-21 19:04:32.005+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data/databases
java.lang.RuntimeException: Error starting Neo4j database server at /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'Server[backup-server]' was successfully initialized, but failed to start. Please see the attached cause exception "bind(..) failed: Address already in use".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
... 6 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
2020-08-21 19:04:32.006+0000 INFO Neo4j Server shutdown initiated by request
debug.log
2020-08-21 19:04:24.897+0000 INFO [o.n.i.d.DiagnosticsManager] ********************************************************************************
2020-08-21 19:04:24.898+0000 INFO [o.n.i.d.DiagnosticsManager] [ System diagnostics ]
2020-08-21 19:04:24.899+0000 INFO [o.n.i.d.DiagnosticsManager] ********************************************************************************
2020-08-21 19:04:24.902+0000 INFO [o.n.i.d.DiagnosticsManager] --------------------------------------------------------------------------------
2020-08-21 19:04:24.902+0000 INFO [o.n.i.d.DiagnosticsManager] [ System memory information ]
2020-08-21 19:04:24.902+0000 INFO [o.n.i.d.DiagnosticsManager] --------------------------------------------------------------------------------
...
2020-08-21 19:04:25.482+0000 INFO [c.n.c.c.TransactionBackupServiceProvider] Binding backup service on address 127.0.0.1:6362
2020-08-21 19:04:26.747+0000 INFO [c.n.s.e.EnterpriseNeoWebServer] ======== Neo4j 4.1.0 ========
2020-08-21 19:04:26.824+0000 INFO [o.n.b.BoltServer] Bolt server loaded
2020-08-21 19:04:26.930+0000 ERROR [c.n.c.n.Server] backup-server: cannot bind to '127.0.0.1:6362' with transport 'EpollServerSocketChannel'. bind(..) failed: Address already in use
io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
2020-08-21 19:04:26.943+0000 ERROR [o.n.g.f.DatabaseManagementServiceFactory] Error starting Neo4j database server at /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data/databases Component 'Server[backup-server]' was successfully initialized, but failed to start. Please see the attached cause exception "bind(..) failed: Address already in use".
org.neo4j.kernel.lifecycle.LifecycleException: Component 'Server[backup-server]' was successfully initialized, but failed to start. Please see the attached cause exception "bind(..) failed: Address already in use".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
2020-08-21 19:04:26.944+0000 INFO [o.n.g.f.DatabaseManagementServiceFactory] Shutdown started
There was a lot of address already in use going on here, so it was kind of hard to parse, but this bit in the debug log turned out to be the important part:
2020-08-21 19:04:25.482+0000 INFO [c.n.c.c.TransactionBackupServiceProvider] Binding backup service on address 127.0.0.1:6362
2020-08-21 19:04:26.747+0000 INFO [c.n.s.e.EnterpriseNeoWebServer] ======== Neo4j 4.1.0 ========
2020-08-21 19:04:26.824+0000 INFO [o.n.b.BoltServer] Bolt server loaded
2020-08-21 19:04:26.930+0000 ERROR [c.n.c.n.Server] backup-server: cannot bind to '127.0.0.1:6362' with transport 'EpollServerSocketChannel'. bind(..) failed: Address already in use
searched for the port:
$ lsof -i :6362
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 530156 user 427u IPv6 3670839 0t0 TCP localhost:6362 (LISTEN)
$ kill 530156
Looks like desktop leaves some java runnin' out in the fields there...
After that DB startup was all good.
I am having an issue trying to connect to a Kafka instance that uses Kerberos SASL authentication using the node-rdkafka npm module, and the lack of documentation and examples out there are making things very difficult to solve the issue. I am using the Alpine setup found here
to pull in all of the necessary librdkafka and node-rdkafka dependencies and create an image. I am trying to run my container with the following configuration object for a node-rdkafka producer to connect to Kafka instance:
'debug': 'all',
'client.id': 'kafka',
'metadata.broker.list': KAFKA_DOMAINS,
'sasl.kerberos.principal': '<id>#<REALM>',
'sasl.kerberos.keytab': '/path/to/<id>.keytab',
'sasl.kerberos.service.name': 'kafka',
'security.protocol': 'sasl_plaintext',
'sasl.kerberos.kinit.cmd': 'kinit -kt /path/to/<id>.keytab <id>#<REALM>',
'dr_cb': true,
'event_cb': true
When I try to run the container, it hangs up and then throws the following error:
[2019-02-05T17:09:59.637Z] ERROR: kafka/7 on 95c927d11411: broker transport failure (err.code=-195)
Error: Local: Broker transport failure
at Function.createLibrdkafkaError [as create] (/opt/rdkafka/node_modules/node-rdkafka/lib/error.js:261:10)
at /opt/rdkafka/node_modules/node-rdkafka/lib/client.js:339:28
Here is the output from the features and libkafkaversion log statements:
[2019-02-05T17:09:29.519Z] INFO: kafka/7 on 95c927d11411:
[ 'gzip',
'snappy',
'ssl',
'sasl',
'regex',
'lz4',
'sasl_gssapi',
'sasl_plain',
'sasl_scram',
'plugins' ]
[2019-02-05T17:09:29.521Z] INFO: kafka/7 on 95c927d11411: 0.11.5
What am I missing that I am unable to connect? Am I missing some environment variables? I know that the Java implementation of librdkafka involves passing the REALM, KDC_HOST, and some .conf files (krb5 and jaas). Do I have to set up some kind of environment variable? I can find no examples or solutions.
I try to set up my own Stanford CoreNLP server following the official guide. However, I am not able to start the server using the following command:
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
I paste the error messages below:
my_server_name$ java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - setting default constituency parser
[main] INFO CoreNLP - warning: cannot find edu/stanford/nlp/models/srparser/englishSR.ser.gz
[main] INFO CoreNLP - using: edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz instead
[main] INFO CoreNLP - to use shift reduce parser download English models jar from:
[main] INFO CoreNLP - http://stanfordnlp.github.io/CoreNLP/download.html
Exception in thread "main" java.lang.IllegalStateException: Could not delete shutdown key file
at edu.stanford.nlp.pipeline.StanfordCoreNLPServer.<init>(StanfordCoreNLPServer.java:195)
at edu.stanford.nlp.pipeline.StanfordCoreNLPServer.main(StanfordCoreNLPServer.java:1323)
[Thread-0] INFO CoreNLP - CoreNLP Server is shutting down.
The main problem is the IllegalSstateException: Could not delete shutdown key file. I just wonder whether the cause of this problem is the sudo access. The official guide doesn't explicitly state this command needs sudo access.
I want to ask 1) whether the above command requires the sudo access and 2) if that command doesn't need sudo access, what could be the potential error for my IllegalSstateException.
Thanks.
PS: I am running on a server with Ubuntu 16.04.3 LTS.
This error happens when the shutdown key file already exists on your filesystem, you're starting a new CoreNLP server instance, and it can't delete the old shutdown key file. Are you running the server as two different users?
More generally, do you have permissions to the directory stored in java property java.io.tmpdir? This is, traditionally, /tmp on Linux machines. The shutdown key is stored in:
System.getProperty("java.io.tmpdir") + File.separator + "corenlp.shutdown"
So, for a Linux system:
/tmp/corenlp.shutdown
The error says that this file exists, and cannot be deleted by Java. You should check your permissions on this file, and that should help you debug what's wrong.
An easy workaround, in the worst case, is to set the tmpdir yourself when starting the server. For example:
java -Djava.io.tmpdir=/path/to/tmp -mx4g edu.stanford.nlp.pipeline.StanfordCoreNLPServer 9000
I have a small Rexster/Titan cluster using Cassandra. A Rexster extension is used to query the graph. I did some benchmarking and did start and stop Rexster/Titan many times. But now I run into a strange issue: Rexster refuses to start but does not display any error message.
I tried to figure out what is causing this and reduced the cluster to a single node 192.168.0.4.
If I remove my extension Rexster manages to start up.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)...... OK
(connected to 127.0.0.1:8184).
Run rexster-console.sh to connect.
but when I place my extension uber JAR in the ext folder Rexster refuses to start.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)............................
timeout exceeded (60 seconds): could not connect to 127.0.0.1:8184
See /var/lib/titan/bin/../log/rexstitan.log for Rexster log output.
If I now check rexstitan.log, as suggested by the console output, I can not find any error message.
# rexstitan.log
0 [main] INFO com.tinkerpop.rexster.Application - .:Welcome to Rexster:.
73 [main] INFO com.tinkerpop.rexster.server.RexsterProperties -
Using [/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml]
as configuration source.
78 [main] INFO com.tinkerpop.rexster.Application - Rexster is watching
[/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml] for change.
244 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
252 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
537 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
538 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
1951 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set cluster.partition=false from store features
1971 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set default timestamp provider MICRO
2019 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Generated unique-instance-id=7f0000012902-node1
2045 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
2046 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2053 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
2054 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2228 [main] INFO com.thinkaurelius.titan.diskstorage.Backend -
Initiated backend operations thread pool of size 4
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
6625 [main] INFO com.tinkerpop.rexster.RexsterApplicationGraph -
Graph [graph] - configured with allowable namespace [*:*]
The only entry that looks strange to me is the one concerning the log:
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
My exception uses the logger for debugging. You can see the instantiation an usage on github: https://github.com/sebschlicht/titan-graphity-kribble/blob/master/src/main/java/de/uniko/sebschlicht/titan/extensions/GraphityExtension.java#L22
Though Rexster failed to start there is a process with the PID displayed in the console but curl fails to connect to Rexster:
$ curl 192.168.0.4:8182
curl: (7) Failed to connect to 192.168.0.4 port 8182: Connection refused
Why doesn't Rexster throw an exception? How can I debug this situation?
edit:
I removed any log messages in my code. I removed all exceptions that may be thrown during startup. Still Rexster refuses to start with my extension and the only hint in the log files is the unidentified read marker. I have to clue what prevents Rexster from starting.
The log message is nothing to worry about.
After rebuilding the application in another project step-by-step Rexster is now able to start with the extension. During this rebuild I noticed two situations, that can cause the behaviour described:
Missing dependency
If your project depends on a second project you might use Maven to inject it as a dependency. However, if you use
mvn clean package
to build the extension's JAR file it does not contain this dependency by default. You need to use a Maven plugin (e.g. maven-shade-plugin) to create a shaded JAR that contains all the dependencies your extension needs. Set the dependency scope to provided for all Titan/Rexster/Blueprints related dependencies. Use the shaded uber-JAR to deploy the extension to Rexster.
However, this was not new to me and should not have caused the problem in my case. There might be more situations that cause this problem or maybe there was a problem with Maven that messed up the shaded JAR. Feel free to browse the commit on github to catch this voodoo.
Missing extension
Another cause of this behaviour is a missing extension.
If you specify an extension in the com.tinkerpop.rexster.extension.RexsterExtension resource file, that is not present on startup, Rexster does neither log nor throw an exception, but refuses to start.