I thought this would be simple. I guess not.
I have an external hard drive mounted at /root/storage - OK
I moved the data directory from /var/lib/cassandra/ to /root/storage - OK
I then created a symbolic link out of /var/lib/cassandra pointing to where the directory is now.....so...... ln -s /root/storage/data /var/lib/cassandra - OK
Now I am unable to start cassandra. I am getting this error in /var/log/cassandra/system.log:
INFO [main] 2013-02-15 10:08:36,329 CacheService.java (line 166)
Scheduling row cache save to each 0 seconds (going to save all keys).
ERROR [main] 2013-02-15 10:08:36,366 FileUtils.java (line 373)
Stopping the gossiper and the RPC server ERROR [main] 2013-02-15
10:08:36,367 CassandraDaemon.java (line 387) Exception encountered
during startup java.lang.IllegalStateException: No configured daemon
at org.apache.cassandra.service.StorageService.stopRPCServer(StorageService.java:314)
at org.apache.cassandra.io.util.FileUtils.handleFSError(FileUtils.java:375)
at org.apache.cassandra.db.Directories.(Directories.java:113)
at org.apache.cassandra.db.Directories.create(Directories.java:91)
at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:403)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:174)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:370)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:413)
[root#cassandra-new6 storage]# vi
/usr/share/cassandra/default.conf/cassandra.yaml
The permissions are exactly the same on this directory. All file permissiosn are teh same too. Any ideas would be appreciated.
When I get rid of the sym link and move the data directory back, everything works again.
The symlink regression is fixed in Cassandra 1.2.2. https://issues.apache.org/jira/browse/CASSANDRA-5185
Instead of creating a sym link, you can change where Cassandra looks for the data directory in the cassandra.yaml file by changing the data_file_directories parameter.
Related
I am using these steps to use apache pulsar on docker: https://github.com/streamnative/tgip/blob/master/episodes/001/demo.md
I was able to use these steps before to install and use pulsar but for some reason now when am creating a directory, it is going to write protected and pulsar zookeeper container is exiting with following logs as soon as it is created:
ERROR org.apache.zookeeper.server.ZooKeeperServerMain - Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:136) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:137) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:112) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:67) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:140) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
Unable to access datadir, exiting abnormally
23:36:15.223 [main] INFO org.apache.zookeeper.audit.ZKAuditProvider - ZooKeeper audit is disabled.
23:36:15.226 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 3
23:36:15.196 [PurgeTask] ERROR org.apache.zookeeper.server.DatadirCleanupManager - Error occurred while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:136) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:80) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:141) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
23:36:15.229 [PurgeTask] INFO org.apache.zookeeper.server.DatadirCleanupManager - Purge task completed
I have made sure that SELinux is disabled and tried changing permission using chmod 777 data/ and every other step available to resolve this but still unable to find any. Please help me with the possible resolution.
cassandra 3.6
openjdk version "1.8.0_91"
when I start cassandra service will throw this error.
INFO 05:57:10 Initializing system_schema.keyspaces
INFO 05:57:10 Initializing system_schema.tables
ERROR 05:57:10 Failed to classify files in /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f
Some old files are missing but the txn log is still there and not completed
Files in folder:
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-CompressionInfo.db
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-Data.db
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-Digest.crc32
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-Filter.db
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-Index.db
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-Statistics.db
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-Summary.db
/var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/ma-404-big-TOC.txt
Txn: [ma_txn_compaction_4c1ecb40-0ff9-11e8-a162-c50ddd47b4bb.log in /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f]
what does it means and what should I have to do to solve this issue.
Thanks in advance
Finally I got the answer.
system files are corrupted and getting issues.
so you need to delete all direcory inside "/var/lib/cassandra/data/" except your keyspace.
*Note: Don't remove your keyspace data.
I try to set up my own Stanford CoreNLP server following the official guide. However, I am not able to start the server using the following command:
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
I paste the error messages below:
my_server_name$ java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - setting default constituency parser
[main] INFO CoreNLP - warning: cannot find edu/stanford/nlp/models/srparser/englishSR.ser.gz
[main] INFO CoreNLP - using: edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz instead
[main] INFO CoreNLP - to use shift reduce parser download English models jar from:
[main] INFO CoreNLP - http://stanfordnlp.github.io/CoreNLP/download.html
Exception in thread "main" java.lang.IllegalStateException: Could not delete shutdown key file
at edu.stanford.nlp.pipeline.StanfordCoreNLPServer.<init>(StanfordCoreNLPServer.java:195)
at edu.stanford.nlp.pipeline.StanfordCoreNLPServer.main(StanfordCoreNLPServer.java:1323)
[Thread-0] INFO CoreNLP - CoreNLP Server is shutting down.
The main problem is the IllegalSstateException: Could not delete shutdown key file. I just wonder whether the cause of this problem is the sudo access. The official guide doesn't explicitly state this command needs sudo access.
I want to ask 1) whether the above command requires the sudo access and 2) if that command doesn't need sudo access, what could be the potential error for my IllegalSstateException.
Thanks.
PS: I am running on a server with Ubuntu 16.04.3 LTS.
This error happens when the shutdown key file already exists on your filesystem, you're starting a new CoreNLP server instance, and it can't delete the old shutdown key file. Are you running the server as two different users?
More generally, do you have permissions to the directory stored in java property java.io.tmpdir? This is, traditionally, /tmp on Linux machines. The shutdown key is stored in:
System.getProperty("java.io.tmpdir") + File.separator + "corenlp.shutdown"
So, for a Linux system:
/tmp/corenlp.shutdown
The error says that this file exists, and cannot be deleted by Java. You should check your permissions on this file, and that should help you debug what's wrong.
An easy workaround, in the worst case, is to set the tmpdir yourself when starting the server. For example:
java -Djava.io.tmpdir=/path/to/tmp -mx4g edu.stanford.nlp.pipeline.StanfordCoreNLPServer 9000
This is the second time my commitlog is corrupted, and the server refuses to start. What worries me is that I get these error issues even if no update were made to the database.
My config says that commitlog are synced every 10s seconds, so how can a file be corrupt unless a crash occurs within these 10 seconds?
Is this a Cassandra bug? Or by design, i.e. bad design?
I am using 3.4 on Windows 10, Datastax installer.
In the stdout log, the last part is
INFO 06:17:39 Replaying C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812251.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471411951134.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471454506802.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471532812678.log
ERROR 06:17:39 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log
at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:373) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.4.0.jar:3.4.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.4.0.jar:3.4.0]
I have seen similar errors. This happens, when Cassandra process gets crahed may be due to OOM. Run "dmesg" and check if it was killed due to OOM. In that case there is possibility that commit log it was writing to was corrupted or its of 0kb file (check size of above file in error), and it throws the above error when Cassandra is restarted and it replays that file.
I have a small Rexster/Titan cluster using Cassandra. A Rexster extension is used to query the graph. I did some benchmarking and did start and stop Rexster/Titan many times. But now I run into a strange issue: Rexster refuses to start but does not display any error message.
I tried to figure out what is causing this and reduced the cluster to a single node 192.168.0.4.
If I remove my extension Rexster manages to start up.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)...... OK
(connected to 127.0.0.1:8184).
Run rexster-console.sh to connect.
but when I place my extension uber JAR in the ext folder Rexster refuses to start.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)............................
timeout exceeded (60 seconds): could not connect to 127.0.0.1:8184
See /var/lib/titan/bin/../log/rexstitan.log for Rexster log output.
If I now check rexstitan.log, as suggested by the console output, I can not find any error message.
# rexstitan.log
0 [main] INFO com.tinkerpop.rexster.Application - .:Welcome to Rexster:.
73 [main] INFO com.tinkerpop.rexster.server.RexsterProperties -
Using [/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml]
as configuration source.
78 [main] INFO com.tinkerpop.rexster.Application - Rexster is watching
[/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml] for change.
244 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
252 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
537 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
538 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
1951 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set cluster.partition=false from store features
1971 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set default timestamp provider MICRO
2019 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Generated unique-instance-id=7f0000012902-node1
2045 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
2046 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2053 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
2054 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2228 [main] INFO com.thinkaurelius.titan.diskstorage.Backend -
Initiated backend operations thread pool of size 4
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
6625 [main] INFO com.tinkerpop.rexster.RexsterApplicationGraph -
Graph [graph] - configured with allowable namespace [*:*]
The only entry that looks strange to me is the one concerning the log:
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
My exception uses the logger for debugging. You can see the instantiation an usage on github: https://github.com/sebschlicht/titan-graphity-kribble/blob/master/src/main/java/de/uniko/sebschlicht/titan/extensions/GraphityExtension.java#L22
Though Rexster failed to start there is a process with the PID displayed in the console but curl fails to connect to Rexster:
$ curl 192.168.0.4:8182
curl: (7) Failed to connect to 192.168.0.4 port 8182: Connection refused
Why doesn't Rexster throw an exception? How can I debug this situation?
edit:
I removed any log messages in my code. I removed all exceptions that may be thrown during startup. Still Rexster refuses to start with my extension and the only hint in the log files is the unidentified read marker. I have to clue what prevents Rexster from starting.
The log message is nothing to worry about.
After rebuilding the application in another project step-by-step Rexster is now able to start with the extension. During this rebuild I noticed two situations, that can cause the behaviour described:
Missing dependency
If your project depends on a second project you might use Maven to inject it as a dependency. However, if you use
mvn clean package
to build the extension's JAR file it does not contain this dependency by default. You need to use a Maven plugin (e.g. maven-shade-plugin) to create a shaded JAR that contains all the dependencies your extension needs. Set the dependency scope to provided for all Titan/Rexster/Blueprints related dependencies. Use the shaded uber-JAR to deploy the extension to Rexster.
However, this was not new to me and should not have caused the problem in my case. There might be more situations that cause this problem or maybe there was a problem with Maven that messed up the shaded JAR. Feel free to browse the commit on github to catch this voodoo.
Missing extension
Another cause of this behaviour is a missing extension.
If you specify an extension in the com.tinkerpop.rexster.extension.RexsterExtension resource file, that is not present on startup, Rexster does neither log nor throw an exception, but refuses to start.