Cassandra stopped.
when i restart Cassandra using "service cassandra start" or "service cassandra restart", i get the following error(from "/var/log/cassandra/system.log"):
ERROR [main] 2014-11-14 02:08:52,379 CommitLogReplayer.java (line 304) Unexpected error deserializing mutation; saved to /tmp/mutation3145492124947244713dat and ignored. This may be caused by replaying a mutation against a table with the same name but incompatible schema. Exception follows:
org.apache.cassandra.serializers.MarshalException: Expected 8 or 0 byte long for date (7)
at org.apache.cassandra.serializers.TimestampSerializer.validate(TimestampSerializer.java:118)
at org.apache.cassandra.db.marshal.AbstractType.validate(AbstractType.java:171)
at org.apache.cassandra.db.marshal.AbstractType.validateCollectionMember(AbstractType.java:289)
at org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:282)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:274)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:95)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:151)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:131)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:336)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
now i cannot start cassandra.
cqlsh is also not available.
I also encounter this problem. I'd like to share how I resolve the problem.
Change the debug mode
The default debug mode is INFO, the output is too less to track the error. You should change the debug mode from INFO to DEBUG. This is is determined by the following line in the log4j-server.properties file:
log4j.rootLogger=INFO,stdout,R
rerun the cassandra
From the output, I find the error appears when replay the commit log file. I think there are something wrong in the log file. But the log file is binary, I don't know how to read it. So I try to delete the log file that results in the error, then restart cassandra. And it works!
Maybe the root problem is different from mine, but you can try to find it in this way. Hope this can help.
Related
I am trying to drop a table before reading in a new set of values for testing purposes. When I run the command
DROP TABLE [dbo].[Table1]
I get the following error after about 3-5 minutes. It is a large table (~50 million rows).
Failed to execute query. Error: A severe error occurred on the current command. The results, if any, should be discarded.
Operation cancelled by user.
Failed to execute query. Error: A severe error occurred on the current command. The results, if any, should be discarded. Operation cancelled by user. what is the cause?
The error can cause for many different reasons, but it is not showing exact reason to find exact reason you can check following things:
It might be because of indexing issue to get that check the consistency of the database should be checked first
DBCC CHECKDB(``'database_name'``);
Check table consistency if you have it nailed down to a table.
DBCC CHECKTABLE(``'table_name'``);
When the problem was reported, search for any files with the name SQLDump* in the LOG folder, which contains ERRORLOG or You can try following actions in SSMS:
Object Explorer >> Management Node >> SQL Server Logs >> View the current log
I m facing issue while starting cassandra . I am using Datastax version. When I start cassandra , it is closing with below issue.
ERROR [main] 2017-08-10 20:42:04,999 CassandraDaemon.java:654 - Exception encountered during startup
org.apache.cassandra.io.FSWriteError: java.nio.file.AccessDeniedException: C:\Program Files\DataStax Community\data\commitlog\CommitLog-5-1502421528137.log
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:148) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.db.commitlog.CommitLogSegmentManager.recycleSegment(CommitLogSegmentManager.java:365) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:165) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:284) [apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:532) [apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:641) [apache-cassandra-2.2.8.jar:2.2.8]
Caused by: java.nio.file.AccessDeniedException: C:\Program Files\DataStax Community\data\commitlog\CommitLog-5-1502421528137.log
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) ~[na:1.7.0_40]
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.7.0_40]
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[na:1.7.0_40]
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) ~[na:1.7.0_40]
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) ~[na:1.7.0_40]
at java.nio.file.Files.delete(Files.java:1077) ~[na:1.7.0_40]
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:125) ~[apache-cassandra-2.2.8.jar:2.2.8]
... 6 common frames omitted
I tried to keep all permissions to the below folder and restarted many times .
C:\Program Files\DataStax Community\data
But still getting same error , I am using windows 10 .
Please suggest .
I got the same issue and in my case, the source was in the incompatible JRE version in the run configuration of executable code.
I've solved a problem with replacing 11.0.2 JRE version to 1.8.0_181
Here is a doc, where you can ensure about Cassandra compatibility.
Some ideas to try or look at:
Any anti virus scanning in background in windows needs to setup exclusions to your data directories or it can cause this
This may be an intermittent issue that isnt fixed because it hasnt been easy to reproduce. (CASSANDRA-9414) There are not many Cassandra users using Windows so it can have more rough edges. If possible you may want to consider a Linux host which gets more testing/use (not most helpful answer I know)
Try upgrading to more recent version or switch to Linux
Check the file permissions on C:\Program Files\DataStax Community\data\commitlog\CommitLog-5-1502421528137.log. You can possibly either change the permissions on that file or delete that commitlog and do a repair to make sure you didnt trigger any inconsistencies.
Something to look for in particular is the user who owns the file. There are a lot of tools like sstablesplit, sstablelevelreset, sstableexpiredblockers, sstablemetadata, sstable2json which will run cassandra in client mode using your cassandra.yaml. Since it has overlapping data and commitlog locations and it does somethings that trigger internal writes (sstable activity table in particular) it can end up flushing a commitlog or creating new sstables with whatever user ran the tool. If that user is different than the one that is used to run Cassandra it can cause failures like that one. Tools are slowly getting rewritten to make this less of an issue (ie sstabledump, 4.0 sstablemetadata) but if you use these tools issues like this is something to look out for.
After a server crash, I get a weird problem concerning database fixup. The console constantly throws a block of errors "Error checking database File does not exist" I did not find any databases with these names.
Here is an image as I am notz allowed to directly include pics:
https://pbs.twimg.com/media/CA87BQfUcAA21Cq.png:large
Where does domino know which databases to fixup?
How may I get rid of these errors?
Any idea appreciated.
Rene
Apparently I found a clue, myself:
http://www-01.ibm.com/support/docview.wss?uid=swg1LO78425
So, my next steps were fixup -j & compact on command line level without the server being up. Also, I deleted the dbdirman.nsf as suggested by Torsten.
I stumbled over a corrrupt database which caused a crash of the fixup. After moving the DB and recreating it from backup, the server could be started without an issue.
For now, the problem seems to be solved.
I have installed condor 8.2.0 on several Win7 (32/64bit) computers according this guide: http://www.slideshare.net/gtelzur/condor8-win-install All the services run on the same machine and therefore I rule out a physical network interrupt.
Whenever a job is created/submitted, it stays in the idle mode. A detailed look a the log files unveil the following issue (ShadowLog):
07/07/14 08:10:47 (1.1) (PID1): **** condor_shadow (condor_SHADOW) pid PID1 EXITING WITH STATUS 107
07/07/14 08:10:47 (1.0) (PID2): condor_read() failed: recv(fd=540) returned -1, errno = 10054 , reading 5 bytes from startd slot1#mycomputer.mydomain.local.
07/07/14 08:10:47 (1.0) (PID2): IO: Failed to read packet header
07/07/14 08:10:47 (1.0) (PID2): Can no longer talk to condor_starter <192.168.25.120:56186>
I couldn't find more details about an IO exception with ID 10054. Beyond that Google does not give me useful hints if I search for "Condor IO: Failed to read packet header".
Do you have a clue what could address the issue?
I had the same issue and it was fixed when I reinstalled Condor in C:\Condor (it was in D:\Condor).
Note that with Condor 8.2.1 I ran into an unrelated problem: I had to edit the condor_config file and remove one $ in the line CONDOR_HOST = $$(FULL_HOSTNAME), as otherwise there was a parsing error.
When you see
condor_read() failed: .... reading 5 bytes from .....
In one of the log files, that usually means that the other side of the connection hung up, so you should look in the log file for the other side of the conversion. In this case, that would be the StarterLog.slot1 on mycomputer.mydomain.local (or possibly just the StarterLog, if the problem happens very early).
Usually when a daemon hangs up, the reason for the hang up is in the log, and very often the problem is that the other side of the conversation isn't authorized. See configuration values that match ALLOW_* to see what is authorized.
I have a simple single node cassandra setup (1.1.0) (default settings). Whenever I try to create a keyspace in cassandra-cli, I get the error:
[default#unknown] create keyspace tax;
org.apache.thrift.transport.TTransportException
In cassandra server log, the exception stacktrace:
ERROR 12:15:04,722 Exception in thread Thread[MigrationStage:1,5,main]
java.lang.AssertionError
at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
I tried deleting the contents in ./var/lib/cassandra/data and restarting the server and my mac, but still ending up with same issue.
Looks like the system keyspace was corrupted. Removing the data files from
/var/lib/cassandra/data
/var/lib/cassandra/commitlog
/var/lib/cassandra/saved_caches
and restarting the cassandra server fixed the issue. (The above directories are defined in $CASSANDRA_HOME/conf/cassandra.yaml)
Following is the flow while adding the keyspace to Cassandra.(As per comments in Cassandrda source code. Correct me if I am getting it wrong)
1) At first step it check if any new keyspaces were added.
2) At second step we check if there were any keyspaces re-created, in this context
re-created means that they were previously deleted but still exist in the low-level schema as empty keys
3) At final step we updating modified keyspaces and saving keyspaces drop them later.
While modifying Keyspace it calls to function "updateKeyspace" and here it seems if the keyspace metadata is corrupt it throws assertion error.
SO in your case it might be that you have deleted the same Keyspace and trying to recreate which was causing this issue or as you mentioned It was a Metadata corruption.