I m facing issue while starting cassandra . I am using Datastax version. When I start cassandra , it is closing with below issue.
ERROR [main] 2017-08-10 20:42:04,999 CassandraDaemon.java:654 - Exception encountered during startup
org.apache.cassandra.io.FSWriteError: java.nio.file.AccessDeniedException: C:\Program Files\DataStax Community\data\commitlog\CommitLog-5-1502421528137.log
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:148) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.db.commitlog.CommitLogSegmentManager.recycleSegment(CommitLogSegmentManager.java:365) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:165) ~[apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:284) [apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:532) [apache-cassandra-2.2.8.jar:2.2.8]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:641) [apache-cassandra-2.2.8.jar:2.2.8]
Caused by: java.nio.file.AccessDeniedException: C:\Program Files\DataStax Community\data\commitlog\CommitLog-5-1502421528137.log
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) ~[na:1.7.0_40]
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.7.0_40]
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[na:1.7.0_40]
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) ~[na:1.7.0_40]
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) ~[na:1.7.0_40]
at java.nio.file.Files.delete(Files.java:1077) ~[na:1.7.0_40]
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:125) ~[apache-cassandra-2.2.8.jar:2.2.8]
... 6 common frames omitted
I tried to keep all permissions to the below folder and restarted many times .
C:\Program Files\DataStax Community\data
But still getting same error , I am using windows 10 .
Please suggest .
I got the same issue and in my case, the source was in the incompatible JRE version in the run configuration of executable code.
I've solved a problem with replacing 11.0.2 JRE version to 1.8.0_181
Here is a doc, where you can ensure about Cassandra compatibility.
Some ideas to try or look at:
Any anti virus scanning in background in windows needs to setup exclusions to your data directories or it can cause this
This may be an intermittent issue that isnt fixed because it hasnt been easy to reproduce. (CASSANDRA-9414) There are not many Cassandra users using Windows so it can have more rough edges. If possible you may want to consider a Linux host which gets more testing/use (not most helpful answer I know)
Try upgrading to more recent version or switch to Linux
Check the file permissions on C:\Program Files\DataStax Community\data\commitlog\CommitLog-5-1502421528137.log. You can possibly either change the permissions on that file or delete that commitlog and do a repair to make sure you didnt trigger any inconsistencies.
Something to look for in particular is the user who owns the file. There are a lot of tools like sstablesplit, sstablelevelreset, sstableexpiredblockers, sstablemetadata, sstable2json which will run cassandra in client mode using your cassandra.yaml. Since it has overlapping data and commitlog locations and it does somethings that trigger internal writes (sstable activity table in particular) it can end up flushing a commitlog or creating new sstables with whatever user ran the tool. If that user is different than the one that is used to run Cassandra it can cause failures like that one. Tools are slowly getting rewritten to make this less of an issue (ie sstabledump, 4.0 sstablemetadata) but if you use these tools issues like this is something to look out for.
Related
While doing system update ( or ant update system), we are getting below error (hmchistoryentries doesn't exist). Any one faced this before?
Per documentation, it seems this is a deprecated item type. Though we are not using hmc, we are not sure which extension is using this itemtype. Appreciate your help.
[java] Caused by: org.apache.ddlutils.DatabaseOperationException: java.sql.SQLSyntaxErrorException: Table 'hybrisD2C.hmchistoryentries' doesn't exist[java] at org.apache.ddlutils.platform.PlatformImplBase.readModelFromDatabase(PlatformImplBase.java:1891)[java] at org.apache.ddlutils.platform.PlatformImplBase.readModelFromDatabase(PlatformImplBase.java:1869)[java] at de.hybris.bootstrap.ddl.HybrisSchemaGenerator.update(HybrisSchemaGenerator.java:225)[java] at de.hybris.platform.core.Initialization.initializeSchemaAndTypeSystemFullyNewStyle(Initialization.java:1245)[java] at de.hybris.platform.core.Initialization.initialize(Initialization.java:1121)[java] at de.hybris.platform.core.Initialization.createEmptySystemOrUpdate(Initialization.java:776)[java] at de.hybris.platform.core.Initialization.access$4(Initialization.java:756)[java] at de.hybris.platform.core.Initialization$4.call(Initialization.java:563)[java] at de.hybris.platform.core.Initialization$4.call(Initialization.java:1)[java] at de.hybris.platform.core.Initialization$SessionRecoveryAfterRegistryStartupAwareExecutor.execute(Initialization.java:698)[java] at de.hybris.platform.core.Initialization.doInitializeImpl(Initialization.java:566)[java] at de.hybris.platform.core.Initialization.access$5(Initialization.java:488)[java] at de.hybris.platform.core.Initialization$5.call(Initialization.java:812)[java] at de.hybris.platform.core.Initialization$5.call(Initialization.java:1)[java] at de.hybris.platform.core.system.InitializationLockHandler.performLocked(InitializationLockHandler.java:80)[java] at de.hybris.platform.core.Initialization.doInitialize(Initialization.java:844)[java] at de.hybris.ant.taskdefs.InitPlatformAntPerformableImpl.performImpl(InitPlatformAntPerformableImpl.java:106)[java] at de.hybris.ant.taskdefs.AbstractAntPerformable.doPerform(AbstractAntPerformable.java:92)[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[java] at java.lang.reflect.Method.invoke(Method.java:498)[java] at bsh.Reflect.invokeMethod(Reflect.java:131)[java] at bsh.Reflect.invokeObjectMethod(Reflect.java:77)[java] at bsh.Name.invokeMethod(Name.java:852)[java] at bsh.BSHMethodInvocation.eval(BSHMethodInvocation.java:69)[java] ... 16 more
Can you search all *-items.xml files to find out what extension it is coming from? You might also want to check your the localextensions.xml file if the extension or any extension depending on it is there. If it's there, you can remove it.
This error happened due to the incorrect version of DB connector. We were using MySQL 8.0 and mysql-connector-java-8.0.19.jar connector. But SAP officially doesn't support MySQL 8.0 with 1808 version. After downgrading it to MySql 5.x and mysql-connector-java-5.1.x-bin.jar, this error no longer exists.
Thank you.
With bdutil, the latest version of tarball I can find is on spark 1.3.1:
gs://spark-dist/spark-1.3.1-bin-hadoop2.6.tgz
There are a few new DataFrame features in Spark 1.4 that I want to use. Any chance the Spark 1.4 image be available for bdutil, or any workaround?
UPDATE:
Following the suggestion from Angus Davis, I downloaded and pointed to spark-1.4.1-bin-hadoop2.6.tgz, the deployment went well; however, run into error when calling SqlContext.parquetFile(). I cannot explain why this exception is possible, GoogleHadoopFileSystem should be a subclass of org.apache.hadoop.fs.FileSystem. Will continue investigate on this.
Caused by: java.lang.ClassCastException: com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem cannot be cast to org.apache.hadoop.fs.FileSystem
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2595)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:112)
at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:144)
at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159)
at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:177)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:504)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
Asked a separate question about the exception here
UPDATE:
The error turned out to be a Spark defect; resolution/workaround provided in the above question.
Thanks!
Haiying
If a local workaround is acceptable, you can copy the spark-1.4.1-bin-hadoop2.6.tgz from an apache mirror into a bucket that you control. You can then edit extensions/spark/spark-env.sh and change SPARK_HADOOP2_TARBALL_URI='<your copy of spark 1.4.1>' (make certain that the service account running your VMs has permission to read the tarball).
Note that I haven't done any testing to see if Spark 1.4.1 works out of the box right now, but I'd be interested in hearing your experience if you decide to give it a go.
Cassandra stopped.
when i restart Cassandra using "service cassandra start" or "service cassandra restart", i get the following error(from "/var/log/cassandra/system.log"):
ERROR [main] 2014-11-14 02:08:52,379 CommitLogReplayer.java (line 304) Unexpected error deserializing mutation; saved to /tmp/mutation3145492124947244713dat and ignored. This may be caused by replaying a mutation against a table with the same name but incompatible schema. Exception follows:
org.apache.cassandra.serializers.MarshalException: Expected 8 or 0 byte long for date (7)
at org.apache.cassandra.serializers.TimestampSerializer.validate(TimestampSerializer.java:118)
at org.apache.cassandra.db.marshal.AbstractType.validate(AbstractType.java:171)
at org.apache.cassandra.db.marshal.AbstractType.validateCollectionMember(AbstractType.java:289)
at org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:282)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:274)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:95)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:151)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:131)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:336)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
now i cannot start cassandra.
cqlsh is also not available.
I also encounter this problem. I'd like to share how I resolve the problem.
Change the debug mode
The default debug mode is INFO, the output is too less to track the error. You should change the debug mode from INFO to DEBUG. This is is determined by the following line in the log4j-server.properties file:
log4j.rootLogger=INFO,stdout,R
rerun the cassandra
From the output, I find the error appears when replay the commit log file. I think there are something wrong in the log file. But the log file is binary, I don't know how to read it. So I try to delete the log file that results in the error, then restart cassandra. And it works!
Maybe the root problem is different from mine, but you can try to find it in this way. Hope this can help.
I have installed condor 8.2.0 on several Win7 (32/64bit) computers according this guide: http://www.slideshare.net/gtelzur/condor8-win-install All the services run on the same machine and therefore I rule out a physical network interrupt.
Whenever a job is created/submitted, it stays in the idle mode. A detailed look a the log files unveil the following issue (ShadowLog):
07/07/14 08:10:47 (1.1) (PID1): **** condor_shadow (condor_SHADOW) pid PID1 EXITING WITH STATUS 107
07/07/14 08:10:47 (1.0) (PID2): condor_read() failed: recv(fd=540) returned -1, errno = 10054 , reading 5 bytes from startd slot1#mycomputer.mydomain.local.
07/07/14 08:10:47 (1.0) (PID2): IO: Failed to read packet header
07/07/14 08:10:47 (1.0) (PID2): Can no longer talk to condor_starter <192.168.25.120:56186>
I couldn't find more details about an IO exception with ID 10054. Beyond that Google does not give me useful hints if I search for "Condor IO: Failed to read packet header".
Do you have a clue what could address the issue?
I had the same issue and it was fixed when I reinstalled Condor in C:\Condor (it was in D:\Condor).
Note that with Condor 8.2.1 I ran into an unrelated problem: I had to edit the condor_config file and remove one $ in the line CONDOR_HOST = $$(FULL_HOSTNAME), as otherwise there was a parsing error.
When you see
condor_read() failed: .... reading 5 bytes from .....
In one of the log files, that usually means that the other side of the connection hung up, so you should look in the log file for the other side of the conversion. In this case, that would be the StarterLog.slot1 on mycomputer.mydomain.local (or possibly just the StarterLog, if the problem happens very early).
Usually when a daemon hangs up, the reason for the hang up is in the log, and very often the problem is that the other side of the conversation isn't authorized. See configuration values that match ALLOW_* to see what is authorized.
I have a simple single node cassandra setup (1.1.0) (default settings). Whenever I try to create a keyspace in cassandra-cli, I get the error:
[default#unknown] create keyspace tax;
org.apache.thrift.transport.TTransportException
In cassandra server log, the exception stacktrace:
ERROR 12:15:04,722 Exception in thread Thread[MigrationStage:1,5,main]
java.lang.AssertionError
at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441)
at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269)
at org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
I tried deleting the contents in ./var/lib/cassandra/data and restarting the server and my mac, but still ending up with same issue.
Looks like the system keyspace was corrupted. Removing the data files from
/var/lib/cassandra/data
/var/lib/cassandra/commitlog
/var/lib/cassandra/saved_caches
and restarting the cassandra server fixed the issue. (The above directories are defined in $CASSANDRA_HOME/conf/cassandra.yaml)
Following is the flow while adding the keyspace to Cassandra.(As per comments in Cassandrda source code. Correct me if I am getting it wrong)
1) At first step it check if any new keyspaces were added.
2) At second step we check if there were any keyspaces re-created, in this context
re-created means that they were previously deleted but still exist in the low-level schema as empty keys
3) At final step we updating modified keyspaces and saving keyspaces drop them later.
While modifying Keyspace it calls to function "updateKeyspace" and here it seems if the keyspace metadata is corrupt it throws assertion error.
SO in your case it might be that you have deleted the same Keyspace and trying to recreate which was causing this issue or as you mentioned It was a Metadata corruption.