Neo4j failing start to start - linux

Question

I am running neo4j 3.5.14 packaged by Bitnami on AWS linux "5.4.0-1078-aws". The error began when the disk filled and the database crashed. After clearing disk space and attempting to restart the database. I have tried starting neo4j at both the CLI and service levels. When attempting to start from the CLI I get the following log in neo4j.log. I have run this both as both the neo4j user and the root user.
2022-06-17 17:13:50.745+0000 INFO ======== Neo4j 3.5.14 ========
2022-06-17 17:13:50.768+0000 INFO Starting...
2022-06-17 17:13:54.203+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#3a6f2de3' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:232)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:148)
at org.neo4j.server.database.CommunityGraphFactory.newGraphDatabase(CommunityGraphFactory.java:41)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource#19648c40' was successfully initialized, but failed to start. Please see the attached cause exception "Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:227)
... 9 more
Caused by: java.lang.RuntimeException: Error reading transaction logs, recovery not possible. To force the database to start anyway, you can specify 'unsupported.dbms.tx_log.fail_on_corrupted_log_files=false'. This will try to recover as much as possible and then truncate the corrupt part of the transaction log. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.
at org.neo4j.kernel.recovery.Recovery.throwUnableToCleanRecover(Recovery.java:160)
at org.neo4j.kernel.recovery.LogTailScanner.findLogTail(LogTailScanner.java:147)
at org.neo4j.kernel.recovery.LogTailScanner.getTailInformation(LogTailScanner.java:260)
at org.neo4j.kernel.impl.transaction.log.LogVersionUpgradeChecker.check(LogVersionUpgradeChecker.java:48)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:349)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 14 more
Caused by: java.io.IOException: java.lang.IllegalArgumentException: Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:115)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LogTailScanner.findLogTail(LogTailScanner.java:99)
... 18 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -10 for version -10. At position LogPosition{logVersion=0, byteOffset=7639039} and entry version V3_0_10
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:130)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:81)
... 20 more
2022-06-17 17:13:54.217+0000 INFO Neo4j Server shutdown initiated by request
I should also add that while I can see that the data of the database is intact in the databases folder, neo4j-admin dump command is dumping an empty database file.

Answer 1

You can
Shut down the database
make a copy of the data folder as a backup
use neo4j-admin check-consistency to check your database integrity
add this line to your /etc/neo4j/neo4j.conf unsupported.dbms.tx_log.fail_on_corrupted_log_files=false
Restart the db.
If that doesn't help, shut down the db again.
remove data/transactions folder
Restart the db.
If that doesn't help, shut down the db again.
You can also use neo4j-admin copy to make a copy of the database while skipping corrupt records.