Cassandra unable to start with error too many files open. (apache-cassandra-1.2.4)
error file contains:
ERROR 11:53:11,893 Exception encountered during startup
java.lang.RuntimeException: java.io.FileNotFoundException: /home/analysis.engine/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-289887-Data.db (Too many open files)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:61)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:44)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:101)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.DefsTable.serializedColumnFamilies(DefsTable.java:274)
at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:155)
at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:563)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:411)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:454)
Caused by: java.io.FileNotFoundException: /home/analysis.engine/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ib-289887-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
... 19 more
Please help me to find out the problem and it's solution.
#N D Thokare
Do you use the new Java driver from datastax to connect to Cassandra ?
I had the same issue recently, and the reason was that I bootstrapped many Session at the same time.
Please use root permission and set:
ulimit -n 1000000
Related
We have deployed our application on rhel 7 from rhel 6 and after deployment we are seeing following error in the catalina.properties, due to this my vm link is getting down frequently. We are using Documentum CMIS 16.4 version on tomcat 8.5 version.
Following is the error's details:
27-Nov-2018 01:57:00.536 SEVERE [https-jsse-nio-0.0.0.0-12510-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:457)
at java.lang.Thread.run(Thread.java:748)
Here are my efforts in order to solve this problem:
I have increased ulimit value from 1024 to 8192 for specific user and rebooted it, recycled tomcat service but nothing happened. I had done changed to file named /etc/security/limits.d/20-nproc.conf/20-nproc.conf. kindly help here.
I don't have privileges to add a comment, so posting as an answer. Try to find out which files are open by using the command
lsof -p <pid> | wc -l.
That will tell you which files are not getting closed.
You can also check the limits of a running process by
cat /proc/<pid>/limits
I tried to add a new list to the arabic plugin gazetteer.
I followed the following steps:
create a new file "tags.lst" to the directory "GATE_Developer_8.1\plugins\Lang_Arabic\resources\gazetteer\"
append the "lists.def" file with: "tags.lst:tags::arabic"
When launching the gate software, a window pop-up with the following message:
Resource could not be created!
gate.creole.ResourceInstantiationException:
gate.util.GateRuntimeException: Error loading list: tags.lst:
java.io.IOException: The system cannot find the path specified.
Here is the full exception:
gate.creole.ResourceInstantiationException: gate.util.GateRuntimeException: Error loading list: tags.lst: java.io.IOException: The system cannot find the path specified
at gate.creole.gazetteer.LinearDefinition.load(LinearDefinition.java:281)
at gate.creole.gazetteer.DefaultGazetteer.init(DefaultGazetteer.java:119)
at gate.Factory.createResource(Factory.java:432)
at gate.gui.NewResourceDialog$4.run(NewResourceDialog.java:257)
at java.lang.Thread.run(Thread.java:745)
Caused by: gate.util.GateRuntimeException: Error loading list: tags.lst: java.io.IOException: The system cannot find the path specified
at gate.creole.gazetteer.LinearDefinition.add(LinearDefinition.java:527)
at gate.creole.gazetteer.LinearDefinition.load(LinearDefinition.java:276)
... 4 more
Caused by: gate.creole.ResourceInstantiationException: java.io.IOException: The system cannot find the path specified
at gate.creole.gazetteer.LinearDefinition.loadSingleList(LinearDefinition.java:199)
at gate.creole.gazetteer.LinearDefinition.loadSingleList(LinearDefinition.java:158)
at gate.creole.gazetteer.LinearDefinition.add(LinearDefinition.java:520)
... 5 more
Caused by: java.io.IOException: The system cannot find the path specified
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at gate.creole.gazetteer.LinearDefinition.loadSingleList(LinearDefinition.java:188)
... 7 more
I will appreciate any help ??
The Problem was due to two major issues which are:
The file was not saved correctly as utf-8 encoding which was resolved by using online converter: http://www.motobit.com/util/charset-codepage-conversion.asp
The file contains special characters which were resolved by using the following replaceAll regular expression [#|"|:]:
line = line.replaceAll("[#|\"|:]", " ");
I am trying to upgrade a Cassandra 2.1.0 cluster to 2.1.8 (latest release).
When I start a first node with 2.1.8 runtime, I get an error and the node refuses to start.
This is the error's stack trace :
org.apache.cassandra.io.FSReadError: java.lang.NullPointerException
at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:642) ~[apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302) [apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.8.jar:2.1.8]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.8.jar:2.1.8]
Caused by: java.lang.NullPointerException: null
at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:634) ~[apache-cassandra-2.1.8.jar:2.1.8]
... 3 common frames omitted
FSReadError in Failed to remove unfinished compaction leftovers (file: /home/nudgeca2/datas/data/main/segment-97b5ba00571011e49a928bffe429b6b5/main-segment-ka-15432-Statistics.db). See log for details.
at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:642)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:302)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:634)
... 3 more
Exception encountered during startup: java.lang.NullPointerException
The cluster has 7 nodes and it turns on AWS Linux EC2 instances.
The node I try to upgrade was stopped after a nodetool drain.
Then I tried to come back to 2.1.0 runtime but I now get a similar error.
I also tried to stop and start another node and everything was ok, the node restarted without any problem.
I tried to touch the missing file (as it should be removed, I thought it would perhaps not need a specific content). I had two other files with the same error that I also touched. And finally the node fails further while trying to read these files.
Anyone has any idea what I should do ?
Thank you for any help.
It might be worth opening a Jira for that issue, so if nothing else, they can catch the NPE and provide a better error message.
It looks like it's trying to open:
file: /home/nudgeca2/datas/data/main/segment-97b5ba00571011e49a928bffe429b6b5/main-segment-ka-15432-Statistics.db
It's possible that it's trying to read that file because it finds the associated data file: (/home/nudgeca2/datas/data/main/segment-97b5ba00571011e49a928bffe429b6b5/main-segment-ka-15432-Data.db). Does that data file exist? I'd be tempted to move it out of the way, and see if it starts properly.
When running shark on spark cluster with one node
I'm getting the following error. can anyone please solve it...
Thanks in advance
error::
Executor updated: app-20140619165031-0000/0 is now FAILED (class java.io.IOException: Cannot run program "/home/trendwise/Hadoop_tools/jdk1.7.0_40/bin/java" (in directory "/home/trendwise/Hadoop_tools/spark/spark-0.9.1-bin-hadoop1/work/app-20140619165031-0000/0"): error=2, No such file or directory)
In my experience "No such file or directory" is often a symptom of some other exception. Usually a "no space left on device" and sometimes "too many files open". Mine the logs for other stack traces and monitor your disk usage and inode usage to confirm.
I am trying to implement titan graph database on cassandra for one of my applications. As per documentation, I have downloaded and extracted titan-cassandra-0.3.2 and when it trying to run titan.sh with titan-server-rexster.xml andtitan-server-cassandra.properties as arguments I am facing below error:
In below stack trace it says "Cannot locate /opt/isv/app/pkgs/titan-cassandra-0.3.2/config/cassandra.yaml" but I do have that file in that location. Could anyone please help?
WARN org.apache.cassandra.utils.CLibrary - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
72 [pool-1-thread-1] ERROR org.apache.cassandra.config.DatabaseDescriptor - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot locate /opt/isv/app/pkgs/titan-cassandra-0.3.2/config/cassandra.yaml
at org.apache.cassandra.config.DatabaseDescriptor.getStorageConfigURL(DatabaseDescriptor.java:113)
at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:130)
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:122)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:150)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:366)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:409)
at com.thinkaurelius.titan.diskstorage.cassandra.embedded.CassandraDaemonWrapper$CassandraStarter.run(CassandraDaemonWrapper.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Cannot locate /opt/isv/app/pkgs/titan-cassandra-0.3.2/config/cassandra.yaml
Fatal configuration error; unable to start server. See log for stacktrace.
A complete walkthrough for ENOMEM is here
http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootInsufficientResources_r.html also do not forget
Basically edit /etc/security/limits.conf file appending the following
memlock unlimited
nofile 100000
nproc 32768
as unlimited
and then run sysctl -p