Configuration:
NFS server and the k8s cluster(single node cluster) run on two machines and use the same OS and NFS software, as below:
[root#test-2 ~]# yum info nfs-utils
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.tuna.tsinghua.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.huaweicloud.com
Installed Packages
Name : nfs-utils
Arch : x86_64
Epoch : 1
Version : 1.3.0
Release : 0.68.el7
Size : 1.1 M
Repo : installed
From repo : base
Summary : NFS utilities and supporting clients and daemons for the kernel NFS server
URL : http://sourceforge.net/projects/nfs
License : MIT and GPLv2 and GPLv2+ and BSD
Description : The nfs-utils package provides a daemon for the kernel NFS server and
: related tools, which provides a much higher level of performance than the
: traditional Linux NFS server used by most users.
:
: This package also contains the showmount program. Showmount queries the
: mount daemon on a remote host for information about the NFS (Network File
: System) server on the remote host. For example, showmount can display the
: clients which are mounted on that host.
:
: This package also contains the mount.nfs and umount.nfs program.
[root#test-2 ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[root#test-2 ~]# uname -a
Linux test-2 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root#test-2 ~]# cat /etc/exports
/home/nfs 192.168.0.0/24(rw,sync,no_root_squash,no_subtree_check,insecure)
K8S version: v1.17.9
Problems:
The application(a statefulset) running on k8s is using a PV that was dynamically provisioned by the k8s-nfs-provisioner, the PV is actually backed by a directory on remote NFS server. The application is keeping "CrashLoopBackOff" because it hits "input/output error" constantly when writing some data to the PV after only a few seconds of running.
Meanwhile, I saw a lot of errors in /var/log/messages:
Dec 2 17:11:36 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:11:36 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:11:36 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:11:36 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:11:36 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:11:36 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:12:05 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:12:05 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:21:41 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:21:41 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:21:41 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:21:41 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
Dec 2 17:21:42 localhost kernel: NFS: nfs4_reclaim_open_state: Lock reclaim failed!
I took a tcpdump until hit "Lock reclaim failed" in system log, and found there are many NFS errors as below:
NFS4ERR_BADSESSION (10052)
NFS4ERR_STALE_CLIENTID (10022)
NFS4ERR_NO_GRACE (10033)
I'm not sure if they're related to the "lock reclaim failed" or the "input/output" error.
I have encountered this problem on different machines with different machines from time to time and it really annoys me.
Anyone knows the root cause or how to fix it? Big thanks in advance.
Screenshots
application pod log
NFS errors in tcpdump
nfsstate -m output on k8s
nfsstate -c output on k8s, NOTE the high open_noat value.
NFS server configuration (my k8s node is 111.1.30.16)
The error generated when executed sudo mongod executed
2018-09-13T20:56:08.259+0530 I CONTROL [initandlisten] MongoDB starting : pid=3419 port=27017 dbpath=/data/db 64-bit host=admin
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] db version v3.6.7
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] git version: 2628472127e9f1826e02c665c1d93880a204075e
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] allocator:
tcmalloc
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] modules: none
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] build environment:
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] distmod: ubuntu1404
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] distarch: x86_64
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] target_arch: x86_64
2018-09-13T20:56:08.260+0530 I CONTROL [initandlisten] options: {}
2018-09-13T20:56:08.312+0530 I STORAGE [initandlisten] exception in initAndListen: NonExistentPath: Data directory /data/db not found., terminating
2018-09-13T20:56:08.312+0530 I CONTROL [initandlisten] now exiting
2018-09-13T20:56:08.312+0530 I CONTROL [initandlisten] shutting down with code:100
I'm a beginner in MEAN stack application development and stuck with this error. Please help me. Thanks in advance
This solution solves my problem
Make a directory as
sudo mkdir -p /data/db
That will make a directory named as db and than try to start with commands
sudo mongod
If you get another error or problem with starting mongod, You may find problem as
Failed to set up listener: SocketException: Address already in use
If you find that another error than you have to kill the running process of mongod by typing to terminal as
ps ax | grep mongod
Find the mongod running port and kill the process.
sudo kill ps_number
Another way is to make a specefic port when starting mongod as
sudo mongod --port 27018
cd ~
mongod
All you need to do above step to prevent the error.
Don't run the mongod at different local disk location.
eg. My computer has two local disk:
1.) Local disk E: - drive : (When I run mongod on E: drive or on targeting any location on E local disk then I will get error message as)
Admin#DESKTOP-CJBHIMO MINGW64 /e/Full-Stack-Developer
$ mongod
...
{"t":{"$date":"2021-03-22T11:33:26.675+05:30"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"NonExistentPath: Data directory E:\\data\\db\\ not found. Create the missing directory or specify another path using (1) the --dbpath command line option, or (2) by adding the 'storage.dbPath' option in the configuration file."}}
...
{"t":{"$date":"2021-03-22T11:33:26.677+05:30"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}
{"error":"NonExistentPath: Data directory E:\data\db\ not found.
2.) Local disk C: - drive : (When I run mongod on C: drive, which I recommend to use, cause, we mostly do the installation of mongodb on C: drive as per the instruction. If in your case, you installed on another drive, then run mongod on that drive's folder on hyper. Eg. as I run mongod on C:
Admin#DESKTOP-CJBHIMO MINGW64 ~/Desktop
$ mongod
{"t":{"$date":"2021-03-22T00:52:23.840-07:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"C:/data/db/","storageEngine":"wiredTiger"}}
...
{"t":{"$date":"2021-03-22T11:34:59.300+05:30"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
"msg":"Storage engine to use detected by data files","attr":{"dbpath":"C:/data/db/","storageEngine":"wiredTiger"}}
This whole example is based on windows 10. It runs successful on root directory cause it is inside C: disk. To get inside root directory :
Just run
sudo mkdir -p /data/db
On the machine running the mongoDB server
Before doing any of the other solutions see my answer here.
Maybe your conf file is being rejected due to outdated options or format changes.
My mongo was working fine. Today when I tried to open, it gives
ubuntu#DELL-PC:/usr/bin$ mongo
MongoDB shell version: 3.2.8
connecting to: test
2016-08-11T13:24:25.600+0530 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2016-08-11T13:24:25.600+0530 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect#src/mongo/shell/mongo.js:229:14
#(connect):1:6
exception: connect failed
I moved to /usr/db and tried to open mongod from there using sudo ./mongod
ubuntu#DELL-PC:/usr/bin$ sudo ./mongod
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] MongoDB starting : pid=9371 port=27017 dbpath=/data/db 64-bit host=DELL-PC
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] db version v3.2.8
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] allocator: tcmalloc
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] modules: none
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] build environment:
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] distmod: ubuntu1404
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] distarch: x86_64
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] target_arch: x86_64
2016-08-11T13:26:34.693+0530 I CONTROL [initandlisten] options: {}
2016-08-11T13:26:34.719+0530 I STORAGE [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating
2016-08-11T13:26:34.719+0530 I CONTROL [initandlisten] dbexit: rc: 100
I check I have free hard disk space.
ubuntu#DELL-PC:/usr/bin$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.4G 0 2.4G 0% /dev
tmpfs 486M 7.5M 478M 2% /run
/dev/sda1 97G 81G 12G 88% /
tmpfs 2.4G 52M 2.4G 3% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 2.4G 0 2.4G 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 486M 72K 485M 1% /run/user/1000
Can someone please suggest what can be the issue here?
What is the location of mongotable which I created earlier. i.e. location of data/db
How can I search it in terminal?
Your help much appreciated.
Quite possibly your configuration was updated by you hosting provider. It looks like your default configuration is pointing to the wrong dbpath. You'll need to update your config file to use the correct path. See https://docs.mongodb.com/manual/reference/configuration-options/
You can also test by just launching mongod using --dbpath.
If you don't tell mongo what is your database directory, it assumes that it's /data/db. Not /usr/db.
So, with ./mongod --dbpath /usr/db it should start..
Host OS Ubuntu 15.10
Guest OS Ubuntu 14.10
Using Vagrant with nfs and Virtualbox and static ip on the private network.
It is working perfectly except that after having suspended the host OS, the entire guest OS will be unusable.
This does not happen when using the normal virtualbox shared folders.
It's not only the nfs shared folder that is unusable, the entire OS is hanging.
Even syslog does not seem to see much action.
This is syslog on the guest, from waking up until vagrant halt is completed.
Feb 26 07:15:33 vagrant kernel: [ 8375.252989] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Feb 26 07:16:11 vagrant kernel: [ 8413.109832] nfs: server 192.168.33.1 not responding, still trying
Feb 26 07:16:38 vagrant kernel: [ 8440.687476] nfs: server 192.168.33.1 not responding, still trying
Feb 26 07:17:01 vagrant CRON[3776]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 26 07:20:33 vagrant rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="753" x-info="http://www.rsyslog.com"] exiting on signal 15.
How can this be fixed?
How should I debug it?
when i setup mongodb in my ubuntu , i try : ./mongo it show this error :
couldn't connect to server 127.0.0.1 shell/mongo.js
so what can i do ,
thanks
Manually remove the lockfile: sudo rm /var/lib/mongodb/mongod.lock
Run the repair script: sudo -u mongodb mongod -f /etc/mongodb.conf --repair
Please note the following:
You must run this command as the mongodb user. If you run it as root,
then root will own files in /var/lib/mongodb/ that are necessary to
run the mongodb daemon and therefore when the daemon trys to run
later as the mongodb user, it won't have permissions to start. In
that case you'll get this error: Unable to create / open lock file
for lockfilepath: /var/lib/mongodb/mongod.lock errno:13 Permission
denied, terminating.
On Ubuntu, you must specify the configuration file /etc/mongodb.conf
using the -f flag. Otherwise it will look for the data files in the
wrong place and you will see the following error: dbpath (/data/db/)
does not exist, terminating.
sudo rm /var/lib/mongodb/mongod.lock
sudo -u mongodb mongod -f /etc/mongodb.conf --repair
sudo service mongodb start
Here is all, sometimes, it takes a little while to start mongo after performing these operations.
Trying running $mongod
If you get en error such as
MongoDB shell version: 2.0.5
connecting to: test
Fri Jun 1 11:20:33 Error: couldn't connect to server 127.0.0.1 shell/mongo.js:84
exception: connect failed
hisham-agil:~ hisham$ mongod
mongod --help for help and startup options
Fri Jun 1 11:24:47 [initandlisten] MongoDB starting : pid=53452 port=27017 dbpath=/data/db/ 64-bit host=hisham-agil.local
Fri Jun 1 11:24:47 [initandlisten] db version v2.0.5, pdfile version 4.5
Fri Jun 1 11:24:47 [initandlisten] git version: nogitversion
Fri Jun 1 11:24:47 [initandlisten] build info: Darwin gamma.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:48:32 PST 2012; root:xnu-1699.24.23~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_49
Fri Jun 1 11:24:47 [initandlisten] options: {}
Fri Jun 1 11:24:47 [initandlisten] exception in initAndListen: 10296 dbpath (/data/db/) does not exist, terminating
Fri Jun 1 11:24:47 dbexit:
Fri Jun 1 11:24:47 [initandlisten] shutdown: going to close listening sockets...
Fri Jun 1 11:24:47 [initandlisten] shutdown: going to flush diaglog...
Fri Jun 1 11:24:47 [initandlisten] shutdown: going to close sockets...
Fri Jun 1 11:24:47 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jun 1 11:24:47 [initandlisten] shutdown: lock for final commit...
Fri Jun 1 11:24:47 [initandlisten] shutdown: final commit...
Fri Jun 1 11:24:47 [initandlisten] shutdown: closing all files...
Fri Jun 1 11:24:47 [initandlisten] closeAllFiles() finished
Fri Jun 1 11:24:47 dbexit: really exiting now
Then you've run into a basic startup error that is pretty common.
By default mongod will try to use /data/db for its database files, which in this case, does not exist.
You can't start
mongo
until you handle
mongod.
Try creating those directories and make sure they are writable by the same user that is running the mongod process.
**See similar question-- Getting an error, "Error: couldn't connect to server 127.0.0.1 shell/mongo.js" & when trying to run mongodb on mac osx lion
This is actually not an error... What happens here is that Mongo relies on a daemon in order to run the local database server, so in order to "fire up" the mongo server in your shell, you have to start the mongo service first.
For Fedora Linux (wich is the Distro I use) You have to run these commands:
1 sudo service mongod start
2 mongo
And there you have it! the server is going to run. Now, If you want Mongo service
to Start when the system boots then you have to run:
sudo chkconfig --levels 235 mongod on
And that's all! If you do that, now in the shell you just have to type mongo in order
to start the server but that's pretty much it, the problem is you have to start the SERVICE first and then the SERVER :)
P.S. The commands I posted might work on other linux distros as well, not just in fedora... In case not maybe you have to tweak some words depending on the distro you're using ;)
I got the same problem when I tried to install mongo. I got Error as,
Error
"Error: couldn't connect to server 127.0.0.1 shell/mongo.js:84"
Solution:
First install mongod by using:
sudo apt-get install mongodb-server
Then type
mongod --dbpath /mongo/db
Then
sudo rm /var/lib/mongodb/mongod.lock
Then
sudo -u mongodb mongod -f /etc/mongodb.conf --repair
Thank You
You need to delete the lockfile mongod.lock or /var/lib/mongodb/mongod.lock on ubuntu, then you need to run mongod.exe or service mongodb start on ubuntu first, then run mongo.exe or mongo on ubuntu.
Either your mongod is not running (check using "ps" command) or it is listening on some outside IP address and not on localhost. So first check the process list if 'mongod' is running. If yes, check with "netstat -nap" for the related port.
Of course you can start mongod on the console manually or even look into the mongod logfile
(if there is one configured...depending on how you installed mongod).
First you have to make sure that all the files and directories in your /var/lib/mongodb/ folder (or whichever folder dbpath points to) belong to the mongodb user and mongodb group.
cd /var/lib/mongodb/
sudo chown mongodb filename.*
sudo chgrp mongodb filename.*
sudo chown -R mongodb directory
sudo chgrp -R mongodb directory
(Replace filename and directory with their respective names)
Then you can remove the lock, repair the database and restart the daemon as other people already mentioned:
sudo rm /var/lib/mongodb/mongod.lock
sudo -u mongodb mongod -f /etc/mongodb.conf --repair
sudo service mongodb start
First start your mongo server by
Users-MacBook-Pro:csv1 Admin$ mongod
all output going to: /usr/local/var/log/mongodb/mongo.log
Then open another terminal window and open shell
Users-MacBook-Pro:csv1 Admin$ mongo
Also check that your root partition has enough space to start mongod.
df -h /
You'll see smth like this on mongod launch:
Mon Aug 12 17:02:59.159 [initandlisten] recover : no journal files present, no recovery needed
Mon Aug 12 17:02:59.159 [initandlisten]
Mon Aug 12 17:02:59.159 [initandlisten] ERROR: Insufficient free space for journal files
Mon Aug 12 17:02:59.159 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles
Mon Aug 12 17:02:59.159 [initandlisten]
Mon Aug 12 17:02:59.159 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Mon Aug 12 17:02:59.159 dbexit:
Mon Aug 12 17:02:59.159 [initandlisten] shutdown: going to close listening sockets...
On Ubuntu, try this:
sudo invoke-rc.d mongodb start
It could be combination of $PATH and Permission issue.
Try following steps given below:
Update your $PATH variable to point to your MongoDB bin file. In my case brew install MongoDB to this folder:
/usr/local/Cellar/mongodb/2.4.6/
In order to update your $PATH variable, do following:
$ sudo vi /etc/paths
Then, press ‘i’ to insert text in Vi and append the your MongoDB path to the end of the ‘paths’ file and restart the terminal.
/usr/local/Cellar/mongodb/2.4.6/bin
Use ‘Esc : w q’ to save and exit from Vi editor.
Use echo to display your path variable:
$ echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/Cellar/mongodb/2.4.6/bin
Now try to check the Mongo version, if you get following, then you are on the right track!
$ mongo --version
MongoDB shell version: 2.4.6
Now we need to create the database directory. I used the default ‘/data/db’ location suggested in MongoDB docs. I also created a log directory to avoid any permission issues while Mongo tries to create any logs. Change ownership and that will do the job.
$ sudo mkdir /data/db
$ sudo mkdir /data/log
$ whoami
username
$ chown -R username /data
Now, we will create a default config file for MongoDB to be provided for the first time we run ‘mongod’ command. Now, I will also like to point out that ‘mongod’ will start a service, which will listen for incoming data connections. This is similar having ‘$service mysqld start’ executed.Let’s go ahead and create the config file. Please keep in mind that I have created very basic config file. However, you can add many other variables to configure MongoDB. This is the first time I am playing with MongoDB, so I just know as much as I read on MongoDB docs!I created ‘mongodb.conf’.
$ sudo vi /etc/mongodb.conf
Add following:
fork = true
port = 27017
quiet = true
dbpath = /data/db
logpath = /data/log/mongod.log
logappend = true
journal = true
Please note that the default port for MongoDB server is 27017. Use your own path for dbpath and logpath you created in Step – 5. Don’t forget to close and save the conf file.
Now we are all set to start our MongoDB service. Open two instances of Terminal.In Terminal 1, type in:
$ sudo mongod -f /etc/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 3516
all output going to: /data/log/mongod.log
child process started successfully, parent exiting
If you get above message, then know that you have successfully started your Mongod service.
Now, to connect to it, in Terminal 2 type following:
$mongo test
MongoDB shell version: 2.4.6
connecting to: test
Server has startup warnings:
Tue Sep 3 16:55:43.527 [initandlisten]
Tue Sep 3 16:55:43.527 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
>
Ignore the warnings, but you are successfully connected to the ‘test’ database! Cool!
That's all. I applied this solution, when I tried to install copy of MongoDB on my Mac for the first time. See if this help you too.
For detailed post you can go here - http://arcanebytes.com/2013/09/03/mongodb-installation-on-mac-os-x/#comment-1036112094.
I hope it helps!
Cheers,
Chinmay
I solved this problem on ubuntu 12.04 by following steps:
1) sudo rm /var/log/mongodb
2) sudo rm /var/lib/mongodb
3) I removed mongo and then installed it again
4) sudo service mongodb restart
and All is Well
For Ubuntu:
Just Open the terminal and enter the below command.
You just have to restart your mongoDB.
sudo service mongodb restart