Is there a good way to detect a stale NFS mount - linux

I have a procedure I want to initiate only if several tests complete successfully.
One test I need is that all of my NFS mounts are alive and well.
Can I do better than the brute force approach:
mount | sed -n "s/^.* on \(.*\) type nfs .*$/\1/p" |
while read mount_point ; do
timeout 10 ls $mount_point >& /dev/null || echo "stale $mount_point" ;
done
Here timeout is a utility that will run the command in the background, and will kill it after a given time, if no SIGCHLD was caught prior to the time limit, returning success/fail in the obvious way.
In English: Parse the output of mount, check (bounded by a timeout) every NFS mount point. Optionally (not in the code above) breaking on the first stale mount.

A colleague of mine ran into your script. This doesn't avoid a "brute force" approach, but if I may in Bash:
while read _ _ mount _; do
read -t1 < <(stat -t "$mount") || echo "$mount timeout";
done < <(mount -t nfs)
mount can list NFS mounts directly. read -t (a shell builtin) can time out a command. stat -t (terse output) still hangs like an ls*. ls yields unnecessary output, risks false positives on huge/slow directory listings, and requires permissions to access - which would also trigger a false positive if it doesn't have them.
while read _ _ mount _; do
read -t1 < <(stat -t "$mount") || lsof -b 2>/dev/null|grep "$mount";
done < <(mount -t nfs)
We're using it with lsof -b (non-blocking, so it won't hang too) in order to determine the source of the hangs.
Thanks for the pointer!
test -d (a shell builtin) would work instead of stat (a standard external) as well, but read -t returns success only if it doesn't time out and reads a line of input. Since test -d doesn't use stdout, a (( $? > 128 )) errorlevel check on it would be necessary - not worth the legibility hit, IMO.

Took me some time, but here is what I found which works in Python:
import signal, os, subprocess
class Alarm(Exception):
pass
def alarm_handler(signum, frame):
raise Alarm
pathToNFSMount = '/mnt/server1/' # or you can implement some function
# to find all the mounts...
signal.signal(signal.SIGALRM, alarm_handler)
signal.alarm(3) # 3 seconds
try:
proc = subprocess.call('stat '+pathToNFSMount, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
stdoutdata, stderrdata = proc.communicate()
signal.alarm(0) # reset the alarm
except Alarm:
print "Oops, taking too long!"
Remarks:
credit to the answer here.
You could also use alternative scheme:
os.fork() and os.stat()
check if the fork finished, if it has timed out you can kill it. You will need to work with time.time() and so on.

In addition to previous answers, which hangs under some circumstances, this snippet checks all suitable mounts, kills with signal KILL, and is tested with CIFS too:
grep -v tracefs /proc/mounts | cut -d' ' -f2 | \
while read m; do \
timeout --signal=KILL 1 ls -d $m > /dev/null || echo "$m"; \
done

You could write a C program and check for ESTALE.
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <iso646.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
int main(){
struct stat st;
int ret;
ret = stat("/mnt/some_stale", &st);
if(ret == -1 and errno == ESTALE){
printf("/mnt/some_stale is stale\n");
return EXIT_SUCCESS;
} else {
return EXIT_FAILURE;
}
}

Writing a C program that checks for ESTALE is a good option if you don't mind waiting for the command to finish because of the stale file system. If you want to implement a "timeout" option the best way I've found to implement it (in a C program) is to fork a child process that tries to open the file. You then check if the child process has finished reading a file successfully in the filesystem within an allocated amount of time.
Here is a small proof of concept C program to do this:
#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/wait.h>
void readFile();
void waitForChild(int pid);
int main(int argc, char *argv[])
{
int pid;
pid = fork();
if(pid == 0) {
// Child process.
readFile();
}
else if(pid > 0) {
// Parent process.
waitForChild(pid);
}
else {
// Error
perror("Fork");
exit(1);
}
return 0;
}
void waitForChild(int child_pid)
{
int timeout = 2; // 2 seconds timeout.
int status;
int pid;
while(timeout != 0) {
pid = waitpid(child_pid, &status, WNOHANG);
if(pid == 0) {
// Still waiting for a child.
sleep(1);
timeout--;
}
else if(pid == -1) {
// Error
perror("waitpid()");
exit(1);
}
else {
// The child exited.
if(WIFEXITED(status)) {
// Child was able to call exit().
if(WEXITSTATUS(status) == 0) {
printf("File read successfully!\n");
return;
}
}
printf("File NOT read successfully.\n");
return;
}
}
// The child did not finish and the timeout was hit.
kill(child_pid, 9);
printf("Timeout reading the file!\n");
}
void readFile()
{
int fd;
fd = open("/path/to/a/file", O_RDWR);
if(fd == -1) {
// Error
perror("open()");
exit(1);
}
else {
close(fd);
exit(0);
}
}

I wrote https://github.com/acdha/mountstatus which uses an approach similar to what UndeadKernel mentioned, which I've found to be the most robust approach: it's a daemon which periodically scans all mounted filesystems by forking a child process which attempts to list the top-level directory and SIGKILL it if it fails to respond in a certain timeout, with both successes and failures recorded to syslog. That avoids issues with certain client implementations (e.g older Linux) which never trigger timeouts for certain classes of error, NFS servers which are partially responsive but e.g. won't respond to actual calls like listdir, etc.
I don't publish them but the included Makefile uses fpm to build rpm and deb packages with an Upstart script.

Another way, using shell script. Works good for me:
#!/bin/bash
# Purpose:
# Detect Stale File handle and remove it
# Script created: July 29, 2015 by Birgit Ducarroz
# Last modification: --
#
# Detect Stale file handle and write output into a variable and then into a file
mounts=`df 2>&1 | grep 'Stale file handle' |awk '{print ""$2"" }' > NFS_stales.txt`
# Remove : ‘ and ’ characters from the output
sed -r -i 's/://' NFS_stales.txt && sed -r -i 's/‘//' NFS_stales.txt && sed -r -i 's/’//' NFS_stales.txt
# Not used: replace space by a new line
# stales=`cat NFS_stales.txt && sed -r -i ':a;N;$!ba;s/ /\n /g' NFS_stales.txt`
# read NFS_stales.txt output file line by line then unmount stale by stale.
# IFS='' (or IFS=) prevents leading/trailing whitespace from being trimmed.
# -r prevents backslash escapes from being interpreted.
# || [[ -n $line ]] prevents the last line from being ignored if it doesn't end with a \n (since read returns a non-zero exit code when it encounters EOF).
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Unmounting due to NFS Stale file handle: $line"
umount -fl $line
done < "NFS_stales.txt"
#EOF

I'll just paste a snippet from our Icinga2 NFS stale mount monitoring Bash script here:
MOUNTS="$(mount -t nfs;mount -t nfs3;mount -t nfs4)"
MOUNT_POINTS=$(echo -e "$MOUNTS \n"|grep -v ^$|awk '{print $3}')
if [ -z "$MOUNT_POINTS" ]; then
OUTPUT="[OK] No nfs mounts"
set_result 0
else
for i in $MOUNT_POINTS;do
timeout 1 stat -t "$i" > /dev/null
TMP_RESULT=$?
set_result $TMP_RESULT
set_output $TMP_RESULT "$i"
done
fi

Related

How to read stdout from a sub process in bash in real time

I have a simple C++ program that counts from 0 to 10 with an increment every 1 second. When the value is incremented, it is written to stdout. This program intentionally uses printf rather than std::cout.
I want to call this program from a bash script, and perform some function (eg echo) on the value when it is written to stdout.
However, my script waits for the program to terminate, and then process all the values at the same time.
C++ prog:
#include <stdio.h>
#include <unistd.h>
int main(int argc, char **argv)
{
int ctr = 0;
for (int i = 0; i < 10; ++i)
{
printf("%i\n", ctr++);
sleep(1);
}
return 0;
}
Bash script:
#!/bin/bash
for c in $(./script-test)
do
echo $c
done
Is there another way to read the output of my program, that will access it in real time, rather than wait for for the process to terminate.
Note: the C++ program is a demo sample - the actual program I am using also uses printf, but I am not able to make changes to this code, hence the solution needs to be in the bash script.
Many thanks,
Stuart
As you correctly observed, $(command) waits for the entire output of command, splits that output, and only after that, the for loop starts.
To read output as soon as is available, use while read:
./script-test | while IFS= read -r line; do
echo "do stuff with $line"
done
or, if you need to access variables from inside the loop afterwards, and your system supports <()
while IFS= read -r line; do
echo "do stuff with $line"
done < <(./script-test)
# do more stuff, that depends on variables set inside the loop
You might be more lucky using a pipe:
#!/bin/bash
./script-test | while IFS= read -r c; do
echo "$c"
done

Get the content on the command line with an external promgram

I would like to write a small program which will analyize my current input on the command line and generate some suggesstions like those search engines do.
The problems is how can an external program get the content on command line? For example
# an external program started and got passed in the PID of the shell below.
# the user typed something in the shell like this...
<PROMPT> $ echo "grab this command"
# the external program now get 'echo "grab this command"'
# and ideally the this could be done in realtime.
More over, can I just modify the content of current command line?
EDIT
bash uses libreadline to manage the command line, but still I can not imagine how to make use of this.
You could write your own shell wrapper using c. Open bash in a process using popen and use fgetc and fputc to write the data to the process and the output file.
A quick dirty hack could look like this (bash isn't started in interactive mode, but otherwise should work fine. --> no prompt):
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
pid_t pid;
void kill_ch(int sig) {
kill(pid, SIGKILL);
}
/**
*
*/
int main(int argc, char** argv) {
int b;
FILE *cmd = NULL;
FILE *log = NULL;
signal(SIGALRM, (void (*)(int))kill_ch);
cmd = popen("/bin/bash -s", "r+");
if (cmd == NULL) {
fprintf(stderr, "Error: Failed to open process");
return EXIT_FAILURE;
}
setvbuf(cmd, NULL, _IOLBF, 0);
log = fopen("out.txt", "a");
if (log == NULL) {
fprintf(stderr, "Error: Failed to open logfile");
return EXIT_FAILURE;
}
setvbuf(log, NULL, _IONBF, 0);
pid = fork();
if (pid != 0)
goto EXEC_WRITE;
else
goto EXEC_READ;
EXEC_READ:
while (1) {
b = fgetc(stdin);
if (b != EOF) {
fputc((char) b, cmd);
fputc((char) b, log);
}
}
EXEC_WRITE:
while (1) {
b = fgetc(cmd);
if (b == EOF) {
return EXIT_SUCCESS;
}
fputc(b, stdout);
fputc(b, log);
}
return EXIT_SUCCESS;
}
I might not fully understand your question but I think you'd basically have two options.
The first option would be to explicitly call your "magic" program by prefixing your call with it like so
<PROMPT> $ magic echo "grab this command"
(magic analyzes $* and says...)
Your input would print "grab this command" to stdout
<PROMPT> $
In this case the arguments to "magic" would be handled as positional parameters ($*, $1 ...)
The second option would be to wrap an interpreter-like something around your typing. E.g. the Python interpreter does so if called without arguments. You start the interpreter, which will basically read anything you type (stdin) in an endless loop, interpret it, and produce some output (typically on stdout).
<PROMPT> $ magic
<MAGIC_PROMPT> $ echo "grab this command"
(your magic interpreter processes the input and says...)
Your input would print "grab this command" to stdout
<MAGIC_PROMPT> $

Faster and precise way to count lines other than wc -l

Usually I use wc -l to count the lines of a file. However for a file with 5*10^7 lines, I get only 10^7 as an answer.
I've tried everything proposed here here:
How to count lines in a document?
But it takes to much time than wc -l.
Is there any other option?
Anyone serious about speed line counting can just create their own implementation:
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#define BUFFER_SIZE (1024 * 16)
char BUFFER[BUFFER_SIZE];
int main(int argc, char** argv) {
unsigned int lines = 0;
int fd, r;
if (argc > 1) {
char* file = argv[1];
if ((fd = open(file, O_RDONLY)) == -1) {
fprintf(stderr, "Unable to open file \"%s\".\n", file);
return 1;
}
} else {
fd = fileno(stdin);
}
while ((r = read(fd, BUFFER, BUFFER_SIZE)) > 0) {
char* p = BUFFER;
while ((p = memchr(p, '\n', (BUFFER + r) - p))) {
++p;
++lines;
}
}
close(fd);
if (r == -1) {
fprintf(stderr, "Read error.\n");
return 1;
}
printf("%d\n", lines);
return 0;
}
Usage
a < input
... | a
a file
Example:
# time ./wc temp.txt
10000000
real 0m0.115s
user 0m0.102s
sys 0m0.014s
# time wc -l temp.txt
10000000 temp.txt
real 0m0.120s
user 0m0.103s
sys 0m0.016s
* Code compiled with -O3 natively on a system with AVX and SSE4.2 using GCC 4.8.2.
You could try sed
sed -n '$=' file
The = says to print the line number, and the dollar says to only do it on the last line. The -n says not to do too much else.
Or here's a way in Perl, save this as wc.pl and do chmod +x wc.pl:
#!/usr/bin/perl
use strict;
use warnings;
my $filename = <#ARGV>;
my $lines = 0;
my $buffer;
open(FILE, $filename) or die "ERROR: Can not open file: $!";
while (sysread FILE, $buffer, 65536) {
$lines += ($buffer =~ tr/\n//);
}
close FILE;
print "$lines\n";
Run it like this:
wc.pl yourfile
Basically it reads your file in in chunks of 64kB at a time and then takes advantage of the fact that tr returns the number of substitutions it has made after asking it to delete all newlines.
Try with nl and see what happens...
You can get the line count using awk as well like below
awk 'END {print NR}' names.txt
(OR) Using while .. do .. done bash loop construct like
CNT=0; while read -r LINE; do (( CNT++ )); done < names.txt; echo $CNT
Depends on how you open the file, but probably reading it from STDIN instead would get the fix:
wc -l < file

how to define script interpreter with shebang

It is clear that one can use the
#!/usr/bin/perl
shebang notation in the very first line of a script to define the interpreter. However, this presupposes an interpreter that ignores hashmark-starting lines as comments. How can one use an interpreter that does not have this feature?
With a wrapper that removes the first line and calls the real interpreter with the remainder of the file. It could look like this:
#!/bin/sh
# set your "real" interpreter here, or use cat for debugging
REALINTERP="cat"
tail -n +2 $1 | $REALINTERP
Other than that: In some cases ignoring the error message about that first line could be an option.
Last resort: code support for the comment char of your interpreter into the kernel.
I think the first line is interpreted by the operating system.
The interpreter will be started and the name of the script is handed down to the script as its first parameter.
The following script 'first.myint' calls the interpreter 'myinterpreter' which is the executable from the C program below.
#!/usr/local/bin/myinterpreter
% 1 #########
2 xxxxxxxxxxx
333
444
% the last comment
The sketch of the personal interpreter:
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define BUFFERSIZE 256 /* input buffer size */
int
main ( int argc, char *argv[] )
{
char comment_leader = '%'; /* define the coment leader */
char *line = NULL;
size_t len = 0;
ssize_t read;
// char buffer[BUFFERSIZE];
// argv[0] : the name of this executable
// argv[1] : the name the script calling this executable via shebang
FILE *input; /* input-file pointer */
char *input_file_name = argv[1]; /* the script name */
input = fopen( input_file_name, "r" );
if ( input == NULL ) {
fprintf ( stderr, "couldn't open file '%s'; %s\n",
input_file_name, strerror(errno) );
exit (EXIT_FAILURE);
}
while ((read = getline(&line, &len, input)) != -1) {
if ( line[0] != comment_leader ) {
printf( "%s", line ); /* print line as a test */
}
else {
printf ( "Skipped a comment!\n" );
}
}
free(line);
if( fclose(input) == EOF ) { /* close input file */
fprintf ( stderr, "couldn't close file '%s'; %s\n",
input_file_name, strerror(errno) );
exit (EXIT_FAILURE);
}
return EXIT_SUCCESS;
} /* ---------- end of function main ---------- */
Now call the script (made executable before) and see the output:
...~> ./first.myint
#!/usr/local/bin/myinterpreter
Skipped a comment!
2 xxxxxxxxxxx
333
444
Skipped a comment!
I made it work. I especially thank holgero for his tail opion trick
tail -n +2 $1 | $REALINTERP
That, and finding this answer on Stack overflow made it possible:
How to compile a linux shell script to be a standalone executable *binary* (i.e. not just e.g. chmod 755)?
"The solution that fully meets my needs would be SHC - a free tool"
SHC is a shell to C translator, see here:
http://www.datsi.fi.upm.es/~frosal/
So I wrote polyscript.sh:
$ cat polyscript.sh
#!/bin/bash
tail -n +2 $1 | poly
I compiled this with shc and in turn with gcc:
$ shc-3.8.9/shc -f polyscript.sh
$ gcc -Wall polyscript.sh.x.c -o polyscript
Now, I was able to create a first script written in ML:
$ cat smlscript
#!/home/gergoe/projects/shebang/polyscript $0
print "Hello World!"
and, I was able to run it:
$ chmod u+x smlscript
$ ./smlscript
Poly/ML 5.4.1 Release
> > # Hello World!val it = (): unit
Poly does not have an option to suppress compiler output, but that's not an issue here. It might be interesting to write polyscript directly in C as fgm suggested, but probably that wouldn't make it faster.
So, this is how simple it is. I welcome any comments.

Reading with cat: Stop when not receiving data

Is there any way to tell the cat command to stop reading when not receiving any data? maybe with some "timeout" that specifies for how long no data is incoming.
Any ideas?
There is a timeout(1) command. Example:
timeout 5s cat /dev/random
Dependening on your circumstances. E.g. you run bash with -e and care normally for the exit code.
timeout 5s cat /dev/random || true
cat itself, no. It reads the input stream until told it's the end of the file, blocking for input if necessary.
There's nothing to stop you writing your own cat equivalent which will use select on standard input to timeout if nothing is forthcoming fast enough, and exit under those conditions.
In fact, I once wrote a snail program (because a snail is slower than a cat) which took an extra argument of characters per second to slowly output a file (a).
So snail 10 myprog.c would output myprog.c at ten characters per second. For the life of me, I can't remember why I did this - I suspect I was just mucking about, waiting for some real work to show up.
Since you're having troubles with it, here's a version of dog.c (based on my afore-mentioned snail program) that will do what you want:
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <sys/select.h>
static int dofile (FILE *fin) {
int ch = ~EOF, rc;
fd_set fds;
struct timeval tv;
while (ch != EOF) {
// Set up for fin file, 5 second timeout.
FD_ZERO (&fds); FD_SET (fileno (fin), &fds);
tv.tv_sec = 5; tv.tv_usec = 0;
rc = select (fileno(fin)+1, &fds, NULL, NULL, &tv);
if (rc < 0) {
fprintf (stderr, "*** Error on select (%d)\n", errno);
return 1;
}
if (rc == 0) {
fprintf (stderr, "*** Timeout on select\n");
break;
}
// Data available, so it will not block.
if ((ch = fgetc (fin)) != EOF) putchar (ch);
}
return 0;
}
int main (int argc, char *argv[]) {
int argp, rc;
FILE *fin;
if (argc == 1)
rc = dofile (stdin);
else {
argp = 1;
while (argp < argc) {
if ((fin = fopen (argv[argp], "rb")) == NULL) {
fprintf (stderr, "*** Cannot open input file [%s] (%d)\n",
argv[argp], errno);
return 1;
}
rc = dofile (fin);
fclose (fin);
if (rc != 0)
break;
argp++;
}
}
return rc;
}
Then, you can simply run dog without arguments (so it will use standard input) and, after five seconds with no activity, it will output:
*** Timeout on select
(a) Actually, it was called slowcat but snail is much nicer and I'm not above a bit of minor revisionism if it makes the story sound better :-)
mbuffer, with its -W option, works for me.
I needed to sink stdin to a file, but with an idle timeout:
I did not need to actually concatenate multiple sources (but perhaps there are ways to use mbuffer for this.)
I did not need any of cat's possible output-formatting options.
I did not mind the progress bar that mbuffer brings to the table.
I did need to add -A /bin/false to suppress a warning, based on a suggestion in the linked man page. My invocation for copying stdin to a file with 10 second idle timeout ended up looking like
mbuffer -A /bin/false -W 10 -o ./the-output-file
Here is the code for timeout-cat:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
void timeout(int sig) {
exit(EXIT_FAILURE);
}
int main(int argc, char* argv[]) {
int sec = 0; /* seconds to timeout (0 = no timeout) */
int c;
if (argc > 1) {
sec = atoi(argv[1]);
signal(SIGALRM, timeout);
alarm(sec);
}
while((c = getchar()) != EOF) {
alarm(0);
putchar(c);
alarm(sec);
}
return EXIT_SUCCESS;
}
It does basically the same as paxdiablo's dog.
It works as a cat without an argument - catting the stdin. As a first argument provide timeout seconds.
One limitation (applies to dog as well) - lines are line-buffered, so you have n-seconds to provide a line (not any character) to reset the timeout alarm. This is because of readline.
usage:
instead of potentially endless:
cat < some_input > some_output
you can do compile code above to timeout_cat and:
./timeout_cat 5 < some_input > some_output
Try to consider tail -f --pid
I am assuming that you are reading some file and when the producer is finished (gone?) you stop.
Example that will process /var/log/messages until watcher.sh finishes.
./watcher.sh&
tail -f /var/log/messages --pid $! | ... do something with the output
I faced same issue of cat command blocking while reading on tty port via adb shell but did not find any solution (timeout command was also not working). Below is the final command I used in my python script (running on ubuntu) to make it non-blocking. Hope this will help someone.
bash_command = "adb shell \"echo -en 'ATI0\\r\\n' > /dev/ttyUSB0 && cat /dev/ttyUSB0\" & sleep 1; kill $!"
response = subprocess.check_output(['bash', '-c', bash_command])
Simply cat then kill the cat after 5 sec.
cat xyz & sleep 5; kill $!
Get the cat output as a reply after 5 seconds
reply="`cat xyz & sleep 5; kill $!`"
echo "reply=$reply"

Resources