Considering the following example, emulating a command which gives output after 10 seconds: exec 5< <(sleep 10; pwd)
In Solaris, if I check the file descriptor earlier than 10 seconds, I can see that it has a size of 0 and this tells me that it hasn't been populated with data yet. I can simply check every second until the file test condition is met (different from 0) and then pull the data:
while true; do
if [[ -s /proc/$$/fd/5 ]]; then
variable=$(cat <&5)
break
fi
sleep 1
done
But in Linux I can't do this (RedHat, Debian etc). All file descriptors appear with a size of 64 bytes no matter if they hold data or not. For various commands that will take a variable amount of time to dump their output, I will not know when I should read the file descriptor. No, I don't want to just wait for cat <&5 to finish, I need to know when I should perform the cat in the first place. Because I am using this mechanism to issue simultaneous commands and assign their output to corresponding file descriptors. As mentioned already, this works great in Solaris.
Here is the skeleton of an idea :
#!/bin/bash
exec 5< <(sleep 4; pwd)
while true
do
if
read -t 0 -u 5 dummy
then
echo Data available
cat <&5
break
else
echo No data
fi
sleep 1
done
From the Bash reference manual :
If timeout is 0, read returns immediately, without trying to read and
data. The exit status is 0 if input is available on the specified file
descriptor, non-zero otherwise.
The idea is to use read with -t 0 (to have zero timeout) and -u 5 (read from file descriptor 5) to instantly check for data availability.
Of course this is just a toy loop to demonstrate the concept.
The solution given by User Fred using only bash builtins works fine, but is a tiny bit non-optimal due to polling for the state of a file descriptor. If calling another interpreter (for example Python) is not a no-go, a non-polling version is possible:
#! /bin/bash
(
sleep 4
echo "This is the data coming now"
echo "More data"
) | (
python3 -c 'import select;select.select([0],[],[])'
echo "Data is now available and can be processed"
# Replace with more sophisticated real-world processing, of course:
cat
)
The single line python3 -c 'import select;select.select([0],[],[])' waits until STDIN has data ready. It uses the standard select(2) system call, for which I have not found a direct shell equivalent or wrapper.
Related
This question already has answers here:
Quick-and-dirty way to ensure only one instance of a shell script is running at a time
(43 answers)
Closed 4 years ago.
I have a bash script and sometimes happened that, even my script was scheduled, it was executed two times. So I added few code lines to check if the script is already in execution. Initially I hadn't a problem, but in the last three days I had got it again
PID=`echo $$`
PROCESS=${SL_ROOT_FOLDER}/xxx/xxx/xxx_xxx_$PID.txt
ps auxww | grep $scriptToVerify | grep -v $$ | grep -v grep > $PROCESS
num_proc=`awk -v RS='\n' 'END{print NR}' $PROCESS`
if [ $num_proc -gt 1 ];
then
sl_log "---------------------------Warning---------------------------"
sl_log "$scriptToVerify already executed"
sl_log "num proc $num_proc"
sl_log "--------"
sl_log $PROCESS
sl_log "--------"
exit 0;
fi
In this way I check how many rows I've got into my log and if the result is more than one, then I have two process in execution and one will be stopped.
This method doesn't work correctly, though. How can I fix my code to check how many scripts in execution I have?
Anything that involves:
read some state information
check results
do action based on results
finish
must do all three steps at the same time otherwise there is a "race" condition. For example:
(A) reads state
(A) checks results (ok)
(A) does action (ok)
(A) finishes
(B) reads state
(B) checks results (bad)
(B) does action (bad)
(B) finishes
but if timing is slightly different:
(A) reads state
(A) checks results (ok)
(B) reads state
(B) checks results (ok)
(A) does action (ok)
(B) does action (ok)
(A) finishes
(B) finishes
The usual example people give is updating bank balances.
Using your method, you may be able to reduce the frequency of your code running when it shouldn't but you can never avoid it entirely.
A better solution is to use locking. This guarantees that only one process can run at a time. For example, using flock, you can wrap all calls to your script:
flock -x /var/lock/myscript-lockfile myscript
Or, inside your script, you can do something like:
exec 300>/var/lock/myscript-lockfile
flock -x 300
# rest of script
flock -u 300
or:
exec 300>/var/logk/myscript-lockfile
if ! flock -nx 300; then
# another process is running
exit 1
fi
# rest of script
flock -u 300
flock(1)
#!/bin/bash
# Makes sure we exit if flock fails.
set -e
(
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200
# Do stuff
) 200>/var/lock/.myscript.exclusivelock
This ensures that code between "(" and ")" is run only by one process at a time and that the process does wait for a lock too long.
Credit goes to Alex B.
I am trying to write a script to provide diagnostics on processes. I have submitted a script to a job scheduling server using qsub. I can easily find the node that the job gets sent to. But I would like to be able to find what process is currently being run.
ie. I have a list of different commands in the submitted script, how can I find the current one that is running, and the arguments passed to it?
example of commands in script
matlab -nodesktop -nosplash -r "display('here'),quit"
python runsomethings.py
I would like to see whether the nodes is currently executing the first or second line.
When you submit a job, pbs_server pass your task to pbs_mom. pbs_mom process/daemon actually executes your script on the execution node. It
"creates a new session as identical user."
This means invoking a shell. You specialize the shell at the top of the script marking your choice with shebang: #!/bin/bash).
It's clear, that pbs_mom stores process (shell) PID somewhere to kill the job and to monitor if the job (shell process) have finished.
UPD. based on #Dmitri Chubarov comment: pbs_mom stores subshell PID internally in memory after calling fork(), and in the .TK file which is under torque installation directory: /var/spool/torque/mom_priv/jobs on my system.
Dumping file internals in decimal mode (<job_number>, <queue_name> should be your own values):
$ hexdump -d /var/spool/torque/mom_priv/jobs/<job_number>.<queue_name>.TK
have disclosed, that in my torque implementation it is stored in position
00000890 + offset 4*2 = 00000898 (it is hex value of first byte of PID in .TK file) and has a length of 2 bytes.
For example, for shell PID=27110 I have:
0000890 00001 00000 00001 00000 27110 00000 00000 00000
Let's recover PID from .TK file:
$ hexdump -s 2200 -n 2 -d /var/spool/torque/mom_priv/jobs/<job_number>.<queue_name>.TK | tr -s ' ' | cut -s -d' ' -f 2
27110
This way you've found subshell PID.
Now, monitor process list on the execution node and find name of child processes (getcpid function is a slighlty modified version of that posted earlier on SO):
function getcpid() {
cpids=`pgrep -P $1|xargs`
for cpid in $cpids;
do
ps -p "$cpid" -o comm=
getcpid $cpid
done
}
At last,
getcpid <your_PID>
gives you the child processes' names (note, there will be some garbage lines, like task numbers). This way you will finally know, what command is currently running on the execution node.
Of course, for each task monitored, you should obtain the PID and process name on the execution node after doing
ssh <your node>
You can automatically retrieve node name(s) in <node/proc+node/proc+...> format (process it further to obtain bare node names):
qstat -n <job number> | awk '{print $NF}' | grep <pattern_for_your_node_names>
Note:
The PID method is reliable and, as I believe, optimal.
Search by name is worse, it provides you unambiguous result only if your invoke different commands in your scripts, and no user executes the same software on the node.
ssh <your node>
ps aux | grep matlab
You will know if matlab runs.
Simple and elegant way to do it is to print to a log file
`
ARGS=" $A $B $test "
echo "running MATLAB now with args: $ARGS" >> $LOGFILE
matlab -nodesktop -nosplash -r "display('here'),quit"
PYARGS="$X $Y"
echo "running Python now with args: $ARGS" >> $LOGFILE
python runsomethings.py
`
And monitor the output of $LOGFILE using tail -f $LOGFILE
I am trying to figure out a way to monitor the files I am dumping from my script. If there is no increment seen in child files then kill my script.
I am doing this to free up the resources when not needed. Here is what I think of , but I think my apporch is going to add burden to CPU. Can anyone please suggest more efficent way of doing this?
Below script is suppose to poll in every 15 sec and collect two file size of same file, if the size of the two samples are same then exit.
checkUsage() {
while true; do
sleep 15
fileSize=$(stat -c%s $1)
sleep 10;
fileSizeNew=$(stat -c%s $1)
if [ "$fileSize" == "$fileSizeNew" ]; then
echo -e "[Error]: No activity noted on this window from 5 sec. Exiting..."
kill -9 $$
fi
done
}
And I am planning to call it as follow (in background):
checkUsage /var/log/messages &
I can also get solution if, someone suggest how to monitor tail command and if nothing printing on tail then exit. NOT SURE WHY PEOPLE ARE CONFUSED. End goal of this question is to ,check if the some file is edited in last 15 seconds. If not exit or throw some error.
I have achived this by above script,but I don't know if this is the smartest way of achiveing this. I have asked this question to know views from other if there is any alternative way or better way of doing it.
I would based the check on file modification time instead of size, so something like this (untested code):
checkUsage() {
while true; do
# Test if file mtime is 'second arg' seconds older than date, default to 10 seconds
if [ $(( $(date +"%s") - $(stat -c%Y /var/log/message) )) -gt ${2-10} ]; then
echo -e "[Error]: No activity noted on this window from ${2-10} sec. Exiting..."
return 1
fi
#Sleep 'first arg' second, 15 seconds by default
sleep ${1-15}
done
}
The idea is to compare the file mtime with current time, if it's greater than second argument seconds, print the message and return.
And then I would call it like this later (or with no args to use defaults):
[ checkusage 20 10 ] || exit 1
Which would exit the script with code 1 as when the function return from it's infinite loop (as long as the file is modified)
Edit: reading me again, the target file could be a parameter too, to allow a better reuse of the function, left as an exercise to the reader.
If on Linux, in a local file system (Ext4, BTRFS, ...) -not a network file system- then you could consider inotify(7) facilities: something could be triggered when some file or directory changes or is accessed.
In particular, you might have some incron job thru incrontab(5) file; maybe it could communicate with some other job ...
PS. I am not sure to understand what you really want to do...
I suppose an external programme is modifying /var/log/messages.
If this is the case, below is my script (with minor changes to yours)
#Bash script to monitor changes to file
#!/bin/bash
checkUsage() # Note that U is in caps
{
while true
do
sleep 15
fileSize=$(stat -c%s $1)
sleep 10;
fileSizeNew=$(stat -c%s $1)
if [ "$fileSize" == "$fileSizeNew" ]
then
echo -e "[Notice : ] no changes noted in $1 : gracefully exiting"
exit # previously this was kill -9 $$
# changing this to exit would end the program gracefully.
# use kill -9 to kill a process which is not under your control.
# kill -9 sends the SIGKILL signal.
fi
done
}
checkUsage $1 # I have added this to your script
#End of the script
Save the script as checkusage and run it like :
./checkusage /var/log/messages &
Edit :
Since you're looking for better solutions I would suggest inotifywait, thanks for the suggestion from the other answerer.
Below would be my code :
while inotifywait -t 10 -q -e modify $1 >/dev/null
do
sleep 15 # as you said the polling would happen in 15 seconds.
done
echo "Script exited gracefully : $1 has not been changed"
Below are the details from the inotifywait manpage
-t <seconds>, --timeout <seconds> Exit if an appropriate event has not occurred within <seconds> seconds. If <seconds> is zero (the default),
wait indefinitely for an event.
-e <event>, --event <event> Listen for specific event(s) only. The events which can be listened for are listed in the EVENTS section.
This option can be specified more than once. If omitted, all events
are listened for.
-q, --quiet If specified once, the program will be less verbose. Specifically, it will not state when it has completed establishing all
inotify watches.
modify(Event) A watched file or a file within a watched directory was
written to.
Notes
You might have to install the inotify-tools first to make use of the inotifywait command. Check the inotify-tools page at Github.
I'd like to provide an optional logging parameter in a bash script, and would like to use exec to tee a pipe from the beginning. However, opening the tee process is causing the script to hang, I believe because stdout is not closed:
# Output to a log file, if set
if [[ $OPT_LOG ]]; then
exec > >(tee -a $OPT_LOG)
fi
I've attempted to close with:
exec >&-
But it still hangs - is there another way to properly close tee so the script will exit properly at the end of execution?
It seems like, for some reason, using tee stops the prompt ($PS1) from appearing because the shell script has not exited. As a workaround, I generally use a short sleep after calling tee.
#!/bin/bash
exec > >(tee -a mylog)
sleep .1
# my code
Converting comments into an answer, with minor edits.
I noted:
The following code exits OK for me on Mac OS X 10.10.3.
OPT_LOG=file.name
if [[ $OPT_LOG ]]
then exec > >(tee -a $OPT_LOG)
fi
for ((i = 0; i < 10; i++))
do
echo "Logging message $i at $(date)"
sleep 1
done
Your problem is probably in the code you've not shown. What commands are you running? What do you get from bash -x yourscript.sh?
And chatraed observed
If you remove the sleep and date calls from your example, the script does not exit properly any more, as told by Andrew:
OPT_LOG=file.name;
if [[ $OPT_LOG ]]; then exec > >(tee -a $OPT_LOG); fi;
for ((i = 0; i < 10; i++)); do echo "Logging message $i"; done
And I responded:
Now that's an interesting observation! I can reproduce your result.
I experimented a bit:
I added a pwd before the for loop with 'just the echo', and that didn't affect things (but pwd is probably a Bash built-in).
I added bash -c 'exit 0' (which is not a built-in) and the code terminated OK.
I tried using > >(tee -a $OPT_LOG &) and the output didn't appear — on screen or in the file. (I find this surprising, but it was an attempted workaround, not a major part of the work.)
My impression is that Andrew has found a bug in Bash that could viably be reported. (See the Bash manual on Reporting Bugs for how to do that.) I don't think that not exiting simply because no external commands have been executed since the I/O redirection is OK. I can confirm that Apple's Bash 3.2.57 has the problem; so does Bash 4.3.27 (which I built for myself so it is patched with the fixes to the ShellShock bug).
It is relatively unusual that a shell script does not invoke any external command, and invoking any external command after the exec redirection seems to suppress the bug.
It also confirms that chatraed's workaround works, albeit more slowly than minimally necessary. For production use, I'd choose sleep 0, which is most of a tenth of a second faster than sleep 0.1. And sleeping for a fraction of a second only works on some systems; classically (and according to POSIX), sleep does not sleep for fractional seconds. OTOH, if you write sleep 0.1, on systems without support for fractional seconds, including the leading 0 probably makes it sleep for zero seconds anyway; writing .1 might or might not have the same effect.
I have tty device in /dev , where I send AT commands. I want to read line by line and stop reading file after timeout.
You can use the program stty to configure the tty device. To see the settings for terminal /dev/ttyS0, try
stty -a -F /dev/ttyS0
The default settings regarding timeout are min = 1; time = 0, which means that the reading program will read until at least one character has been read and there is no timeout. Using e.g.
stty -F /dev/ttyS0 min 0 time 10
the reading program (e.g. cat) will finish reading after one second whether anything has been read or not. The unit for the parameter time is tenths of a second; you can check out man stty for more information.
Compiling some info from here, you can have a script in the sorts of:
#!/bin/bash
#SPECIFYING THE SERIAL PORT
SERIAL=ttyS0
#SETTING UP AN ERROR FLAG
FLAG="GO"
#OPENING SERIAL PORT FOR READING
exec 99</dev/${SERIAL}
#READING FROM SERIAL
while ["${FLAG}" == "GO" ]
do
#IF NO INPUT IS READ AFTER 5 SECONDS, AN ERROR FLAG IS RAISED
read -t 5 INPUT <&99
STATUS=$?
if test $STATUS -ne 0;
then
FLAG="ERROR"
fi
done
#CLOSING SERIAL PORT
exec 99>&-
While FLAG==GO, the script will read one line at a time from the serial port. The STATUS variable gets the return of READ command. According to the manual READ will return anything different than 0 if the specified timeout is reached; when that happens, FLAG is updated, exiting the read loop.