I have an external process that start a write to a file. How do I write script that waits until the file is closed (when the other process done with the writing).
There are several ways to achieve this:
If you can, start the process from your script. The script will continue when the process terminates and that means it can't write any more data to the file.
If you can't control the process but you know that the process terminates after writing the file, you can find out the process ID and then check if the process is still running with kill -0 $PID. If $? is 0 afterwards, the process is still alive.
If that's not possible, then you can use lsof -np $PID to get a list of all open files for this process and check if your file is in the list. This is somewhat slow, though.
[EDIT] Note that all these approaches are somewhat brittle. The correct solution is to have the external process write the file using a temporary name and then rename it as soon as it's done.
The rename makes sure that everyone else either sees the whole file with all the data or nothing.
The easy way: let the script execute the program and wait until it's finished.
Create a small C program using inotify. The program should:
Create an inotify instance.
Add a watch to the instance for the IN_CLOSE_WRITE event for the file path of interest.
Wait for the event.
Exit with an appropriate code.
Then in your script, invoke the program with the file path as an argument.
You could extend this by adding a timeout argument, and allowing different events to be specified.
This isn't nice, and it makes me feel dirty writing it... /tmp/foo.txt is the file being tested...
#!/bin/bash
until [ "$( find /proc/*/fd 2> /dev/null |
xargs -i{} readlink {} |
grep -c '^/tmp/foo.txt$' )" == "0" ]; do
sleep 1;
done;
Loop until file is stable approach, this should work if you are waiting for experiment results (so you don't need real-time event handling):
EXPECTED_WRITE_INTERVAL_SECONDS=1
FILE="file.txt"
while : ; do
omod=$(stat -c %Y $FILE)
# echo "OLD: $omod"
sleep $EXPECTED_WRITE_INTERVAL_SECONDS
nmod=$(stat -c %Y $FILE)
# echo "NEW: $nmod"
if [ $omod == $nmod ] ; then
break
fi
done
Related
On a Linux server, I have a script here that will work fine when I start it from the terminal, but fail when started and then detached by another process. So there is probably a difference in the script's environment to fix.
The trouble is, the other process integrating that script does not provide access to its error messages when the script fails. What is an easy (and ideally generic) way to see the output of such a script when it's failing?
Let's assume I have no easy way to change the code of the process calling this script. The failure happens right at the start of the script's run, so there is not enough time to manually attach to it with strace to see its output.
(The specifics should not matter, but for what it's worth: the failing script is the backup script of Discourse, a widespread open source forum software. Discourse and this script are written in Ruby.)
The idea is to substitute original script with wrapper which calls original script and saves its stdin and stderr to files. Wrapper may be like this:
#!/bin/bash
exec /path/to/original/script "$#" 1> >(tee /tmp/out.log) 2> >(tee /tmp/err.log >&2)
1> >(tee /tmp/out.log) redirects stdout to tee /tmp/out.log input in subshell. tee /tmp/out.log passes it to stdout but saves copy to the file.
2> >(tee /tmp/err.log) redirects stderr to tee /tmp/err.log input in subshell. tee /tmp/err.log >&2 passes it to stderr but saves copy to the file.
If script is invoked multiple times you may want to append stdout and stderr to files. Use tee -a in this case.
The problem is how to force caller to execute wrapper script instead of original one.
If caller invokes script in a way that it is searched in PATH you can put wrapper script to a separate directory and provide modified PATH to the caller. For example, script name is script. Put wrapper to /some/dir/script and run caller as
$ PATH="/some/dir:$PATH" caller
/path/to/original/script in wrapper must be absolute.
If caller invokes script from specific path then you can rename original script e.g. to original-script and name wrapper as script. In this case wrapper should call /path/to/original/original-script.
Another problem may rise if script behaves differently depending on name it's called. In this case exec -a ... may be needed.
You can use a bash script that (1) does "busy waiting" until it sees the targeted process, and then (2) immediately attaches to it with strace and prints its output to the terminal.
#!/bin/sh
# Adapt to a regex that matches only your target process' full command.
name_pattern="bin/ruby.*spawn_backup_restore.rb"
# Wait for a process to start, based on its name, and capture its PID.
# Inspiration and details: https://unix.stackexchange.com/a/410075
pid=
while [ -z "$pid" ] ; do
pid="$(pgrep --full "$name_pattern" | head -n 1)"
# Set delay for next check to 1ms to try capturing all output.
# Remove completely if this is not enough to capture from the start.
sleep 0.001
done
echo "target process has started, pid is $pid"
# Print all stdout and stderr output of the process we found.
# Source and explanations: https://unix.stackexchange.com/a/58601
strace -p "$pid" -s 9999 -e write
First of all, sorry if the title is not clear or misleading, my question is not not exactly easy to be understood out of context.
So here it is: I am running a shell script (hello.sh) that needs to relocate itself from /root to /.
thus I made a simple recursion, to test from where the script is running and to make a temporary copy and launch it and exit(this last temporary copy will move the original file, and delete itself while still running).
#!/bin/sh
IsTMP=$(echo $0 | grep "tmp")
if [ -z "$IsTMP" ]; then
cp /root/hello.sh /tmp/hello.sh
/bin/sh /tmp/hello.sh &
exit
else
unlink /hello.sh
rsync /root/hello.sh /hello.sh
rm /root/hello.sh
rm /tmp/hello.sh
fi
while true; do
sleep 5
echo "Still Alive"
done
This script works totally well and suits my needs (even though it is a horrendous hack): the script is moved, and re-executed from a temporary place. However, when i pipe the shell script with a tee, just like:
/hello.sh | tee -a /log&
The behaviour is not the same:
hello.sh is exiting but not tee
When i try to kill tee, the temporary copy is automatically killed after a few seconds, without entering the infinite loop
This behaviour is the exact same if i replace tee with another binary (e.g. watch,...), so I am wondering if it comes from piping.
Sorry if i am not too clear about my problem.
Thanks in advance.
When i try to kill tee, the temporary copy is automatically killed after a few seconds, without entering the infinite loop
That's not the case. The script is entering the infinite loop, the few seconds are the five the sleep 5 in the loop pauses, and then it is killed by the signal SIGPIPE (Broken pipe) because it tries to echo "Still Alive" to the pipe which is closed on the read end since tee has been killed.
There is no link between tee and the second instance
That's not the case. There is a link, namely the pipe, the write end of which is the standard output of the parent as well as (inherited) the child shell script, and the read end is the standard input of tee. You can see this if you look at ls -l /proc/pid/fd, where pid is the process id of the script's shell on the one hand, and of tee on the other.
I am trying to stream a specific numbered file descriptor into a variable in Bash. I can do this from normal standard in using the following function, but, how do it do it from a specific file descriptor. I need to direct the FD into the sub-shell if I use the same approach. I could always do it reading line by line, but, if I can do it in a continuous stream then that would be massively preferable.
The function I have is:
streamStdInTo ()
{
local STORE_INvar="${1}" ; shift
printf -v "${STORE_INvar}" '%s' "$( cat - )"
}
Yes, I know that this wouldn't work normally as the end of a pipeline would be lost (due to its execution in a sub-shell), however, either in the context of the Bash 4 set +m ; shopt -s lastpipe method of executing the end of a pipeline in the same shell as the start, or, by directing into this via a different file descriptor I am hoping to be able to use it.
So, my question is, How do I use the above but with different file descriptors than the normal?
It's not entirely clear what you mean, but perhaps you are looking for something like:
cat - <&4 # read from fd 4
Or, just call your current function with the redirect:
streamStdInTo foo <&4
edit:
Addressing some questions from the comment, you can use a fifo:
#!/bin/bash
trap 'rm -f $f' 0
f=$(mktemp xxx)
rm $f
mkfifo $f
echo foo > $f &
exec 4< $f
cat - <&4
wait
I think there's a lot of confusion about what exactly you're trying to do. If I understand correctly the end goal here is to run a pipeline and capture the output in a variable, right? Kind of like this:
var=$(cmd1 | cmd2)
Except I guess the idea here is that the name of "$var" is stored in another variable:
varname=var
You can do an end-run around Bash's usual job control situation by using process substitution. So instead of this normal pipeline (which would work in ksh or zsh, but not in bash unless you set lastpipe):
cmd1 | cmd2 | read "$varname"
You would use this command, which is equivalent apart from how the shell handles the job:
read "$varname" < <(cmd1 | cmd2)
With process substitution, "read $varname" isn't run in a pipeline, so Bash doesn't fork to run it. (You could use your streamStdInTo() function there as well, of course)
As I understand it, you wanted to solve this problem by using numeric file descriptors:
cmd1 | cmd2 >&$fd1 &
read "$varname" <&$fd2
To create those file descriptors that connect the pipeline background job to the "read" command, what you need is called a pipe, or a fifo. These can be created without touching the file system (the shell does it all the time!) but the shell doesn't directly expose this functionality, which is why we need to resort to mkfifo to create a named pipe. A named pipe is a special file that exists on the filesystem, but the data you write to it doesn't go to the disk. It's a data queue stored in memory (a pipe). It doesn't need to stay on the filesystem after you've opened it, either, it can be deleted almost immediately:
pipedir=$(mktemp -d /tmp/pipe_maker_XXXX)
mkfifo ${pipedir}/pipe
exec {temp_fd}<>${pipedir}/pipe # Open both ends of the pipe
exec {fd1}>${pipedir}/pipe
exec {fd2}<${pipedir}/pipe
exec {temp_fd}<&- # Close the read/write FD
rm -rf ${pipedir} # Don't need the named FIFO any more
One of the difficulties in working with named pipes in the shell is that attempting to open them just for reading, or just for writing causes the call to block until something opens the other end of the pipe. You can get around that by opening one end in a background job before trying to open the other end, or by opening both ends at once as I did above.
The "{fd}<..." syntax dynamically assigns an unused file descriptor number to the variable $fd and opens the file on that file descriptor. It's been around in ksh for ages (since 1993?), but in Bash I think it only goes back to 4.1 (from 2010).
I followed this blog entry to parallelize sort by splitting a large file, sorting and merging.
The steps are:
split -l5000000 data.tsv '_tmp'
ls -1 _tmp* | while read FILE; do sort $FILE -o $FILE & done
sort -m _tmp* -o data.tsv.sorted
Between step 2 and 3, one must wait until the sorting step has finished.
I assumed that wait without any arguments would be the right thing, since according to the man page, if wait is called without arguments all currently active child processes are waited for.
However, when I try this in the shell (i.e. executing steps 1 and 2, and then wait), wait returns immediately, although top shows the sort processes are still running.
Ultimately I want to increase the speed of a script with that, so its not a one time thing I could do manually on the shell.
I know sort has a --parallel option since version 8, however on the cluster I am running this, an older version is installed, and I am also curious about how to solve this issue.
Here's a simple test case reproducing your problem:
true | { sleep 10 & }
wait
echo "This echos immediately"
The problem is that the pipe creates a subshell, and the forked processes are part of that subshell. The solution is to wait in that subshell instead of your main parent shell:
true | { sleep 10 & wait }
echo "This waits"
Translated back into your code, this means:
ls -1 _tmp* | { while read FILE; do sort $FILE -o $FILE & done; wait; }
From the bash man page:
Each command in a pipeline is executed as a separate process (i.e., in a subshell).
So when you pipe to while, a subshell is created. Everything else in step 2 is executed within this subshell, (ie, all the background processes). The script then exits the while loop, leaving the subshell, and wait is executed in the parent shell, where there is nothing to wait for. You can avoid using the pipeline by using a process substitution:
while read FILE; do
sort $FILE -o $FILE &
done < <(ls -1 _tmp*)
I am trying to write a script that starts rtmpsrv and waits for some output from it. rtmpsrv gives the desirable output and continues running, but the script is waiting for a termination of rtmpsrv. How do I gain access to the output of rtmpsrv without stopping it?
Well, I'm not familiar with rtmpsrv, but unless necessary you should wait for it to finish. However, you can probably redirect its output to a file, and then grep the file to see if it contains the string you are looking for.
(fictional code... you can expect syntax hell, just want to give you an idea)
nohup rtmpsrv >log.rtmpsrv 2>&1 &
...
while :; do
if ! result=$(grep "your desired line" log.rtmpsrv); then
echo "success: found $result"
break
fi
done
Note: the if constructs should work as per http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html - just to have nicer code, as #Charles Duffy suggested.
The most simple way is this:
rtmpsrv > logfile &
Then you can search logfile for the text that you're looking for. Meanwhile, rtmpsrv will do it's thing, completely unaware of your script.
This question contains examples how to wait in your script for a certain pattern to appear in the logfile (so you don't have to search it again and again): Do a tail -F until matching a pattern
Note: If you start the rtmpsrv process in the script and the script terminates, it will probably kill the rtmpsrv process. To avoid that use nohup.
Just attach to the process using gdb -p <pid> where the pid is the process id of your script.
You can find the pid of your running script by doing smth like this ps ax | grep <My Script's Name>
http://etbe.coker.com.au/2008/02/27/redirecting-output-from-a-running-process/