I want a daemon that continuously watches a named pipe for input, and then creates two files: one that contains all the data that went through the named pipe, and one that contains a filtered version of it.
When I run this command from the command line it works as intended, $ACTIONABLE_LOG_FILE is created as a filtered version of $LOG_FILE_BASENAME:
cat $NAMED_PIPE | tee -a "$LOG_FILE_BASENAME" | grep -P -v "$EXCEPTIONS" >> "$ACTIONABLE_LOG_FILE" &
But when I leave the following code running in the background, nothing gets appended to $ACTIONABLE_LOG_FILE:
while true
do
cat $NAMED_PIPE | tee -a "$LOG_FILE_BASENAME" | grep -P -v "$EXCEPTIONS" >> "$ACTIONABLE_LOG_FILE" &
wait $!
done
The file $ACTIONABLE_LOG_FILE gets created, but nothing gets appended to it. What's going on here?
My suspicion would be that when daemonized, you do not have a full environment available, and hence no $PATH. The full path to the command (likely /usr/bin/tee) might help a lot. You can confirm that locally with which tee.
Related
How can I wait for a program to complete and then start another program automatically in another console?
EDIT: The first program can be long running and the other program should start right after the completion of the first program.
tail -f --pid=xyzu /dev/null && second_program
for example, in one console (terminal) invoke cat. In another console, use pstree -p | grep cat to find its process id. Then, in that other console, type tail -f --pid=6169 /dev/null && ls (replace the number with the correct one and use the needed command instead of ls). Then, end cat with CTL-C.
EDIT: Another example is, you want to shutdown the computer automatically when a long running program has completed:
first, find the pid of the long running program:
pstree -p | grep long_running_program
For instance, you find the pid 3373. Now, use the command:
sudo tail -f --pid=3373 /dev/null && shutdown -h now
I am trying to use multiple process substitutions in a BASH command but I seem to be misunderstanding the order in which they resolve and redirect to each other.
The System
Ubuntu 18.04
BASH version - GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
The Problem
I am trying to redirect an output of a command into tee, have that redirect into ts (adding a timestamp) and then have that redirect into split (splitting the output into separate files). I can get the output to redirect into tee and ts but when redirecting into split I run into a problem.
My Attempts
command >(tee -a >(ts '[%Y-%m-%d %H:%M:%S]' > tempfile.txt)) - this will redirect the output into process substitution of tee then redirext to process substitution ts and add the timestamp then redirect to tempfile.txt this is what I would expect
command >(tee -a >(ts '[%Y-%m-%d %H:%M:%S]' >(split -d -b 10 -))) - this does nothing even though I would hope that the result would have been a bunch of 10 byte files with timestamps on the different rows.
To continue testing I tried with echo instead to see what happens
command >(tee -a >(ts '[%Y-%m-%d %H:%M:%S]' >(echo))) - the print from the initial tee prints (as it should) but the echo prints an empty line apparently this irrelevant because of new result I got - see edit at the bottom
command >(tee -a >(ts '[%Y-%m-%d %H:%M:%S]') >(split -d -b 10 -)) - This prints the command with the timestamp (as tee and ts should) and in addition creates 10 byte files with the command output (no timestamp on them). - this is what I expected and makes sense as the tee gets redirected to both process substitutions separately, it was mostly a sanity check
What I think is happening
From what I can tell >(ts '[%Y-%m-%d %H:%M:%S]' >(split -d -b 10 -)) are resolving first as a complete and separate command of its own. Thus split (and echo) are receiving an empty output from ts which has no output on its own. Only after this does the actual command resolve and send its output to its substitution tee.
This doesn't explain why command >(tee -a >(ts '[%Y-%m-%d %H:%M:%S]' > tempfile.txt)) does work as by this theory tee by itself has no output so ts should be receiving not input and should also output a blank.
All this is to say I am not really sure what is happening.
What I want
Basically I just want to understand how to make command >(tee -a >(ts '[%Y-%m-%d %H:%M:%S]' >(split -d -b 10 -))) work in the way it seems it should. I need the commands output to send itself to the process substitution tee which will send it to the process substitution ts and add the timestamps which will sent it to split and split the output to several small files.
I have tried command > >(echo) and saw the output is blank, which is not what I expected (i expected echo to receive and then output the command output). I think I am just very much misunderstanding how process substitution works at this point.
You can split send the error stream from the command into a different pipeline than the output, if that is desired:
{ { cmd 2>&3 | ts ... | split; } 3>&1 >&4 | ts ... | split; } 4>&1
This sends the output of cmd to the first pipeline, while the error stream from cmd goes into the 2nd pipe. File descriptor 3 is introduced to keep the error streams from ts and split separate, but that may be undesirable. fd 4 is introduced to prevent the output of split from being consumed by the second pipeline, and that may be unnecessary (if split does not produce any output, for example.)
One thing you could do if you really want to have one command redirect stdin/stderr to a separate ts|tee|split is this
command 1> >(ts '[%Y-%m-%d %H:%M:%S]' | tee -a >(split -d -b 10 -)) 2> >(ts '[%Y-%m-%d %H:%M:%S]' | tee -a >(split -d -b 10 -))
But the downside is tee only prints after the prompt gets printed. There is probably a way to avoid this by duplicating file descriptors, but this is the best I could think of.
This:
ts '[%Y-%m-%d %H:%M:%S]' >(split -d -b 10 -)
expands the file name generated by the process substitution on the command line of ts, so what gets run is something like ts '[%Y-%m-%d %H:%M:%S]' /dev/fd/63. ts then tries to open the fd that goes to split to read input from there, instead of reading from the original stdin.
That's probably not what you want, and on my machine, I got some copies of ts and split stuck in the background while testing. Possibly successfully connected to each other, which may explain the lack of error messages.
You probably meant to write
ts '[%Y-%m-%d %H:%M:%S]' > >(split -d -b 10 -)
^
with a redirection to the process substitution.
That said, you could just use a pipe there between ts and split.
I have been monitoring the performance of my Linux server with ioping (had some performance degradation last year). For this purpose I created a simple script:
echo $(date) | tee -a ../sb-output.log | tee -a ../iotest.txt
./ioping -c 10 . 2>&1 | tee -a ../sb-output.log | grep "requests completed in\|ioping" | grep -v "ioping statistics" | sed "s/^/IOPing I\/O\: /" | tee -a ../iotest.txt
./ioping -RD . 2>&1 | tee -a ../sb-output.log | grep "requests completed in\|ioping" | grep -v "ioping statistics" | sed "s/^/IOPing seek rate\: /" | tee -a ../iotest.txt
etc
The script calls ioping in the folder /home/bench/ioping-0.6. Then it saves the output in readable form in /home/bench/iotest.txt. It also adds the date so I can compare points in time.
Unfortunately I am no experienced programmer and this version of the script only works if you first enter the right directory (/home/bench/ioping-0.6).
I would like to call this script from anywhere. For example by calling
sh /home/bench/ioping.sh
Googling this and reading about path variables was a bit over my head. I kept up ending up with different version of
line 3: ./ioping: No such file or directory
Any thoughts on how to upgrade my scripts so that it works anywhere?
The trick is the shell's $0 variable. This is set to the path of the script.
#!/bin/sh
set -x
cd $(dirname $0)
pwd
cd ${0%/*}
pwd
If dirname isn't available for some reason, like some limited busybox distributions, you can try using shell parameter expansion tricks like the second one in my example.
Isn't it obvious? ioping is not in . so you can't use ./ioping.
Easiest solution is to set PATH to include the directory where ioping is. perhaps more robust - figure out the path to $0 and use that path as the location for ioping (assing your script sits next to ioping).
If iopinf itself depend on being ruin in a certain directory, you might have to make your script cd to the ioping directory before running.
How can a bash script ensure that not more than one copy of it is running?
I have tried:
ps -ef| grep /user/loca/shell/sh | wc -l
This shows me 2 because of the grep command. Change it to:
ps -ef | grep /user/loca/shell/sh | grep -v 'grep' wc -l
then its shows 1. However, if I then vim /user/loca/shell/sh and then execute:
ps -ef| grep /user/loca/shell/sh | grep -v 'grep' wc -l
that shows 2. But there is only one process, I have started.
How can the bash script check whether the process is running?
The idiom often used in Unix Daemons is to create a file with the PID in it when they start.
Then you can check the existence and/or content of the file when you start, if it's there, you exit and leave that instance running.
At the end of the script you delete the file, ready to run next time.
The only problem comes when the script runs, but does not complete for some reason, leaving the PID file in existence. This can be taken care of by checking its timestamp, if it's too long ago, you assume that it was an aborted run and continue with the current one.
Try pgrep instead of 'ps -ef ...' in your script:
pgrep /usr/loca/shell/sh
If that won't work then I'd resort to locking by attempting to create a symbolic link from the script if one does not exist yet and bail out if it already exists, and deleting it upon exit. Creating symbolic link is an atomic operation on unix that can be used as sort of a poorman's lock for shell scripts.
I use a file lock:
MY_LOCK_FILE='whatever.lock'
( if (flock -nx 200); then
# Do stuff here...
fi
) 200>"$MY_LOCK_FILE"
I have some scripts where I need to see the output and log the result to a file, with the simplest example being:
$ update-client > my.log
I want to be able to see the output of the command while it's running, but also have it logged to the file. I also log stderr, so I would want to be able to log the error stream while seeing it as well.
update-client 2>&1 | tee my.log
2>&1 redirects standard error to standard output, and tee sends its standard input to standard output and the file.
Just use tail to watch the file as it's updated. Background your original process by adding & after your above command After you execute the command above just use
$ tail -f my.log
It will continuously update. (note it won't tell you when the file has finished running so you can output something to the log to tell you it finished. Ctrl-c to exit tail)
You can use the tee command for that:
command | tee /path/to/logfile
The equivelent without writing to the shell would be:
command > /path/to/logfile
If you want to append (>>) and show the output in the shell, use the -a option:
command | tee -a /path/to/logfile
Please note that the pipe will catch stdout only, errors to stderr are not processed by the pipe with tee. If you want to log errors (from stderr), use:
command 2>&1 | tee /path/to/logfile
This means: run command and redirect the stderr stream (2) to stdout (1). That will be passed to the pipe with the tee application.
Learn about this at askubuntu site
another option is to use block based output capture from within the script (not sure if that is the correct technical term).
Example
#!/bin/bash
{
echo "I will be sent to screen and file"
ls ~
} 2>&1 | tee -a /tmp/logfile.log
echo "I will be sent to just terminal"
I like to have more control and flexibility - so I prefer this way.