Shell script - Output to both the terminal and a log file in a sub-shell - linux

I have a few shell scripts that are intended to work together. The first script (script1.sh) calls the next script in a sub-shell. The second script (script2.sh) needs to "return" something for the first script to use. I need the last line that is echoed in the second script. However, when I use it this way, any output via echo in the second script does not get output to the terminal. I want all output from the first and second (and third, and fourth, ...) to be output to the terminal, but also written to a log file.
script1.sh:
#!/bin/sh
func_one() {
RESULT=$(./script2.sh | tail -1)
echo "RESULT: $RESULT"
}
func_one | tee log_file.log
script2.sh:
#!/bin/sh
echo "Hello"
echo "World!"
Attempt 1 output:
$ ./script1.sh
RESULT: World!
$
log_file.log contents:
RESULT: World!
If I try to redirect output in the second script, then it outputs to the terminal, but not to the log file:
script2.sh:
#!/bin/sh
echo "Hello" >&2
echo "World!" >&2
Attempt 2 output:
$ ./script1.sh
Hello
World!
RESULT:
log_file.log contents:
RESULT:
I also tried outputting to terminal and tee on the same line in script 1:
func_one >&2 | tee log_file.log
But that gives the same result as the first attempt.
What I would like is to have both output to the terminal AND written to the .log file: (if it was working correctly)
$ ./script1.sh
Hello
World!
RESULT: World!
$
log_file.log contents:
Hello
World!
RESULT: World!
How can I go about getting this result? Also, it would be preferred to NOT use bash, as a few of our machines we are going to be running this on do not have bash.
I've looked here:
How do I get both STDOUT and STDERR to go to the terminal and a log file?
but that didn't help in my case.

To get all the output of script2.sh sent to the terminal without interfering with the work of script1.sh, try this modification of script1.sh:
$ cat script1.sh
#!/bin/bash
func_one() {
RESULT=$(./script2.sh | tee >(cat >&2) | tail -1)
echo "RESULT: $RESULT"
}
func_one | tee log_file.log
Here, the first tee command makes sure that all script2.sh output appears, via stderr, on the terminal. To do this, process substitution is needed (and this, in turn, requires an upgrade from sh to bash).
The output is:
$ ./script1.sh
Hello
World!
RESULT: World!
Variation
This is the same as the above except that we don't touch stderr (you may want to reserve that errors). Here, we create an additional file descripter, 3, to duplicate stdout:
#!/bin/bash
exec 3>&1
func_one() {
RESULT=$(./script2.sh | tee >(cat >&3) | tail -1)
echo "RESULT: $RESULT"
}
func_one | tee log_file.log

Related

Bash command with pipe('|') alway return exit code of 0, even in error case [duplicate]

I want to execute a long running command in Bash, and both capture its exit status, and tee its output.
So I do this:
command | tee out.txt
ST=$?
The problem is that the variable ST captures the exit status of tee and not of command. How can I solve this?
Note that command is long running and redirecting the output to a file to view it later is not a good solution for me.
There is an internal Bash variable called $PIPESTATUS; it’s an array that holds the exit status of each command in your last foreground pipeline of commands.
<command> | tee out.txt ; test ${PIPESTATUS[0]} -eq 0
Or another alternative which also works with other shells (like zsh) would be to enable pipefail:
set -o pipefail
...
The first option does not work with zsh due to a little bit different syntax.
Dumb solution: Connecting them through a named pipe (mkfifo). Then the command can be run second.
mkfifo pipe
tee out.txt < pipe &
command > pipe
echo $?
using bash's set -o pipefail is helpful
pipefail: the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
There's an array that gives you the exit status of each command in a pipe.
$ cat x| sed 's///'
cat: x: No such file or directory
$ echo $?
0
$ cat x| sed 's///'
cat: x: No such file or directory
$ echo ${PIPESTATUS[*]}
1 0
$ touch x
$ cat x| sed 's'
sed: 1: "s": substitute pattern can not be delimited by newline or backslash
$ echo ${PIPESTATUS[*]}
0 1
This solution works without using bash specific features or temporary files. Bonus: in the end the exit status is actually an exit status and not some string in a file.
Situation:
someprog | filter
you want the exit status from someprog and the output from filter.
Here is my solution:
((((someprog; echo $? >&3) | filter >&4) 3>&1) | (read xs; exit $xs)) 4>&1
echo $?
See my answer for the same question on unix.stackexchange.com for a detailed explanation and an alternative without subshells and some caveats.
By combining PIPESTATUS[0] and the result of executing the exit command in a subshell, you can directly access the return value of your initial command:
command | tee ; ( exit ${PIPESTATUS[0]} )
Here's an example:
# the "false" shell built-in command returns 1
false | tee ; ( exit ${PIPESTATUS[0]} )
echo "return value: $?"
will give you:
return value: 1
So I wanted to contribute an answer like lesmana's, but I think mine is perhaps a little simpler and slightly more advantageous pure-Bourne-shell solution:
# You want to pipe command1 through command2:
exec 4>&1
exitstatus=`{ { command1; printf $? 1>&3; } | command2 1>&4; } 3>&1`
# $exitstatus now has command1's exit status.
I think this is best explained from the inside out - command1 will execute and print its regular output on stdout (file descriptor 1), then once it's done, printf will execute and print icommand1's exit code on its stdout, but that stdout is redirected to file descriptor 3.
While command1 is running, its stdout is being piped to command2 (printf's output never makes it to command2 because we send it to file descriptor 3 instead of 1, which is what the pipe reads). Then we redirect command2's output to file descriptor 4, so that it also stays out of file descriptor 1 - because we want file descriptor 1 free for a little bit later, because we will bring the printf output on file descriptor 3 back down into file descriptor 1 - because that's what the command substitution (the backticks), will capture and that's what will get placed into the variable.
The final bit of magic is that first exec 4>&1 we did as a separate command - it opens file descriptor 4 as a copy of the external shell's stdout. Command substitution will capture whatever is written on standard out from the perspective of the commands inside it - but since command2's output is going to file descriptor 4 as far as the command substitution is concerned, the command substitution doesn't capture it - however once it gets "out" of the command substitution it is effectively still going to the script's overall file descriptor 1.
(The exec 4>&1 has to be a separate command because many common shells don't like it when you try to write to a file descriptor inside a command substitution, that is opened in the "external" command that is using the substitution. So this is the simplest portable way to do it.)
You can look at it in a less technical and more playful way, as if the outputs of the commands are leapfrogging each other: command1 pipes to command2, then the printf's output jumps over command 2 so that command2 doesn't catch it, and then command 2's output jumps over and out of the command substitution just as printf lands just in time to get captured by the substitution so that it ends up in the variable, and command2's output goes on its merry way being written to the standard output, just as in a normal pipe.
Also, as I understand it, $? will still contain the return code of the second command in the pipe, because variable assignments, command substitutions, and compound commands are all effectively transparent to the return code of the command inside them, so the return status of command2 should get propagated out - this, and not having to define an additional function, is why I think this might be a somewhat better solution than the one proposed by lesmana.
Per the caveats lesmana mentions, it's possible that command1 will at some point end up using file descriptors 3 or 4, so to be more robust, you would do:
exec 4>&1
exitstatus=`{ { command1 3>&-; printf $? 1>&3; } 4>&- | command2 1>&4; } 3>&1`
exec 4>&-
Note that I use compound commands in my example, but subshells (using ( ) instead of { } will also work, though may perhaps be less efficient.)
Commands inherit file descriptors from the process that launches them, so the entire second line will inherit file descriptor four, and the compound command followed by 3>&1 will inherit the file descriptor three. So the 4>&- makes sure that the inner compound command will not inherit file descriptor four, and the 3>&- will not inherit file descriptor three, so command1 gets a 'cleaner', more standard environment. You could also move the inner 4>&- next to the 3>&-, but I figure why not just limit its scope as much as possible.
I'm not sure how often things use file descriptor three and four directly - I think most of the time programs use syscalls that return not-used-at-the-moment file descriptors, but sometimes code writes to file descriptor 3 directly, I guess (I could imagine a program checking a file descriptor to see if it's open, and using it if it is, or behaving differently accordingly if it's not). So the latter is probably best to keep in mind and use for general-purpose cases.
(command | tee out.txt; exit ${PIPESTATUS[0]})
Unlike #cODAR's answer this returns the original exit code of the first command and not only 0 for success and 127 for failure. But as #Chaoran pointed out you can just call ${PIPESTATUS[0]}. It is important however that all is put into brackets.
In Ubuntu and Debian, you can apt-get install moreutils. This contains a utility called mispipe that returns the exit status of the first command in the pipe.
Outside of bash, you can do:
bash -o pipefail -c "command1 | tee output"
This is useful for example in ninja scripts where the shell is expected to be /bin/sh.
The simplest way to do this in plain bash is to use process substitution instead of a pipeline. There are several differences, but they probably don't matter very much for your use case:
When running a pipeline, bash waits until all processes complete.
Sending Ctrl-C to bash makes it kill all the processes of a pipeline, not just the main one.
The pipefail option and the PIPESTATUS variable are irrelevant to process substitution.
Possibly more
With process substitution, bash just starts the process and forgets about it, it's not even visible in jobs.
Mentioned differences aside, consumer < <(producer) and producer | consumer are essentially equivalent.
If you want to flip which one is the "main" process, you just flip the commands and the direction of the substitution to producer > >(consumer). In your case:
command > >(tee out.txt)
Example:
$ { echo "hello world"; false; } > >(tee out.txt)
hello world
$ echo $?
1
$ cat out.txt
hello world
$ echo "hello world" > >(tee out.txt)
hello world
$ echo $?
0
$ cat out.txt
hello world
As I said, there are differences from the pipe expression. The process may never stop running, unless it is sensitive to the pipe closing. In particular, it may keep writing things to your stdout, which may be confusing.
PIPESTATUS[#] must be copied to an array immediately after the pipe command returns.
Any reads of PIPESTATUS[#] will erase the contents.
Copy it to another array if you plan on checking the status of all pipe commands.
"$?" is the same value as the last element of "${PIPESTATUS[#]}",
and reading it seems to destroy "${PIPESTATUS[#]}", but I haven't absolutely verified this.
declare -a PSA
cmd1 | cmd2 | cmd3
PSA=( "${PIPESTATUS[#]}" )
This will not work if the pipe is in a sub-shell. For a solution to that problem,
see bash pipestatus in backticked command?
Base on #brian-s-wilson 's answer; this bash helper function:
pipestatus() {
local S=("${PIPESTATUS[#]}")
if test -n "$*"
then test "$*" = "${S[*]}"
else ! [[ "${S[#]}" =~ [^0\ ] ]]
fi
}
used thus:
1: get_bad_things must succeed, but it should produce no output; but we want to see output that it does produce
get_bad_things | grep '^'
pipeinfo 0 1 || return
2: all pipeline must succeed
thing | something -q | thingy
pipeinfo || return
Pure shell solution:
% rm -f error.flag; echo hello world \
| (cat || echo "First command failed: $?" >> error.flag) \
| (cat || echo "Second command failed: $?" >> error.flag) \
| (cat || echo "Third command failed: $?" >> error.flag) \
; test -s error.flag && (echo Some command failed: ; cat error.flag)
hello world
And now with the second cat replaced by false:
% rm -f error.flag; echo hello world \
| (cat || echo "First command failed: $?" >> error.flag) \
| (false || echo "Second command failed: $?" >> error.flag) \
| (cat || echo "Third command failed: $?" >> error.flag) \
; test -s error.flag && (echo Some command failed: ; cat error.flag)
Some command failed:
Second command failed: 1
First command failed: 141
Please note the first cat fails as well, because it's stdout gets closed on it. The order of the failed commands in the log is correct in this example, but don't rely on it.
This method allows for capturing stdout and stderr for the individual commands so you can then dump that as well into a log file if an error occurs, or just delete it if no error (like the output of dd).
It may sometimes be simpler and clearer to use an external command, rather than digging into the details of bash. pipeline, from the minimal process scripting language execline, exits with the return code of the second command*, just like a sh pipeline does, but unlike sh, it allows reversing the direction of the pipe, so that we can capture the return code of the producer process (the below is all on the sh command line, but with execline installed):
$ # using the full execline grammar with the execlineb parser:
$ execlineb -c 'pipeline { echo "hello world" } tee out.txt'
hello world
$ cat out.txt
hello world
$ # for these simple examples, one can forego the parser and just use "" as a separator
$ # traditional order
$ pipeline echo "hello world" "" tee out.txt
hello world
$ # "write" order (second command writes rather than reads)
$ pipeline -w tee out.txt "" echo "hello world"
hello world
$ # pipeline execs into the second command, so that's the RC we get
$ pipeline -w tee out.txt "" false; echo $?
1
$ pipeline -w tee out.txt "" true; echo $?
0
$ # output and exit status
$ pipeline -w tee out.txt "" sh -c "echo 'hello world'; exit 42"; echo "RC: $?"
hello world
RC: 42
$ cat out.txt
hello world
Using pipeline has the same differences to native bash pipelines as the bash process substitution used in answer #43972501.
* Actually pipeline doesn't exit at all unless there is an error. It executes into the second command, so it's the second command that does the returning.
Why not use stderr? Like so:
(
# Our long-running process that exits abnormally
( for i in {1..100} ; do echo ploop ; sleep 0.5 ; done ; exit 5 )
echo $? 1>&2 # We pass the exit status of our long-running process to stderr (fd 2).
) | tee ploop.out
So ploop.out receives the stdout. stderr receives the exit status of the long running process. This has the benefit of being completely POSIX-compatible.
(Well, with the exception of the range expression in the example long-running process, but that's not really relevant.)
Here's what this looks like:
...
ploop
ploop
ploop
ploop
ploop
ploop
ploop
ploop
ploop
ploop
5
Note that the return code 5 does not get output to the file ploop.out.

Log command line plus its output in a bash script

Is there a way for a script to log both, the command line run (including piped ones) plus its output without duplicating the line for the command?
The intention is that the script should have a clean output, but should log verbosely into a log file (so no set -x). Apart from the output, it shall also log the command line causing the output, which could be a piped command-one liner.
The most basic approach is to duplicate the command line in the script and then dump it into the log followed by the captured output of the actual command being run:
echo "command argument1 \"quoted argument2\" | grep -oE \"some output\"" >> file.log
output="$(command argument1 "quoted argument2" 2>&1 | grep -oE "some output")"
echo "${output}" >> file.log
This has the side effect that quoted sections would need to be escaped for the log, which can lead to errors resulting in confusion.
If none of the commands were piped, one could store the command line in an array and then "run" the array.
command=(command argument1 "quoted argument2")
echo "${command[#]}" >> file.log
output="$("${command[#]}" 2>&1)"
echo "${output}" >> file.log
Though with this approach "quoted argument2" would become quoted argument2 in the log.
Is there a way (in bash) to realize this without having to duplicate the commands?
You could play with redirections, switch the x option on and off on demand, unset PS4 to get rid of the leading + , and define log_on and log_off functions for easier coding. Something like this:
$ cat script.sh
#!/usr/bin/env bash
function log_on {
exec 3>&1 4>&2
exec &> >( sed -E '/^(set \+x|log_off)$/d' >> file.log )
ps4=$PS4
PS4=
set -x
}
function log_off {
set +x
exec 1>&3 2>&4
PS4=$ps4
}
echo something not logged
log_on
echo something logged
log_off
echo something else not logged
$ rm -f file.log
$ ./script.sh
something not logged
something else not logged
$ cat file.log
echo something logged
something logged
The exec <redirection> commands look a bit cryptic (as most redirections) but they are rather simple:
exec 3>&1 4>&2 makes copies of file descriptors fd1 and fd2 (stdout and stderr by default) to be able to restore these in log_off. After this fd3 and fd4 are copies of fd1 and fd2, respectively. Pick other fd than 3 or 4 if you already use them.
exec &> >( sed ... ) redirect fd1 and fd2 to the standard input of a sed command.
The sed command sed -E '/^(set \+x|log_off)$/d' >> file.log deletes lines containing only set + or log_off and appends its output to file.log. Without this sed command you would always see the two following lines:
log_off
set +x
in your logs, after a group of logged commands.
exec 1>&3 2>&4 restores fd1 and fd2 from their copies in fd3 and fd4.
The rest is straightforward: save PS4 in ps4 such that it can be restored, enable/disable the x option. This should be easy to adapt or extend if needed.
The x option displays the simple commands separately. It breaks pipes, for instance. If you prefer a command log that looks more like the commands you wrote you can replace set -/+x by set -/+v.
IMHO this has already been answered here:
For simplicity the set linux command is what you need.
set -x or set -v

Assigning the output of the "at" command in a bash script

I'm trying to capture the results of the "at" command inside a Bash script. The various ways of capturing command output don't seem to work, but I'm not sure if it's the pipe in the command or something else.
echo $cmd | at $deployat
produces the output
job 42 at 2014-04-03 12:00
And I'm trying to get at the time the job was set for.
However, I expected something like
v=$($cmd | at $deployat)
echo $v
Would work, or
v=$(echo $cmd | at $deployat)
echo $v
Or
v=`$cmd | at $deployat`
echo $v
But all of those leave the script hung, looking like it's waiting for some input.
What is the proper way to do this to end up with a variable like:
2014-04-03 12:00
============================
Edit:
One possible complication is that the $cmd has flags with it:
ls -l
for example.
The expanded command could be something like:
echo ls -l | at noon tomorrow
Solution:
v=$(echo $cmd | at $deployat 2>&1)
echo $v
at prints its output to stderr not stdout. Use 2>&1 to pipe the stderr of at into stdout. Example:
~$ out=$(echo cat Hello | at -v 2014-04-03 2>&1 | head -n 1)
~$ echo $out
Thu Apr 3 01:21:00 2014
With -v it prints the execution time on the first line which is taken by head -n 1.

How to duplicate stdin into file

I have sophisticated bash script that uses "read -p"(stderr output) very often. And now I need to duplicate all script input from terminal into log file.
tee file.log | script.sh
this command does'nt work carefully because ignores output to user.
Example:
#!/bin/sh
echo "start"
read -p "input value: " val
echo $val
echo "finish"
Terminal run:
start
input value: 3
3
finish
Tee run:
# tee file.log | ./script.sh
start
3
3
finish
No idea why you're using tee here. What I suspect is happening is it needs input, so waits for it, then pipes 3 to stdout
-p prompt
Display prompt, without a trailing newline, before attempting
to read any input. The prompt is displayed only if input is coming from a
terminal.
However input isn't sent from tty here so prompt is never printed. Still feels very weird for me to use tee here, but you can just use echo -n instead of the -p flag for read and it should work.
#!/bin/sh
echo "start"
echo -n "input value: "
read val
echo $val
echo "finish"
e.g.
> tee file.log | ./abovescript
start
input value: 3
3
finish
> cat file.log
3
Also not sure how to get tee to terminate properly from in-script here, so you need to press return key at end which of course causes newline.
That said, since it's an extra line each time anyway, seems worse than just be doing echo "$val" >> file.log each time, though a better option would be just to use a function
#!/bin/bash
r() {
read -p "input value: " val
echo "$val" >> file.log
echo "$val"
}
echo "start"
val=$(r)
echo "$val"
echo "finish"

Print on terminal and into file simultaneously?

I have a shell script that greps some data.. I want to print the result into a file, but doing that prevents the result being displayed on the terminal. Is there a way that can both print the result on the screen and also write into a file.
Thanks in advance.
Pipe your output to the tee command.
Example:
[me#home]$ echo hello | tee out.txt
hello
[me#home]$ cat out.txt
hello
Note that the stdout of echo is printed out as well as written to the file specified by thr tee command.
Note you can add the -a flag to tee to append to the output file
[me#home]$ echo hello | tee out.txt
hello
[me#home]$ echo hello again | tee -a out.txt
hello again
[me#home]$ cat out.txt
hello
hello again
Does exactly your thing
http://linux.die.net/man/1/tee

Resources