How to disable stdout or stderr in bash scripts temporarily?
Of course the most common way is to redirect stdout or stderr to /dev/null.
But on some systems /dev/null may be unwritable for normal users.
I am writing some scripts that is aim to be portable, so I do not prefer using /dev/null
Some blogs/posts say that >&- can close stdout, but when I tried echo 123 >&- in a bash terminal, it just failed with the message "bash: echo: write error: Bad file descriptor"
Surely I can do it by redirecting stdout or stderr to a tmp file like this:
some_command > /tmp/null
But what I want is a more "elegant" way
I think perhaps I can achieve this by using pipe like this:
some_command | :
But in this way, it may "pollutes" the exit code of the original command
Here is a possible way to do what you want:
( my_cmd 3>&1 1>&2 2>&3- ) | :
This temporarily send stdout to a new file handle, 3 and redirect stderr to stdout so that the stderr pipes into the command (in this case, :). Then the new file handle is routed back out to stdout. These avoid piping the stdout of my_cmd into :. The - in closes the handle after it's used.
To check the exist status of my_cmd after the above you examine the environment variable $PIPESTATUS[0]. $PIPESTATUS is a bash environment array variable that holds the exit status of each piped command in the last pipe that was done.
I think the really correct answer is to investigate why /dev/null isn't world writable. Having it not so is an off-standard system configuration and may cause system problems. The above work-around is a little messy by comparison.
Based on what I wrote earlier and #nos's comment above, here's an example:
(assuming you have no file called 'zzz' in your current directory, and that '.' is readable)
#!/bin/bash
set -o pipefail
ls . 2>&1 |:
echo $?
ls zzz 2>&1 |:
echo $?
The pipelines succeed and fail silently and maintain the exit code. Note that you can probably still make a pipeline example where this would not produce the desired results. I haven't come up with one in my head yet, but that doesn't mean it's not out there. The best answer, as many have noted already, is to fix the system so that /dev/null is world writable.
EDIT: Changed /bin/sh to /bin/bash, although this probably isn't necessary. But since I haven't tested this against a true Bourne Shell, I decided to err on the side of caution.
EDIT: Another script, showing several different redirections, and using the |& shortcut for 2>&1 |. If you run this, you'll notice that some of the ls failures return a 141 exit status rather than the expected 2. This is a broken pipe exit status, but still represents a failure.
#!/bin/bash
set -o pipefail
# start with commands that should succeed
# redirect everything to ':'
echo "ls . |& :"
ls . |& :
echo $?
# redirect only stdout to ':'
echo "ls . | :"
ls . | :
echo $?
# redirect only stderr to ':'
echo "((ls . 1>&3) |& : ) 3>&1"
((ls . 1>&3) |& : ) 3>&1
echo $?
# now move to failures
# redirect everything to ':'
echo "ls zzz |& :"
ls zzz |& :
echo $?
# redirect only stdout to ':'
echo "ls zzz |:"
ls zzz |:
echo $?
# redirect only stderr to ':'
echo "((ls zzz 1>&3) |& : ) 3>&1"
((ls zzz 1>&3) |& : ) 3>&1
echo $?
I use two subshells when I'm attempting to destroy stdout but keep stderr. You could do it without the outer one. In fact, that might be better. Instead of getting a broken pipe error, you get a 1 exit status.
Related
I want to execute a long running command in Bash, and both capture its exit status, and tee its output.
So I do this:
command | tee out.txt
ST=$?
The problem is that the variable ST captures the exit status of tee and not of command. How can I solve this?
Note that command is long running and redirecting the output to a file to view it later is not a good solution for me.
There is an internal Bash variable called $PIPESTATUS; it’s an array that holds the exit status of each command in your last foreground pipeline of commands.
<command> | tee out.txt ; test ${PIPESTATUS[0]} -eq 0
Or another alternative which also works with other shells (like zsh) would be to enable pipefail:
set -o pipefail
...
The first option does not work with zsh due to a little bit different syntax.
Dumb solution: Connecting them through a named pipe (mkfifo). Then the command can be run second.
mkfifo pipe
tee out.txt < pipe &
command > pipe
echo $?
using bash's set -o pipefail is helpful
pipefail: the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
There's an array that gives you the exit status of each command in a pipe.
$ cat x| sed 's///'
cat: x: No such file or directory
$ echo $?
0
$ cat x| sed 's///'
cat: x: No such file or directory
$ echo ${PIPESTATUS[*]}
1 0
$ touch x
$ cat x| sed 's'
sed: 1: "s": substitute pattern can not be delimited by newline or backslash
$ echo ${PIPESTATUS[*]}
0 1
This solution works without using bash specific features or temporary files. Bonus: in the end the exit status is actually an exit status and not some string in a file.
Situation:
someprog | filter
you want the exit status from someprog and the output from filter.
Here is my solution:
((((someprog; echo $? >&3) | filter >&4) 3>&1) | (read xs; exit $xs)) 4>&1
echo $?
See my answer for the same question on unix.stackexchange.com for a detailed explanation and an alternative without subshells and some caveats.
By combining PIPESTATUS[0] and the result of executing the exit command in a subshell, you can directly access the return value of your initial command:
command | tee ; ( exit ${PIPESTATUS[0]} )
Here's an example:
# the "false" shell built-in command returns 1
false | tee ; ( exit ${PIPESTATUS[0]} )
echo "return value: $?"
will give you:
return value: 1
So I wanted to contribute an answer like lesmana's, but I think mine is perhaps a little simpler and slightly more advantageous pure-Bourne-shell solution:
# You want to pipe command1 through command2:
exec 4>&1
exitstatus=`{ { command1; printf $? 1>&3; } | command2 1>&4; } 3>&1`
# $exitstatus now has command1's exit status.
I think this is best explained from the inside out - command1 will execute and print its regular output on stdout (file descriptor 1), then once it's done, printf will execute and print icommand1's exit code on its stdout, but that stdout is redirected to file descriptor 3.
While command1 is running, its stdout is being piped to command2 (printf's output never makes it to command2 because we send it to file descriptor 3 instead of 1, which is what the pipe reads). Then we redirect command2's output to file descriptor 4, so that it also stays out of file descriptor 1 - because we want file descriptor 1 free for a little bit later, because we will bring the printf output on file descriptor 3 back down into file descriptor 1 - because that's what the command substitution (the backticks), will capture and that's what will get placed into the variable.
The final bit of magic is that first exec 4>&1 we did as a separate command - it opens file descriptor 4 as a copy of the external shell's stdout. Command substitution will capture whatever is written on standard out from the perspective of the commands inside it - but since command2's output is going to file descriptor 4 as far as the command substitution is concerned, the command substitution doesn't capture it - however once it gets "out" of the command substitution it is effectively still going to the script's overall file descriptor 1.
(The exec 4>&1 has to be a separate command because many common shells don't like it when you try to write to a file descriptor inside a command substitution, that is opened in the "external" command that is using the substitution. So this is the simplest portable way to do it.)
You can look at it in a less technical and more playful way, as if the outputs of the commands are leapfrogging each other: command1 pipes to command2, then the printf's output jumps over command 2 so that command2 doesn't catch it, and then command 2's output jumps over and out of the command substitution just as printf lands just in time to get captured by the substitution so that it ends up in the variable, and command2's output goes on its merry way being written to the standard output, just as in a normal pipe.
Also, as I understand it, $? will still contain the return code of the second command in the pipe, because variable assignments, command substitutions, and compound commands are all effectively transparent to the return code of the command inside them, so the return status of command2 should get propagated out - this, and not having to define an additional function, is why I think this might be a somewhat better solution than the one proposed by lesmana.
Per the caveats lesmana mentions, it's possible that command1 will at some point end up using file descriptors 3 or 4, so to be more robust, you would do:
exec 4>&1
exitstatus=`{ { command1 3>&-; printf $? 1>&3; } 4>&- | command2 1>&4; } 3>&1`
exec 4>&-
Note that I use compound commands in my example, but subshells (using ( ) instead of { } will also work, though may perhaps be less efficient.)
Commands inherit file descriptors from the process that launches them, so the entire second line will inherit file descriptor four, and the compound command followed by 3>&1 will inherit the file descriptor three. So the 4>&- makes sure that the inner compound command will not inherit file descriptor four, and the 3>&- will not inherit file descriptor three, so command1 gets a 'cleaner', more standard environment. You could also move the inner 4>&- next to the 3>&-, but I figure why not just limit its scope as much as possible.
I'm not sure how often things use file descriptor three and four directly - I think most of the time programs use syscalls that return not-used-at-the-moment file descriptors, but sometimes code writes to file descriptor 3 directly, I guess (I could imagine a program checking a file descriptor to see if it's open, and using it if it is, or behaving differently accordingly if it's not). So the latter is probably best to keep in mind and use for general-purpose cases.
(command | tee out.txt; exit ${PIPESTATUS[0]})
Unlike #cODAR's answer this returns the original exit code of the first command and not only 0 for success and 127 for failure. But as #Chaoran pointed out you can just call ${PIPESTATUS[0]}. It is important however that all is put into brackets.
In Ubuntu and Debian, you can apt-get install moreutils. This contains a utility called mispipe that returns the exit status of the first command in the pipe.
Outside of bash, you can do:
bash -o pipefail -c "command1 | tee output"
This is useful for example in ninja scripts where the shell is expected to be /bin/sh.
The simplest way to do this in plain bash is to use process substitution instead of a pipeline. There are several differences, but they probably don't matter very much for your use case:
When running a pipeline, bash waits until all processes complete.
Sending Ctrl-C to bash makes it kill all the processes of a pipeline, not just the main one.
The pipefail option and the PIPESTATUS variable are irrelevant to process substitution.
Possibly more
With process substitution, bash just starts the process and forgets about it, it's not even visible in jobs.
Mentioned differences aside, consumer < <(producer) and producer | consumer are essentially equivalent.
If you want to flip which one is the "main" process, you just flip the commands and the direction of the substitution to producer > >(consumer). In your case:
command > >(tee out.txt)
Example:
$ { echo "hello world"; false; } > >(tee out.txt)
hello world
$ echo $?
1
$ cat out.txt
hello world
$ echo "hello world" > >(tee out.txt)
hello world
$ echo $?
0
$ cat out.txt
hello world
As I said, there are differences from the pipe expression. The process may never stop running, unless it is sensitive to the pipe closing. In particular, it may keep writing things to your stdout, which may be confusing.
PIPESTATUS[#] must be copied to an array immediately after the pipe command returns.
Any reads of PIPESTATUS[#] will erase the contents.
Copy it to another array if you plan on checking the status of all pipe commands.
"$?" is the same value as the last element of "${PIPESTATUS[#]}",
and reading it seems to destroy "${PIPESTATUS[#]}", but I haven't absolutely verified this.
declare -a PSA
cmd1 | cmd2 | cmd3
PSA=( "${PIPESTATUS[#]}" )
This will not work if the pipe is in a sub-shell. For a solution to that problem,
see bash pipestatus in backticked command?
Base on #brian-s-wilson 's answer; this bash helper function:
pipestatus() {
local S=("${PIPESTATUS[#]}")
if test -n "$*"
then test "$*" = "${S[*]}"
else ! [[ "${S[#]}" =~ [^0\ ] ]]
fi
}
used thus:
1: get_bad_things must succeed, but it should produce no output; but we want to see output that it does produce
get_bad_things | grep '^'
pipeinfo 0 1 || return
2: all pipeline must succeed
thing | something -q | thingy
pipeinfo || return
Pure shell solution:
% rm -f error.flag; echo hello world \
| (cat || echo "First command failed: $?" >> error.flag) \
| (cat || echo "Second command failed: $?" >> error.flag) \
| (cat || echo "Third command failed: $?" >> error.flag) \
; test -s error.flag && (echo Some command failed: ; cat error.flag)
hello world
And now with the second cat replaced by false:
% rm -f error.flag; echo hello world \
| (cat || echo "First command failed: $?" >> error.flag) \
| (false || echo "Second command failed: $?" >> error.flag) \
| (cat || echo "Third command failed: $?" >> error.flag) \
; test -s error.flag && (echo Some command failed: ; cat error.flag)
Some command failed:
Second command failed: 1
First command failed: 141
Please note the first cat fails as well, because it's stdout gets closed on it. The order of the failed commands in the log is correct in this example, but don't rely on it.
This method allows for capturing stdout and stderr for the individual commands so you can then dump that as well into a log file if an error occurs, or just delete it if no error (like the output of dd).
It may sometimes be simpler and clearer to use an external command, rather than digging into the details of bash. pipeline, from the minimal process scripting language execline, exits with the return code of the second command*, just like a sh pipeline does, but unlike sh, it allows reversing the direction of the pipe, so that we can capture the return code of the producer process (the below is all on the sh command line, but with execline installed):
$ # using the full execline grammar with the execlineb parser:
$ execlineb -c 'pipeline { echo "hello world" } tee out.txt'
hello world
$ cat out.txt
hello world
$ # for these simple examples, one can forego the parser and just use "" as a separator
$ # traditional order
$ pipeline echo "hello world" "" tee out.txt
hello world
$ # "write" order (second command writes rather than reads)
$ pipeline -w tee out.txt "" echo "hello world"
hello world
$ # pipeline execs into the second command, so that's the RC we get
$ pipeline -w tee out.txt "" false; echo $?
1
$ pipeline -w tee out.txt "" true; echo $?
0
$ # output and exit status
$ pipeline -w tee out.txt "" sh -c "echo 'hello world'; exit 42"; echo "RC: $?"
hello world
RC: 42
$ cat out.txt
hello world
Using pipeline has the same differences to native bash pipelines as the bash process substitution used in answer #43972501.
* Actually pipeline doesn't exit at all unless there is an error. It executes into the second command, so it's the second command that does the returning.
Why not use stderr? Like so:
(
# Our long-running process that exits abnormally
( for i in {1..100} ; do echo ploop ; sleep 0.5 ; done ; exit 5 )
echo $? 1>&2 # We pass the exit status of our long-running process to stderr (fd 2).
) | tee ploop.out
So ploop.out receives the stdout. stderr receives the exit status of the long running process. This has the benefit of being completely POSIX-compatible.
(Well, with the exception of the range expression in the example long-running process, but that's not really relevant.)
Here's what this looks like:
...
ploop
ploop
ploop
ploop
ploop
ploop
ploop
ploop
ploop
ploop
5
Note that the return code 5 does not get output to the file ploop.out.
Is there a way for a script to log both, the command line run (including piped ones) plus its output without duplicating the line for the command?
The intention is that the script should have a clean output, but should log verbosely into a log file (so no set -x). Apart from the output, it shall also log the command line causing the output, which could be a piped command-one liner.
The most basic approach is to duplicate the command line in the script and then dump it into the log followed by the captured output of the actual command being run:
echo "command argument1 \"quoted argument2\" | grep -oE \"some output\"" >> file.log
output="$(command argument1 "quoted argument2" 2>&1 | grep -oE "some output")"
echo "${output}" >> file.log
This has the side effect that quoted sections would need to be escaped for the log, which can lead to errors resulting in confusion.
If none of the commands were piped, one could store the command line in an array and then "run" the array.
command=(command argument1 "quoted argument2")
echo "${command[#]}" >> file.log
output="$("${command[#]}" 2>&1)"
echo "${output}" >> file.log
Though with this approach "quoted argument2" would become quoted argument2 in the log.
Is there a way (in bash) to realize this without having to duplicate the commands?
You could play with redirections, switch the x option on and off on demand, unset PS4 to get rid of the leading + , and define log_on and log_off functions for easier coding. Something like this:
$ cat script.sh
#!/usr/bin/env bash
function log_on {
exec 3>&1 4>&2
exec &> >( sed -E '/^(set \+x|log_off)$/d' >> file.log )
ps4=$PS4
PS4=
set -x
}
function log_off {
set +x
exec 1>&3 2>&4
PS4=$ps4
}
echo something not logged
log_on
echo something logged
log_off
echo something else not logged
$ rm -f file.log
$ ./script.sh
something not logged
something else not logged
$ cat file.log
echo something logged
something logged
The exec <redirection> commands look a bit cryptic (as most redirections) but they are rather simple:
exec 3>&1 4>&2 makes copies of file descriptors fd1 and fd2 (stdout and stderr by default) to be able to restore these in log_off. After this fd3 and fd4 are copies of fd1 and fd2, respectively. Pick other fd than 3 or 4 if you already use them.
exec &> >( sed ... ) redirect fd1 and fd2 to the standard input of a sed command.
The sed command sed -E '/^(set \+x|log_off)$/d' >> file.log deletes lines containing only set + or log_off and appends its output to file.log. Without this sed command you would always see the two following lines:
log_off
set +x
in your logs, after a group of logged commands.
exec 1>&3 2>&4 restores fd1 and fd2 from their copies in fd3 and fd4.
The rest is straightforward: save PS4 in ps4 such that it can be restored, enable/disable the x option. This should be easy to adapt or extend if needed.
The x option displays the simple commands separately. It breaks pipes, for instance. If you prefer a command log that looks more like the commands you wrote you can replace set -/+x by set -/+v.
IMHO this has already been answered here:
For simplicity the set linux command is what you need.
set -x or set -v
Could someone help me under stand the condition ls /etc/*release 1>/dev/null 2>&1 that's contained in the code:
if ls /etc/*release 1>/dev/null 2>&1; then
echo "<h2>System release info</h2>"
echo "<pre>"
for i in /etc/*release; do
# Since we can't be sure of the
# length of the file, only
# display the first line.
head -n 1 $i
done
uname -orp
echo "</pre>"
fi
I pretty much don't understand any of that line but specifically what I wanted to know was:
Why dose it not have to use the 'test' syntax i.e. [ expression ]?
The spacing in the condition also confuses, is 1>/dev/null a variable in the ls statement?
what is 2>&1?
I understand the purpose of this statement, which is; if there exists a file with release in it's name under the /etc/ directory the statement will continue, I just don't understand how this achieves this.
Thanks for you help
[ isn't a special character, it's a command (/bin/[ or /usr/bin/[, usually a link to test). That means
if [ ...
if test ...
are the same. For this to work, test ignores ] as last argument if it's being called [.
if simply responds to the exit code of the command it invokes. An exit code of 0 means success or "true".
1>/dev/null 2>&1 redirects stdout (1) to the device /dev/null and then stderr (2) to stdout which means the command can't display and output or errors on the terminal.
Since stdout isn't a normal file or device, you have to use >& for the redirection.
At first glance, one would think that if [ -e /etc/*release ] would be a better solution but test -e doesn't work with patterns.
The test programm just evaluate its arguments and return a code 0 or 1 to tell whether it was true or not.
But you can use any shell commands/function with a if. It will do the then part if the return code ($?) was 0.
So, here, we look if ls return 0 (a file matched), or not.
So, in the end, it's equivalent to write if [ -e /etc/*release ] ; then, which is more "shell-liked".
The last two statements 1>/dev/null and 2>&1 are just here to avoid displaying the output of the ls
1>/dev/null redirect stdout to /dev/null, so the standard out is not shown
2>&1 redirect stderr to stdout. Here, stdout is redirected to /dev/null, so everything is redirected to /dev/null
I'm writing a bash script that is supposed to be "transparent" to the user. It reads commands from the user and intercepts them, allowing only some of them to be executed by bash, depending on some criteria. It (basically) works like this:
while true; do
read COMMAND
can_be_done $COMMAND
if [ $? == 0 ]; then
eval $COMMAND
if [ $? != 0 ]; then
echo "Error: command not found"
fi
fi
done
The problem is, when the command fails, you also get stuff printed to the console. BUT, if I keep the result in a variable and only print it when it doesn't fail, like so:
RESULT=$(eval $COMMAND)
Then there's another problem: The special formatting gets lost (for example, "ls --color" doesn't show colors anymore)
My question is: Is there a way to have the command print to STDOUT if successful, but to /dev/null if it fails?
Do you really need the second part, replacing the output of the command with an error message? Linux commands print their own error messages, which aren't necessarily "command not found". You'd be hiding the true error (permission denied, file not found, out of memory, segfault, etc.) with an oftentimes incorrect error message (command not found).
If you remove that check, you could simplify the loop to something like this:
while true; do
read -e COMMAND
if can_be_done "$COMMAND"; then
eval "$COMMAND"
fi
done
read -e uses readline to obtain the command, making the prompt a lot more shell-like (↑ and ↓ for history, for instance).
command; if [ $? == 0 ]; then is more idiomatically written as if <command>; then.
Quoting makes sure special characters and whitespace are handled properly.
I would argue strongly that you should not do this. If you do not want to see output, redirect it to /dev/null. If you do want to see errors, do not redirect stderr. If you are using a program that prints its error messages on stdout instead of stderr, FIX THE PROGRAM! Error messages belong on stderr. Note that this means your program is broken, as it ought to read:
echo "Error: command not found" >&2
I'm not sure if it is rule number 1, but it certainly belongs in the top 10, and it may be the most often violated rule: Error messages belong on stderr. A program which prints error messages on stdout is broken.
if false > /dev/null;then echo 1; else echo 2; fi 2> /dev/null
Will output 2
if true > /dev/null;then echo 1; else echo 2; fi 2> /dev/null
Will output 1
remove the > /dev/null to print the command also to stdout
for example
if echo 123;then echo 1; else echo 2; fi 2> /dev/null
Will output
123 & 1
Assuming that the command is not very expensive to run you can do this:
test `ls /mooo 2>/dev/null` || echo moo not found
test will return true only if the command exits with 0, in this case ls is the command. You could have put this in an if statement too like so:
if [ `ls /moo 2>/dev/null` ];then
echo moo is a folder
fi
I know how to use tee to write the output (standard output) of aaa.sh to bbb.out, while still displaying it in the terminal:
./aaa.sh | tee bbb.out
How would I now also write standard error to a file named ccc.out, while still having it displayed?
I'm assuming you want to still see standard error and standard output on the terminal. You could go for Josh Kelley's answer, but I find keeping a tail around in the background which outputs your log file very hackish and cludgy. Notice how you need to keep an extra file descriptor and do cleanup afterward by killing it and technically should be doing that in a trap '...' EXIT.
There is a better way to do this, and you've already discovered it: tee.
Only, instead of just using it for your standard output, have a tee for standard output and one for standard error. How will you accomplish this? Process substitution and file redirection:
command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)
Let's split it up and explain:
> >(..)
>(...) (process substitution) creates a FIFO and lets tee listen on it. Then, it uses > (file redirection) to redirect the standard output of command to the FIFO that your first tee is listening on.
The same thing for the second:
2> >(tee -a stderr.log >&2)
We use process substitution again to make a tee process that reads from standard input and dumps it into stderr.log. tee outputs its input back on standard output, but since its input is our standard error, we want to redirect tee's standard output to our standard error again. Then we use file redirection to redirect command's standard error to the FIFO's input (tee's standard input).
See Input And Output
Process substitution is one of those really lovely things you get as a bonus of choosing Bash as your shell as opposed to sh (POSIX or Bourne).
In sh, you'd have to do things manually:
out="${TMPDIR:-/tmp}/out.$$" err="${TMPDIR:-/tmp}/err.$$"
mkfifo "$out" "$err"
trap 'rm "$out" "$err"' EXIT
tee -a stdout.log < "$out" &
tee -a stderr.log < "$err" >&2 &
command >"$out" 2>"$err"
Simply:
./aaa.sh 2>&1 | tee -a log
This simply redirects standard error to standard output, so tee echoes both to log and to the screen. Maybe I'm missing something, because some of the other solutions seem really complicated.
Note: Since Bash version 4 you may use |& as an abbreviation for 2>&1 |:
./aaa.sh |& tee -a log
This may be useful for people finding this via Google. Simply uncomment the example you want to try out. Of course, feel free to rename the output files.
#!/bin/bash
STATUSFILE=x.out
LOGFILE=x.log
### All output to screen
### Do nothing, this is the default
### All Output to one file, nothing to the screen
#exec > ${LOGFILE} 2>&1
### All output to one file and all output to the screen
#exec > >(tee ${LOGFILE}) 2>&1
### All output to one file, STDOUT to the screen
#exec > >(tee -a ${LOGFILE}) 2> >(tee -a ${LOGFILE} >/dev/null)
### All output to one file, STDERR to the screen
### Note you need both of these lines for this to work
#exec 3>&1
#exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3)
### STDOUT to STATUSFILE, stderr to LOGFILE, nothing to the screen
#exec > ${STATUSFILE} 2>${LOGFILE}
### STDOUT to STATUSFILE, stderr to LOGFILE and all output to the screen
#exec > >(tee ${STATUSFILE}) 2> >(tee ${LOGFILE} >&2)
### STDOUT to STATUSFILE and screen, STDERR to LOGFILE
#exec > >(tee ${STATUSFILE}) 2>${LOGFILE}
### STDOUT to STATUSFILE, STDERR to LOGFILE and screen
#exec > ${STATUSFILE} 2> >(tee ${LOGFILE} >&2)
echo "This is a test"
ls -l sdgshgswogswghthb_this_file_will_not_exist_so_we_get_output_to_stderr_aronkjegralhfaff
ls -l ${0}
In other words, you want to pipe stdout into one filter (tee bbb.out) and stderr into another filter (tee ccc.out). There is no standard way to pipe anything other than stdout into another command, but you can work around that by juggling file descriptors.
{ { ./aaa.sh | tee bbb.out; } 2>&1 1>&3 | tee ccc.out; } 3>&1 1>&2
See also How to grep standard error stream (stderr)? and When would you use an additional file descriptor?
In bash (and ksh and zsh), but not in other POSIX shells such as dash, you can use process substitution:
./aaa.sh > >(tee bbb.out) 2> >(tee ccc.out)
Beware that in bash, this command returns as soon as ./aaa.sh finishes, even if the tee commands are still executed (ksh and zsh do wait for the subprocesses). This may be a problem if you do something like ./aaa.sh > >(tee bbb.out) 2> >(tee ccc.out); process_logs bbb.out ccc.out. In that case, use file descriptor juggling or ksh/zsh instead.
To redirect standard error to a file, display standard output to the screen, and also save standard output to a file:
./aaa.sh 2>ccc.out | tee ./bbb.out
To display both standard error and standard output to screen and also save both to a file, you can use Bash's I/O redirection:
#!/bin/bash
# Create a new file descriptor 4, pointed at the file
# which will receive standard error.
exec 4<>ccc.out
# Also print the contents of this file to screen.
tail -f ccc.out &
# Run the command; tee standard output as normal, and send standard error
# to our file descriptor 4.
./aaa.sh 2>&4 | tee bbb.out
# Clean up: Close file descriptor 4 and kill tail -f.
exec 4>&-
kill %1
If using Bash:
# Redirect standard out and standard error separately
% cmd >stdout-redirect 2>stderr-redirect
# Redirect standard error and out together
% cmd >stdout-redirect 2>&1
# Merge standard error with standard out and pipe
% cmd 2>&1 |cmd2
Credit (not answering from the top of my head) goes here: Re: bash : stderr & more (pipe for stderr)
If you're using Z shell (zsh), you can use multiple redirections, so you don't even need tee:
./cmd 1>&1 2>&2 1>out_file 2>err_file
Here you're simply redirecting each stream to itself and the target file.
Full example
% (echo "out"; echo "err">/dev/stderr) 1>&1 2>&2 1>/tmp/out_file 2>/tmp/err_file
out
err
% cat /tmp/out_file
out
% cat /tmp/err_file
err
Note that this requires the MULTIOS option to be set (which is the default).
MULTIOS
Perform implicit tees or cats when multiple redirections are attempted (see Redirection).
Like the accepted answer well explained by lhunath, you can use
command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)
Beware than if you use bash you could have some issue.
Let me take the matthew-wilcoxson example.
And for those who "seeing is believing", a quick test:
(echo "Test Out";>&2 echo "Test Err") > >(tee stdout.log) 2> >(tee stderr.log >&2)
Personally, when I try, I have this result:
user#computer:~$ (echo "Test Out";>&2 echo "Test Err") > >(tee stdout.log) 2> >(tee stderr.log >&2)
user#computer:~$ Test Out
Test Err
Both messages do not appear at the same level. Why does Test Out seem to be put like if it is my previous command?
The prompt is on a blank line letting me think the process is not finished, and when I press Enter this fix it.
When I check the content of the files, it is ok, and redirection works.
Let’s take another test.
function outerr() {
echo "out" # stdout
echo >&2 "err" # stderr
}
user#computer:~$ outerr
out
err
user#computer:~$ outerr >/dev/null
err
user#computer:~$ outerr 2>/dev/null
out
Trying again the redirection, but with this function:
function test_redirect() {
fout="stdout.log"
ferr="stderr.log"
echo "$ outerr"
(outerr) > >(tee "$fout") 2> >(tee "$ferr" >&2)
echo "# $fout content: "
cat "$fout"
echo "# $ferr content: "
cat "$ferr"
}
Personally, I have this result:
user#computer:~$ test_redirect
$ outerr
# stdout.log content:
out
out
err
# stderr.log content:
err
user#computer:~$
No prompt on a blank line, but I don't see normal output. The stdout.log content seem to be wrong, and only stderr.log seem to be ok.
If I relaunch it, the output can be different...
So, why?
Because, like explained here:
Beware that in bash, this command returns as soon as [first command] finishes, even if the tee commands are still executed (ksh and zsh do wait for the subprocesses)
So, if you use Bash, prefer use the better example given in this other answer:
{ { outerr | tee "$fout"; } 2>&1 1>&3 | tee "$ferr"; } 3>&1 1>&2
It will fix the previous issues.
Now, the question is, how to retrieve exit status code?
$? does not work.
I have no found better solution than switch on pipefail with set -o pipefail (set +o pipefail to switch off) and use ${PIPESTATUS[0]} like this:
function outerr() {
echo "out"
echo >&2 "err"
return 11
}
function test_outerr() {
local - # To preserve set option
! [[ -o pipefail ]] && set -o pipefail; # Or use second part directly
local fout="stdout.log"
local ferr="stderr.log"
echo "$ outerr"
{ { outerr | tee "$fout"; } 2>&1 1>&3 | tee "$ferr"; } 3>&1 1>&2
# First save the status or it will be lost
local status="${PIPESTATUS[0]}" # Save first, the second is 0, perhaps tee status code.
echo "==="
echo "# $fout content :"
echo "<==="
cat "$fout"
echo "===>"
echo "# $ferr content :"
echo "<==="
cat "$ferr"
echo "===>"
if (( status > 0 )); then
echo "Fail $status > 0"
return "$status" # or whatever
fi
}
user#computer:~$ test_outerr
$ outerr
err
out
===
# stdout.log content:
<===
out
===>
# stderr.log content:
<===
err
===>
Fail 11 > 0
In my case, a script was running command while redirecting both stdout and stderr to a file, something like:
cmd > log 2>&1
I needed to update it such that when there is a failure, take some actions based on the error messages. I could of course remove the dup 2>&1 and capture the stderr from the script, but then the error messages won't go into the log file for reference. While the accepted answer from lhunath is supposed to do the same, it redirects stdout and stderr to different files, which is not what I want, but it helped me to come up with the exact solution that I need:
(cmd 2> >(tee /dev/stderr)) > log
With the above, log will have a copy of both stdout and stderr and I can capture stderr from my script without having to worry about stdout.
The following will work for KornShell (ksh) where the process substitution is not available,
# create a combined (standard input and standard output) collector
exec 3 <> combined.log
# stream standard error instead of standard output to tee, while draining all standard output to the collector
./aaa.sh 2>&1 1>&3 | tee -a stderr.log 1>&3
# cleanup collector
exec 3>&-
The real trick here, is the sequence of the 2>&1 1>&3 which in our case redirects the standard error to standard output and redirects the standard output to file descriptor 3. At this point the standard error and standard output are not combined yet.
In effect, the standard error (as standard input) is passed to tee where it logs to stderr.log and also redirects to file descriptor 3.
And file descriptor 3 is logging it to combined.log all the time. So the combined.log contains both standard output and standard error.
Thanks lhunath for the answer in POSIX.
Here's a more complex situation I needed in POSIX with the proper fix:
# Start script main() function
# - We redirect standard output to file_out AND terminal
# - We redirect standard error to file_err, file_out AND terminal
# - Terminal and file_out have both standard output and standard error, while file_err only holds standard error
main() {
# my main function
}
log_path="/my_temp_dir"
pfout_fifo="${log_path:-/tmp}/pfout_fifo.$$"
pferr_fifo="${log_path:-/tmp}/pferr_fifo.$$"
mkfifo "$pfout_fifo" "$pferr_fifo"
trap 'rm "$pfout_fifo" "$pferr_fifo"' EXIT
tee -a "file_out" < "$pfout_fifo" &
tee -a "file_err" < "$pferr_fifo" >>"$pfout_fifo" &
main "$#" >"$pfout_fifo" 2>"$pferr_fifo"; exit
Compilation errors which are sent to standard error (STDERR) can be redirected or save to a file by:
Bash:
gcc temp.c &> error.log
C shell (csh):
% gcc temp.c |& tee error.log
See: How can I redirect compilation/build error to a file?