I have a shell script whose stdout and stderr I want to write to a logfile. I know that this can be achieved via
sh script.sh >> both.log 2>&1
However, I also want to simultaneously write the stderr to a separate file, "error.log". Is this achievable?
You can use tee to duplicate output to two locations. Combine that with some tricky redirections, and...
script.sh 2>&1 >> both.log | tee -a both.log >> error.log
This redirects stderr to stdout, and then stdout to both.log. stderr remains, and is piped to tee, which copies it to both log files.
For this you need to first switch stdout and stderr, which requires an additional file descriptor:
sh script.sh 3>&2 2>&1 1>&3 3>&-
The last operator closes the auxiliary file descriptor.
After that you can use tee to duplicate the error stream (which is now on stdin) and append it to your error log:
sh script.sh 3>&2 2>&1 1>&3 3>&- | tee -a error.log
And after that you can then direct both stdin and stderr to your combined log:
(sh script.sh 3>&2 2>&1 1>&3 3>&- | tee -a error.log) >> both.log 2>&1
The parentheses around the command are important to capture the error stream of the whole command. Without them only the (empty) error stream of the tee command would be captured and the rest would still go to the terminal.
Note: this does not check wheater the file descriptor 3 was in use (open) before. In bash you can use this to choose a previously unused file descriptor and close it on the last redirection:
sh script.sh {tmpfd}>&2 2>&1 1>&${tmpfd}-
I have a question for redirection.
I always use anycommands > /dev/null 2>&1 when I need not any output. But I have never used anycommands 2> /dev/null >&2
Question: Which one is the best way to expect no outputs? What's the difference between anycommands > /dev/null 2>&1 and anycommands 2> /dev/null >&2
case#1:(echo stdout;echo stderr>&2) >/dev/null 2>&1
stdout(1) is replaced by an fd to /dev/null
stderr(2) descriptor is copied from &1 which now is an fd to /dev/null
result: no output at all
case#2:(echo stdout;echo stderr>&2) 2>&1 >/dev/null
stderr(2) descriptor is copied from &1 which is the default stdout
stdout(1) is replaced by an fd to /dev/null
result: stderr is empty, stdout not shown, stderr on stdout
case#3: (echo stdout; echo stderr >&2) 2> /dev/null >&2
same as case#1, stderr and stdout have switched roles
Effectively, the two are equivalent. cmd > /dev/null 2>&1 connects stdout of the command to /dev/null, and then connects stderr to the same file. cmd 2>/dev/null >&2 connects stderr to /dev/null, and then connects stdout to it. The only difference is in the order in which the two streams are associated with /dev/null, which has no bearing on the status of the command when it is run. In both cases, both streams are redirected to the bit bucket.
If you're using only BASH, use &> to redirect both stdout and stderr. That's the most compact, safe and simple solution.
Regarding your question, the first one is equivalent to &> (it redirects both stdout and stderr to /dev/null.
The second connects stderr to /dev/null and redirects stdout to the new stderr, so it's equivalent to as far as the output is concerned. Just the order of file descriptor operations is reversed.
The following writes stdout to a logfile and prints stderr:
bash script.sh >> out.log
This again writes both stdout and stderr to a logfile:
bash script.sh >> out.log 2>&1
How to combine both features, so that stdout and stderr are logged to a file and stderr is emailed to my inbox?
bash script.sh 2>&1 >> out.log | tee -a out.log
First, I'm redirecting stdout to file and stderr to stdout (stdout line gets to the out.log file and stderr to pipe).
The tee command prints stdin to both stdout and file (resemblance with the letter T). Thus second, I'm printing the original stderr to both stdout and the out.log file (the -a argument means append).
You can keep stdout in a separate file and stderr in separate file:
0 * * * * bash script.sh > out.log 2> err.log
and then email yourself err.log file.
Here is a working solution:
./myscript.sh &> all.txt 2> stderr.txt
&> all.txt to have both stderr and stdout
2> stderr.txt to have only stderr
And then just do whatever you want with those files, such as email logging for instance!
Using process substitution you can try:
0 * * * * bash script.sh >> out.log 2> >(exec tee >(exec cat >> mail))
Or
0 * * * * bash -c 'exec bash script.sh >> out.log 2> >(exec tee >(exec cat >> mail))'
exec cat >> mail imitates mailing. Replace it with a command that actually does the mailing.
I have checked couple of relevant posts regarding this in stackoverflow and other sources regarding the usage of 2>&1.
Unfortunately so far have not get my head around it completely.
I understand that 2 is the stderr and 1 is the stdout and we are combining with the 2>&1.
But my question is what is difference between:
1. mycommand > /dev/null
2. mycommand 2> /dev/null
3. mycommand > /dev/null 2>&1
I was thinking:
will redirect stdout and stderr to /dev/null
will redirect stderr to /dev/null
will redirect stdout and stderr to /dev/null
Relevant posts:
What does "/dev/null" mean at the end of shell commands)
i/o stream redirection on linux shell. how does the shell process a command with redirection?
What does “> /dev/null 2>&1″ mean? (http://www.xaprb.com/blog/2006/06/06/what-does-devnull-21-mean/)
See this:
mycommand > /dev/null
it will redirect channel 1 (which is stdout) of mycommand to /dev/null
mycommand 2> /dev/null
it will redirect channel 2 (which is stderr) to /dev/null
mycommand > /dev/null 2>&1
it will redirect channel 1 to /dev/null and then bind channel 2 (stderr) to channel 1 (stdout). Both will go into /dev/null
There is another one (just to complete)
mycommand 2>&1 > /dev/null
In this second case, I bind (the child's) stderr to stdout (of the
parent) and then I find the child's stdout to /dev/null. The result is
that you now get the child's stderr output on stdout and the stdout
goes to the file. This is useful for processing stderr in a pipe, for
example. (see this answer)
(errfile doesn't exist)
$ cat errfile
cat: 0652-050 Cannot open errfile.
$ cat errfile > /tmp/stream.out
cat: 0652-050 Cannot open errfile.
$ cat errfile > /tmp/stream.out 2>&1
$ cat /tmp/stream.out
cat: 0652-050 Cannot open errfile.
($ rm /tmp/stream.out)
$ cat errfile 2>&1 > /tmp/stream.out
cat: 0652-050 Cannot open errfile.
$ cat /tmp/stream.out
$
Order is thus important and 2>&1 1>out is different than 1>out 2>&1 due to stream redirection at shell interpretation. You shoud redirect in "reverse" order. stdout > final than source > stdout
Try these to get the differences:
echo "stderr" > /dev/fd/2 | >/dev/null
stderr
echo "stdout" > /dev/fd/1 | >/dev/null
both commands redirected to /dev/null but in first one we're writing to stderr which prints stderr but in second one it prints nothing
1: redirect STDOUT to /dev/null, you use default file descriptor in this case, e.g. command [default]> filename, the default file descriptor is STDOUT.
2: redirect STDERR to /dev/null
3: redirect STDOUT to /dev/null and redirect STDERROR to STDOUT, which means both STDOUT and STDERROR will be redirected to /dev/null
Hope the tips make you clear.
0, 1, 2...9 are file descriptors in bash. 0 stands for stdin, 1 stands for stdout, 2 stands for stderror. 3~9 is spare for any other temporary usage.
Any file descriptor can be redirected to other file descriptor or file by using operator > or >>(append).
Usage: >
Please reference to http://www.tldp.org/LDP/abs/html/io-redirection.html
I know how to use tee to write the output (standard output) of aaa.sh to bbb.out, while still displaying it in the terminal:
./aaa.sh | tee bbb.out
How would I now also write standard error to a file named ccc.out, while still having it displayed?
I'm assuming you want to still see standard error and standard output on the terminal. You could go for Josh Kelley's answer, but I find keeping a tail around in the background which outputs your log file very hackish and cludgy. Notice how you need to keep an extra file descriptor and do cleanup afterward by killing it and technically should be doing that in a trap '...' EXIT.
There is a better way to do this, and you've already discovered it: tee.
Only, instead of just using it for your standard output, have a tee for standard output and one for standard error. How will you accomplish this? Process substitution and file redirection:
command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)
Let's split it up and explain:
> >(..)
>(...) (process substitution) creates a FIFO and lets tee listen on it. Then, it uses > (file redirection) to redirect the standard output of command to the FIFO that your first tee is listening on.
The same thing for the second:
2> >(tee -a stderr.log >&2)
We use process substitution again to make a tee process that reads from standard input and dumps it into stderr.log. tee outputs its input back on standard output, but since its input is our standard error, we want to redirect tee's standard output to our standard error again. Then we use file redirection to redirect command's standard error to the FIFO's input (tee's standard input).
See Input And Output
Process substitution is one of those really lovely things you get as a bonus of choosing Bash as your shell as opposed to sh (POSIX or Bourne).
In sh, you'd have to do things manually:
out="${TMPDIR:-/tmp}/out.$$" err="${TMPDIR:-/tmp}/err.$$"
mkfifo "$out" "$err"
trap 'rm "$out" "$err"' EXIT
tee -a stdout.log < "$out" &
tee -a stderr.log < "$err" >&2 &
command >"$out" 2>"$err"
Simply:
./aaa.sh 2>&1 | tee -a log
This simply redirects standard error to standard output, so tee echoes both to log and to the screen. Maybe I'm missing something, because some of the other solutions seem really complicated.
Note: Since Bash version 4 you may use |& as an abbreviation for 2>&1 |:
./aaa.sh |& tee -a log
This may be useful for people finding this via Google. Simply uncomment the example you want to try out. Of course, feel free to rename the output files.
#!/bin/bash
STATUSFILE=x.out
LOGFILE=x.log
### All output to screen
### Do nothing, this is the default
### All Output to one file, nothing to the screen
#exec > ${LOGFILE} 2>&1
### All output to one file and all output to the screen
#exec > >(tee ${LOGFILE}) 2>&1
### All output to one file, STDOUT to the screen
#exec > >(tee -a ${LOGFILE}) 2> >(tee -a ${LOGFILE} >/dev/null)
### All output to one file, STDERR to the screen
### Note you need both of these lines for this to work
#exec 3>&1
#exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3)
### STDOUT to STATUSFILE, stderr to LOGFILE, nothing to the screen
#exec > ${STATUSFILE} 2>${LOGFILE}
### STDOUT to STATUSFILE, stderr to LOGFILE and all output to the screen
#exec > >(tee ${STATUSFILE}) 2> >(tee ${LOGFILE} >&2)
### STDOUT to STATUSFILE and screen, STDERR to LOGFILE
#exec > >(tee ${STATUSFILE}) 2>${LOGFILE}
### STDOUT to STATUSFILE, STDERR to LOGFILE and screen
#exec > ${STATUSFILE} 2> >(tee ${LOGFILE} >&2)
echo "This is a test"
ls -l sdgshgswogswghthb_this_file_will_not_exist_so_we_get_output_to_stderr_aronkjegralhfaff
ls -l ${0}
In other words, you want to pipe stdout into one filter (tee bbb.out) and stderr into another filter (tee ccc.out). There is no standard way to pipe anything other than stdout into another command, but you can work around that by juggling file descriptors.
{ { ./aaa.sh | tee bbb.out; } 2>&1 1>&3 | tee ccc.out; } 3>&1 1>&2
See also How to grep standard error stream (stderr)? and When would you use an additional file descriptor?
In bash (and ksh and zsh), but not in other POSIX shells such as dash, you can use process substitution:
./aaa.sh > >(tee bbb.out) 2> >(tee ccc.out)
Beware that in bash, this command returns as soon as ./aaa.sh finishes, even if the tee commands are still executed (ksh and zsh do wait for the subprocesses). This may be a problem if you do something like ./aaa.sh > >(tee bbb.out) 2> >(tee ccc.out); process_logs bbb.out ccc.out. In that case, use file descriptor juggling or ksh/zsh instead.
To redirect standard error to a file, display standard output to the screen, and also save standard output to a file:
./aaa.sh 2>ccc.out | tee ./bbb.out
To display both standard error and standard output to screen and also save both to a file, you can use Bash's I/O redirection:
#!/bin/bash
# Create a new file descriptor 4, pointed at the file
# which will receive standard error.
exec 4<>ccc.out
# Also print the contents of this file to screen.
tail -f ccc.out &
# Run the command; tee standard output as normal, and send standard error
# to our file descriptor 4.
./aaa.sh 2>&4 | tee bbb.out
# Clean up: Close file descriptor 4 and kill tail -f.
exec 4>&-
kill %1
If using Bash:
# Redirect standard out and standard error separately
% cmd >stdout-redirect 2>stderr-redirect
# Redirect standard error and out together
% cmd >stdout-redirect 2>&1
# Merge standard error with standard out and pipe
% cmd 2>&1 |cmd2
Credit (not answering from the top of my head) goes here: Re: bash : stderr & more (pipe for stderr)
If you're using Z shell (zsh), you can use multiple redirections, so you don't even need tee:
./cmd 1>&1 2>&2 1>out_file 2>err_file
Here you're simply redirecting each stream to itself and the target file.
Full example
% (echo "out"; echo "err">/dev/stderr) 1>&1 2>&2 1>/tmp/out_file 2>/tmp/err_file
out
err
% cat /tmp/out_file
out
% cat /tmp/err_file
err
Note that this requires the MULTIOS option to be set (which is the default).
MULTIOS
Perform implicit tees or cats when multiple redirections are attempted (see Redirection).
Like the accepted answer well explained by lhunath, you can use
command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)
Beware than if you use bash you could have some issue.
Let me take the matthew-wilcoxson example.
And for those who "seeing is believing", a quick test:
(echo "Test Out";>&2 echo "Test Err") > >(tee stdout.log) 2> >(tee stderr.log >&2)
Personally, when I try, I have this result:
user#computer:~$ (echo "Test Out";>&2 echo "Test Err") > >(tee stdout.log) 2> >(tee stderr.log >&2)
user#computer:~$ Test Out
Test Err
Both messages do not appear at the same level. Why does Test Out seem to be put like if it is my previous command?
The prompt is on a blank line letting me think the process is not finished, and when I press Enter this fix it.
When I check the content of the files, it is ok, and redirection works.
Let’s take another test.
function outerr() {
echo "out" # stdout
echo >&2 "err" # stderr
}
user#computer:~$ outerr
out
err
user#computer:~$ outerr >/dev/null
err
user#computer:~$ outerr 2>/dev/null
out
Trying again the redirection, but with this function:
function test_redirect() {
fout="stdout.log"
ferr="stderr.log"
echo "$ outerr"
(outerr) > >(tee "$fout") 2> >(tee "$ferr" >&2)
echo "# $fout content: "
cat "$fout"
echo "# $ferr content: "
cat "$ferr"
}
Personally, I have this result:
user#computer:~$ test_redirect
$ outerr
# stdout.log content:
out
out
err
# stderr.log content:
err
user#computer:~$
No prompt on a blank line, but I don't see normal output. The stdout.log content seem to be wrong, and only stderr.log seem to be ok.
If I relaunch it, the output can be different...
So, why?
Because, like explained here:
Beware that in bash, this command returns as soon as [first command] finishes, even if the tee commands are still executed (ksh and zsh do wait for the subprocesses)
So, if you use Bash, prefer use the better example given in this other answer:
{ { outerr | tee "$fout"; } 2>&1 1>&3 | tee "$ferr"; } 3>&1 1>&2
It will fix the previous issues.
Now, the question is, how to retrieve exit status code?
$? does not work.
I have no found better solution than switch on pipefail with set -o pipefail (set +o pipefail to switch off) and use ${PIPESTATUS[0]} like this:
function outerr() {
echo "out"
echo >&2 "err"
return 11
}
function test_outerr() {
local - # To preserve set option
! [[ -o pipefail ]] && set -o pipefail; # Or use second part directly
local fout="stdout.log"
local ferr="stderr.log"
echo "$ outerr"
{ { outerr | tee "$fout"; } 2>&1 1>&3 | tee "$ferr"; } 3>&1 1>&2
# First save the status or it will be lost
local status="${PIPESTATUS[0]}" # Save first, the second is 0, perhaps tee status code.
echo "==="
echo "# $fout content :"
echo "<==="
cat "$fout"
echo "===>"
echo "# $ferr content :"
echo "<==="
cat "$ferr"
echo "===>"
if (( status > 0 )); then
echo "Fail $status > 0"
return "$status" # or whatever
fi
}
user#computer:~$ test_outerr
$ outerr
err
out
===
# stdout.log content:
<===
out
===>
# stderr.log content:
<===
err
===>
Fail 11 > 0
In my case, a script was running command while redirecting both stdout and stderr to a file, something like:
cmd > log 2>&1
I needed to update it such that when there is a failure, take some actions based on the error messages. I could of course remove the dup 2>&1 and capture the stderr from the script, but then the error messages won't go into the log file for reference. While the accepted answer from lhunath is supposed to do the same, it redirects stdout and stderr to different files, which is not what I want, but it helped me to come up with the exact solution that I need:
(cmd 2> >(tee /dev/stderr)) > log
With the above, log will have a copy of both stdout and stderr and I can capture stderr from my script without having to worry about stdout.
The following will work for KornShell (ksh) where the process substitution is not available,
# create a combined (standard input and standard output) collector
exec 3 <> combined.log
# stream standard error instead of standard output to tee, while draining all standard output to the collector
./aaa.sh 2>&1 1>&3 | tee -a stderr.log 1>&3
# cleanup collector
exec 3>&-
The real trick here, is the sequence of the 2>&1 1>&3 which in our case redirects the standard error to standard output and redirects the standard output to file descriptor 3. At this point the standard error and standard output are not combined yet.
In effect, the standard error (as standard input) is passed to tee where it logs to stderr.log and also redirects to file descriptor 3.
And file descriptor 3 is logging it to combined.log all the time. So the combined.log contains both standard output and standard error.
Thanks lhunath for the answer in POSIX.
Here's a more complex situation I needed in POSIX with the proper fix:
# Start script main() function
# - We redirect standard output to file_out AND terminal
# - We redirect standard error to file_err, file_out AND terminal
# - Terminal and file_out have both standard output and standard error, while file_err only holds standard error
main() {
# my main function
}
log_path="/my_temp_dir"
pfout_fifo="${log_path:-/tmp}/pfout_fifo.$$"
pferr_fifo="${log_path:-/tmp}/pferr_fifo.$$"
mkfifo "$pfout_fifo" "$pferr_fifo"
trap 'rm "$pfout_fifo" "$pferr_fifo"' EXIT
tee -a "file_out" < "$pfout_fifo" &
tee -a "file_err" < "$pferr_fifo" >>"$pfout_fifo" &
main "$#" >"$pfout_fifo" 2>"$pferr_fifo"; exit
Compilation errors which are sent to standard error (STDERR) can be redirected or save to a file by:
Bash:
gcc temp.c &> error.log
C shell (csh):
% gcc temp.c |& tee error.log
See: How can I redirect compilation/build error to a file?