I want to grep the output of my script - which itself contains call to different binaries...
Since the script has multiple binaries within I can't simply put exec and dump the output in file (it does not copy output from the binaries)...
And to let you know, I am monitoring the script output to determine if the system has got stuck!
Why don't you append instead?
mybin1 | grep '...' >> mylog.txt
mybin2 | grep '...' >> mylog.txt
mybin3 | grep '...' >> mylog.txt
Does this not work?
#!/bin/bash
exec 11>&1 12>&2 > >(exec tee /var/log/somewhere) 2>&1 ## Or add -a option to tee to append.
# call your binaries here
exec >&- 2>&- >&11 2>&12 11>&- 12>&-
Related
Is there a way for a script to log both, the command line run (including piped ones) plus its output without duplicating the line for the command?
The intention is that the script should have a clean output, but should log verbosely into a log file (so no set -x). Apart from the output, it shall also log the command line causing the output, which could be a piped command-one liner.
The most basic approach is to duplicate the command line in the script and then dump it into the log followed by the captured output of the actual command being run:
echo "command argument1 \"quoted argument2\" | grep -oE \"some output\"" >> file.log
output="$(command argument1 "quoted argument2" 2>&1 | grep -oE "some output")"
echo "${output}" >> file.log
This has the side effect that quoted sections would need to be escaped for the log, which can lead to errors resulting in confusion.
If none of the commands were piped, one could store the command line in an array and then "run" the array.
command=(command argument1 "quoted argument2")
echo "${command[#]}" >> file.log
output="$("${command[#]}" 2>&1)"
echo "${output}" >> file.log
Though with this approach "quoted argument2" would become quoted argument2 in the log.
Is there a way (in bash) to realize this without having to duplicate the commands?
You could play with redirections, switch the x option on and off on demand, unset PS4 to get rid of the leading + , and define log_on and log_off functions for easier coding. Something like this:
$ cat script.sh
#!/usr/bin/env bash
function log_on {
exec 3>&1 4>&2
exec &> >( sed -E '/^(set \+x|log_off)$/d' >> file.log )
ps4=$PS4
PS4=
set -x
}
function log_off {
set +x
exec 1>&3 2>&4
PS4=$ps4
}
echo something not logged
log_on
echo something logged
log_off
echo something else not logged
$ rm -f file.log
$ ./script.sh
something not logged
something else not logged
$ cat file.log
echo something logged
something logged
The exec <redirection> commands look a bit cryptic (as most redirections) but they are rather simple:
exec 3>&1 4>&2 makes copies of file descriptors fd1 and fd2 (stdout and stderr by default) to be able to restore these in log_off. After this fd3 and fd4 are copies of fd1 and fd2, respectively. Pick other fd than 3 or 4 if you already use them.
exec &> >( sed ... ) redirect fd1 and fd2 to the standard input of a sed command.
The sed command sed -E '/^(set \+x|log_off)$/d' >> file.log deletes lines containing only set + or log_off and appends its output to file.log. Without this sed command you would always see the two following lines:
log_off
set +x
in your logs, after a group of logged commands.
exec 1>&3 2>&4 restores fd1 and fd2 from their copies in fd3 and fd4.
The rest is straightforward: save PS4 in ps4 such that it can be restored, enable/disable the x option. This should be easy to adapt or extend if needed.
The x option displays the simple commands separately. It breaks pipes, for instance. If you prefer a command log that looks more like the commands you wrote you can replace set -/+x by set -/+v.
IMHO this has already been answered here:
For simplicity the set linux command is what you need.
set -x or set -v
I have been trying to get a bash script to output different things on the terminal and logfile but am unsure of what command to use.
For example,
#!/bin/bash
freespace=$(df -h / | grep -E "/" | awk '{print $4}')
greentext="\033[32m"
bold="\033[1m"
normal="\033[0m"
logdate=$(date +"%Y%m%d")
logfile="$logdate"_report.log
exec > >(tee -i $logfile)
echo -e $bold"Quick system report for "$greentext"$HOSTNAME"$normal
printf "\tSystem type:\t%s\n" $MACHTYPE
printf "\tBash Version:\t%s\n" $BASH_VERSION
printf "\tFree Space:\t%s\n" $freespace
printf "\tFiles in dir:\t%s\n" $(ls | wc -l)
printf "\tGenerated on:\t%s\n" $(date +"%m/%d/%y") # US date format
echo -e $greentext"A summary of this info has been saved to $logfile"$normal
I want to omit the last output (echo "A summary...") in the logfile while displaying it in the terminal. Is there a command to do so? It would be great if a general solution can be provided instead of a specific one because I want to apply this to other scripts.
EDIT 1 (after applying >&6):
Files in dir: 7
A summary of this info has been saved to 20160915_report.log
Generated on: 09/15/16
One option:
exec 6>&1 # save the existing stdout
exec > >(tee -i $logfile) # like you had it
#... all your outputs
echo -e $greentext"A summary of this info has been saved to $logfile"$normal >&6
# writes to the original stdout, saved in file descriptor 6 ------------^^^
The >&6 sends echo's output to the saved file descriptor 6 (the terminal, if you're running this from an interactive shell) rather than to the output path set up by tee (which is on file descriptor 1). Tested on bash 4.3.46.
References: "Using exec" and "I/O Redirection"
Edit As OP found, the >&6 message is not guaranteed to appear after the lines printed by tee off stdout. One option is to use script, e.g., as in the answers to this question, instead of tee, and then print the final message outside of the script. Per the docs, the stdbuf answers to that question won't work with tee.
Try a dirty hack:
#... all your outputs
echo >&6 # <-- New line
echo -e $greentext ... >&6
Or, equally hackish, (Note that, per OP, this worked)
#... all your outputs
sleep 0.25s # or whatever time you want <-- New line
echo -e ... >&6
I'm using "strace -f -T -tt -o foo.txt -p 1234" to print the time spent in system calls. This makes the output file huge, is there a way to just print the system calls that took greater than 1second. I can grep it out from the file later, but is there a better way?
If we simply omit the -o foo.txt argument, the output goes to standard output. We can pipe it through grep and redirect to the file:
strace -f -T -tt -p 1234 | grep pattern > foo.txt
To watch the output at the same time:
strace -f -T -tt -p 1234 | grep pattern | tee foo.txt
If a command prints only to a file that is passed as an argument, and we want to filter/redirect its output, the first step is to check whether it implements the dash convention: can you specify standard input or output using - as a filename argument:
some_command - | our_pipe > file.txt
If not, then the recourse is to use Bash process substitution substitution syntax: >(output command) and <(input command):
some_command >(our_pipe > file.txt)
The process substitution syntax expands into a token that is suitable as a filename argument for a command or function. When the program opens that token, it gets a file descriptor to the command's input or output, depending on direction.
With process substitution, we can redirect the input or output of stubborn programs which work only with files passed as by name as arguments, and which do not support any convention for requesting that standard input or output be used in place of a file.
The token used by process substitution is platform-dependent; we can see what it is using echo. For instance on GNU/Linux, Bash takes advantage of the /dev/fd operating system feature:
$ echo <(ls -l)
/dev/fd/63
You can use the following command:
strace -T command 2>&1 >/dev/null | awk '{gsub(/[<>]/,"",$NF)}$NF+.0>1.0'
Explanation:
strace -T adds the time spent in the syscall end the end of the line, enclosed in <...>
2>&1 >/dev/null | awk pipes stderr to awk. (strace writes it's output to stderr!)
The awk command removes the <> from the last field $NF and prints lines where the time spent is higher than a second.
Probably you'll also want to pass the threshold as a variable to the awk command:
strace -T command 2>&1 >/dev/null \
| awk -v thres=0.001 '{gsub(/[<>]/,"",$NF)}$NF+.0>thres+.0'
New to bash programming. I am not sure what is meant by 'output to stdout'. Does it mean print out to the command line?
If I have a simple bash script:
#!/bin/bash
wget -q http://192.168.0.1/test -O - | grep -m 1 'Hello'
it outputs a string to the terminal. Does this mean it's 'outputting to stdout' ?
Thanks
Yes, stdout is the terminal (unless it's redirected to a file using the > operator or into the stdin of another process using |)
In your specific example, you're actually redirecting using | grep ... through grep then to the terminal.
Every process on a Linux system (and most others) has at least 3 open file descriptors:
stdin (0)
stdout (1)
stderr (2)
Regualary every of this file descriptors will point to the terminal from where the process was started. Like this:
cat file.txt # all file descriptors are pointing to the terminal where you type the command
However, bash allows to modify this behaviour using input / output redirection:
cat < file.txt # will use file.txt as stdin
cat file.txt > output.txt # redirects stdout to a file (will not appear on terminal anymore)
cat file.txt 2> /dev/null # redirects stderr to /dev/null (will not appear on terminal anymore
The same is happening when you are using the pipe symbol like:
wget -q http://192.168.0.1/test -O - | grep -m 1 'Hello'
What is actually happening is that the stdout of the wget process (the process before the | ) is redirected to the stdin of the grep process. So wget's stdout isn't a terminal anymore while grep's output is the current terminal. If you want to redirect grep's output to a file for example, then use this:
wget -q http://192.168.0.1/test -O - | grep -m 1 'Hello' > output.txt
Unless redirected, standard output is the text terminal which initiated the program.
Here's a wikipedia article: http://en.wikipedia.org/wiki/Standard_streams#Standard_output_.28stdout.29
I'm making a script and every time something is done I would like to write into my custom .log file. How do I do that?
And in the end.. I'd just like to read it with Bash,.. do I just use cat?
Thanks.
The simplest syntax I always use is 2>&1 | tee -a file_name.log.
The syntax can be used after a command or execution of a file. e.g.
find . -type f 2>&1 | tee -a file_name.log
or
./test.sh 2>&1 | tee -a file_name.log
Just cat <log message here> >> custom.log.
The >> means add on to the bottom of the file rather than > which would delete the contents of the file and then write the message.