main 2>&1 is giving INFO, ERROR and Running status lines.
But when I use sed on it, it is giving INFO and Running status lines along with ERROR lines but they are not in the order where they have to.
Jumbled status are at the end after job executed succefully.
main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)/ s/ERROR/NOTE/" | tee -a $log_file
These are the lines which are missing after I use sed command:
if you append /\(Running\|INFO\ERROR... in sed as below, this can also print Running and info along with ERROR lines
main 2>&1 | sed "/\(Running\|INFO\|ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)\)/ s/ERROR/NOTE/" | tee -a $log_file
Related
When using here script in a linux bash to run commands on a remote server all the lines are printed. How can the comments be suppressed?
The output of the below code should be:
ls
... (whatever is in this folder)
echo -e this is a test\ndone
this is a testndone
exit
Is this possible?
The reason for doing this is that the commands and comments are more complex making the output hard to read. That should be prettier.
#!/bin/bash
ssh -tt hogan#123.123.123.123 <<EOF
# this line get printed
ls
# and this comment also
echo -e this is a test\ndone
# exit ssh
exit
EOF
#end of script
I usually use sed to filter out comments and blank lines. The following will also strip comments that follow a command on the same line:
#!/bin/bash
sed 's/[[:blank:]]*#.*//; /^$/d' <<EOF | ssh -tt hogan#123.123.123.123
# this line get printed
ls
# and this comment also
echo -e "this is a test\ndone"
# exit ssh
exit
EOF
After the command try typing
| grep -v "^[[:space:]]*#"
so for example,
cat temp.txt | grep -v "^[[:space:]]*#"
I think you might be asking for this
grep -v '^[[:space:]]*#' <<EOF | ssh -tt hogan#123.123.123.123
# this line get printed
ls
# and this comment also
echo -e this is a test\ndone
# exit ssh
exit
EOF
I wrote a linux bash script with tee and grep to log and timestamp the actions I take in my various ssh sessions. It works, but the logged lines are mixed together sometimes and are full of control characters. How can I properly escape control and other characters not visible in the original sessions and log each line separately?
I am learning bash and the linux interface, so any other suggestions to improve the script would be extremely welcome!
Here is my script (used as a wrapper for the ssh command):
#! /bin/bash
logfile=~/logs/ssh.log
desc="sshlog ${#}"
tab="\t"
format_line() {
while IFS= read -r line; do
echo -e "$(date +"%Y-%m-%d %H:%M:%S %z")${tab}${desc}${tab}${line}"
done
}
echo "[START]" | format_line >> ${logfile}
# grep is used to filter out command line output while keeping commands
ssh "$#" | tee >(grep -e '\#.*\:.*\$' --color=never --line-buffered | format_line >> ${logfile})
echo "[END]" | format_line >> ${logfile}
And here is a screenshot of the jarbled output in the log file:
A note on the solution: Tiago's answer took care of the nonprinting characters very well. Unfortunately, I just realized that the jumbling is being caused by backspaces and using the up and down keys for command completion. That is, the characters are being piped to grep as soon as they appear, and not line-by-line. I will have to ask about this in another question.
Update: I figured out a way to (almost always) handle up/down completion, backspace completion, and control characters.
You can remove those characters with:
perl -lpe 's/[^[:print:]]//g'
Not filtered:
perl -e 'for($i=0; $i<=255; $i++){print chr($i);}' | cat -A
^#^A^B^C^D^E^F^G^H^I$
^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^?M-^#M-^AM-^BM-^CM-^DM-^EM-^FM-^GM-^HM-^IM-^JM-^KM-^LM-^MM-^NM-^OM-^PM-^QM-^RM-^SM-^TM-^UM-^VM-^WM-^XM-^YM-^ZM-^[M-^\M-^]M-^^M-^_M- M-!M-"M-#M-$M-%M-&M-'M-(M-)M-*M-+M-,M--M-.M-/M-0M-1M-2M-3M-4M-5M-6M-7M-8M-9M-:M-;M-<M-=M->M-?M-#M-AM-BM-CM-DM-EM-FM-GM-HM-IM-JM-KM-LM-MM-NM-OM-PM-QM-RM-SM-TM-UM-VM-WM-XM-YM-ZM-[M-\M-]M-^M-_M-`M-aM-bM-cM-dM-eM-fM-gM-hM-iM-jM-kM-lM-mM-nM-oM-pM-qM-rM-sM-tM-uM-vM-wM-xM-yM-zM-{M-|M-}M-~M-^?
Filtered:
perl -e 'for($i=0; $i<=255; $i++){print chr($i);}' | perl -lpe 's/[^[:print:]]//g' | cat -A
$
!"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~$
Explanation:
I am printing the whole ASCII table with:
perl -e 'for($i=0; $i<=255; $i++){print chr($i);}'
I am identifying non printable chars with:
cat -A
I am filtering non printable chars with:
perl -lpe 's/[^[:print:]]//g'
Edit: It seems to me that you need to remove ANSI color chars:
Example:
perl -MTerm::ANSIColor -e 'print colored("yellow on_magenta","yellow on_magenta"),"\n"'| sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | perl -lpe 's/[^[:print:]]//g'
Adapting to your code:
format_line() {
while IFS= read -r line; do
line=$(sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" <<< "$line")
line=$(perl -lpe 's/[^[:print:]]//g' <<< "$line")
echo -e "$(date +"%Y-%m-%d %H:%M:%S %z")${tab}${desc}${tab}${line}"
done
}
I also edited your grep command:
ssh "$#" | tee >(grep -Po '(?<=\$).*' --color=never --line-buffered | format_line >> ${logfile})
Below the output of my test:
2014-06-26 10:11:10 +0100 sshlog tiago#localhost [START]
2014-06-26 10:11:15 +0100 sshlog tiago#localhost whoami
2014-06-26 10:11:16 +0100 sshlog tiago#localhost exit
2014-06-26 10:11:16 +0100 sshlog tiago#localhost [END]
While writing your own script is a great learning experience, you can also use script to record everything printed on your terminal to a file.
The resulting file will still contains the control characters but there are multiple ways to get rid of them as described in How to clean up output of linux 'script' command.
I want to grep the output of my script - which itself contains call to different binaries...
Since the script has multiple binaries within I can't simply put exec and dump the output in file (it does not copy output from the binaries)...
And to let you know, I am monitoring the script output to determine if the system has got stuck!
Why don't you append instead?
mybin1 | grep '...' >> mylog.txt
mybin2 | grep '...' >> mylog.txt
mybin3 | grep '...' >> mylog.txt
Does this not work?
#!/bin/bash
exec 11>&1 12>&2 > >(exec tee /var/log/somewhere) 2>&1 ## Or add -a option to tee to append.
# call your binaries here
exec >&- 2>&- >&11 2>&12 11>&- 12>&-
Some command pipelines that I expect to write to the terminal or to a file don't write anything until a lot of time has passed.
It looks like the data is not being line-buffered but if I use Ctrl-C to send SIGINT none of the expected output appears. What causes that?
I want to modify and collect each line of vmstat output as it comes without losing data when I finally decide to stop collecting.
Here are some closely related examples:
output to terminal: vmstat -n 1 | sed ''
output to terminal: echo foo | sed '' | tee /dev/null
output to file: vmstat -n 1 > somefile
output to file: echo foo | sed '' > somefile
no output to terminal: vmstat -n 1 | sed '' | tee /dev/null
no output to file: vmstat -n 1 | sed '' > somefile
You can try stdbuf with option --output=L or --output=0 to make the output line buffered or unbuffered respectively
stdbuf --output=L vmstat -n 1 >somefile
I have this bash script whose job is to monitor a log file for the occurrence of a certain line. When located, the script will send out an email warning and then terminate itself. For some reason, it keeps on running. How can I be sure to terminate bash script below:
#!/bin/sh
tail -n 0 -f output.err | grep --line-buffered "Exception" | while read line
do
echo "An exception has been detected!" | mail -s "ALERT" monitor#company.com
exit 0
done
You are opening a subshell in the while read and that subshell is who is exiting, not the proper one.
Try before entering the while loop:
SHELLPID=$$
And then in the loop:
kill $SHELLPID
exit 0
Or change your loop to not use a subshell.
Since the parent script is always going to be in the tail -f which never ends I think you have no other choice than killing it from the inner subshell.
Try something like this:
tail -n 0 -f output.err | grep --line-buffered "Exception" | while read line
do
echo "An exception has been detected!" | mail -s "ALERT" monitor#company.com
kill -term `ps ax | grep tail | grep output.err | awk '{print $1}'`
done
This should work, provided you have only one tail keeping an eye on this particular file.