I'm wanting to use the grep in a bash script to find matching lines in a file, highlight the matches with color, and then print out the results in a table using the column command. Something like this:
data=`cat file.data | egrep -i --color "$search"`
echo $'\n'"col1"$'\t'"col2"$'\t'"col3"$'\t'"col4"$'\n'"$data" | column -t -s$'\t'
The above code does everything as desired, except that the color is lost.
Here's a simplified example:
As you can see, when I used grep the results were printed on individual lines and in color, but when I save the results to a variable and then print the variable out, the line breaks and colors are gone.
Is there any way to do what I'm asking?
Use the option --color=always:
data=$(egrep -i --color=always "$search" file.data)
By default, grep does not produce color unless the output is going directly to a terminal. This is normally a good thing. The option --color=always overrides that.
For occasions when you don't want color, use --color=never.
Related
I need to highlight certain keywords like "fail, failed, error, fatal, missing" over my terminal.
I need this with the output of ALL the commands, not any specific command. I assume I need to tweak my bashrc file for this.
To color I can use:
<input coming to terminal>|grep -P --color=auto 'fail|failed|error|fatal|missing|$'
I tried the following command but not helped:
tail -f $(tty) |grep -P --color=auto 'fail|failed|error|fatal|missing|$' &
[1]+ Stopped(SIGTTIN) tail -f $(tty) | grep -P --color=auto 'fail|failed|error|fatal|missing|$'
Searched SO for answers but could not find any question which provides desired an answer.
I don't think there's really an elegant way to do this using the shell. Ideally, you'd get a terminal emulator with this kind a keyword highlighting built in. You can get some of the way by piping the output of bash through a filter that adds ANSI colour escapes. Here is a sed script, that replaces "fail" with (red)fail(normal):
s/fail/\x1B[31m&\x1B[0m/
t done
:done
Run bash with its output piped through sed like this:
$bash | sed -f color.sed
This mechanism is not without problems, but it works in some cases. Usually it's better just to collect up the output you want, and then pipe it through sed, rather than working directly with the bash output.
I see that when I use rsync with the -v option it prints the changed files list and some useful infos at the end, like the total transfer size.
Is it somewhat possible to cut out the first (long) part and just print the stats? I am using it in a script, and the log shouldn't be so long. Only the stats are useful.
Thank you.
As I was looking for an answer and came across this question:
rsync also supports the --stats option.
Best solution for now i think :
rsync --info=progress0,name0,flist0,stats2 ...
progress0 hides progress
progress2 display progress
name0 hides file names
stats2 displays stats at the end of transfer
This solution is more a "hack" than the right way to do it because the output is generated but only filtered afterwards. You can use the option --out-format.
rsync ... --out-format="" ... | grep -v -E "^sending|^created" | tr -s "\n"
The grep filter should probably be updated with unwanted lines you see in the output. The tr is here to filter the long sequence of carriage returns.
grep -E for extended regexes
grep -v to invert the match. "Selected lines are those not matching any of the specified patterns."
tr -s to squeeze the repeated carriage returns into a single one
I've been trying to make tail a little more readable for server startups. My current command filters out most of the INFO and DEBUG messages from the startup:
tail -F ../server/durango/log/server.log | grep -e "ERROR" -e "WARN" -e "Shutdown" -e "MicroKernel" | grep --color=auto -E 'MicroKernel|$'
What I would like to do is craft something that would highlight WARN in yellow and ERROR in red, and MicroKernel in green. I tried just piping grep --color=auto multiple times, but the only color that survives is the last command in the pipe.
Is there a one liner to do this? Or even a many-liner?
yes, there is way to do this. That is, as long as your terminal supports ANSI escape sequences. This is most terminals that exist.
I think I don't need explain how to grep, sed etc. point is the color right?
see below, this will make
WARN yellow
ERROR red
foo green
here is example:
kent$ echo "WARN
ERROR
foo"|sed 's#WARN#\x1b[33m&#; s#ERROR#\x1b[31m&#; s#foo#\x1b[32m&#'
Note: \x1b is hexadecimal for the ESC character (^VEsc).
to see the result:
I wrote a script for this years ago. You can easily cover the case of multiple colors by piping successive invocations of highlight to each other.
From the README:
Usage: ./highlight [-i] [--color=COLOR_STRING] [--] <PATTERN0> [PATTERN1...]
This is highlight version 1.0.
This program takes text via standard input and outputs it with the given
perlre(1) pattern(s) highlighted with the given color. If no color option
is specified, it defaults to 'bold red'. Colors may be anything
that Perl's Term::ANSIColor understands. This program is similar to
"grep --color PATTERN" except both matching and non-matching lines are
printed.
The default color can be selected via the $HIGHLIGHT_COLOR environment
variable. The command-line option takes precedence.
Passing -i or --ignore-case will enable case-insensitive matching.
If your pattern begins with a dash ('-'), you can pass a '--' argument
after any options and before your pattern to distinguish it from an
option.
I have been using a tool called grc for this for years. works like a charm. It comes with some quite good templates for many standard log outputs and formats and it is easy to define your own.
A command I use often is
grc tail -f /var/log/syslog
It colorizes the syslog output so it is easy to spot errors (typically marked red.
Find the tool here:
https://github.com/garabik/grc
(it is also available as package for most common linux flavours).
I wrote TxtStyle, a small utility for colorising logs. You define regular expressions to highlight in ~/.txts.conf file:
[Style="example"]
!red: regex("error")
green: regex("\d{4}-\d\d-\d\d")
# ...
And then apply the styles:
txts -n example example.log
or you can also pipe the output
tail -f example.log | txts -n example
You can create a colored log instead of using a complex command.
For php is like this:
echo "^[[30;43m".$ip."^[[0m";
The key point is to use Ctrl-v ctrl-[ to input a green ^[ under insert mode in vim, direct input ^[ does not work.
More info here
My sample using awk. Match log format like: xxxx [debug] xxxxx xxxx xxxx
black=30m
red=31m
green=32m
yellow=33m
blue=34m
magenta=35m
cyan=36m
white=37m
blacklog="\"\033[$black\" \$0 \"\033[39m\""
redlog="\"\033[$red\" \$0 \"\033[39m\""
greenlog="\"\033[$green\" \$0 \"\033[39m\""
yellowlog="\"\033[$yellow\" \$0 \"\033[39m\""
bluelog="\"\033[$blue\" \$0 \"\033[39m\""
magentalog="\"\033[$magenta\" \$0 \"\033[39m\""
cyanlog="\"\033[$cyan\" \$0 \"\033[39m\""
whitelog="\"\033[$white\" \$0 \"\033[39m\""
trace="/\[trace\]/ {print $redlog}"
debug="/\[debug\]/ {print $magentalog}"
info="/\[info\]/ {print $greenlog}"
warning="/\[warning\]/ {print $bluelog}"
error="/\[error\]/ {print $yellowlog}"
yourcommand | awk "$trace $debug $info $warning $error"
This question already has answers here:
Colorized grep -- viewing the entire file with highlighted matches
(24 answers)
Closed 7 years ago.
When using grep, it will highlight any text in a line with a match to your regular expression.
What if I want this behaviour, but have grep print out all lines as well? I came up empty after a quick look through the grep man page.
Use ack. Checkout its --passthru option here: ack. It has the added benefit of allowing full perl regular expressions.
$ ack --passthru 'pattern1' file_name
$ command_here | ack --passthru 'pattern1'
You can also do it using grep like this:
$ grep --color -E '^|pattern1|pattern2' file_name
$ command_here | grep --color -E '^|pattern1|pattern2'
This will match all lines and highlight the patterns. The ^ matches every start of line, but won't get printed/highlighted since it's not a character.
(Note that most of the setups will use --color by default. You may not need that flag).
You can make sure that all lines match but there is nothing to highlight on irrelevant matches
egrep --color 'apple|' test.txt
Notes:
egrep may be spelled also grep -E
--color is usually default in most distributions
some variants of grep will "optimize" the empty match, so you might want to use "apple|$" instead (see: https://stackoverflow.com/a/13979036/939457)
EDIT:
This works with OS X Mountain Lion's grep:
grep --color -E 'pattern1|pattern2|$'
This is better than '^|pattern1|pattern2' because the ^ part of the alternation matches at the beginning of the line whereas the $ matches at the end of the line. Some regular expression engines won't highlight pattern1 or pattern2 because ^ already matched and the engine is eager.
Something similar happens for 'pattern1|pattern2|' because the regex engine notices the empty alternation at the end of the pattern string matches the beginning of the subject string.
[1]: http://www.regular-expressions.info/engine.html
FIRST EDIT:
I ended up using perl:
perl -pe 's:pattern:\033[31;1m$&\033[30;0m:g'
This assumes you have an ANSI-compatible terminal.
ORIGINAL ANSWER:
If you're stuck with a strange grep, this might work:
grep -E --color=always -A500 -B500 'pattern1|pattern2' | grep -v '^--'
Adjust the numbers to get all the lines you want.
The second grep just removes extraneous -- lines inserted by the BSD-style grep on Mac OS X Mountain Lion, even when the context of consecutive matches overlap.
I thought GNU grep omitted the -- lines when context overlaps, but it's been awhile so maybe I remember wrong.
You can use my highlight script from https://github.com/kepkin/dev-shell-essentials
It's better than grep cause you can highlight each match with it's own color.
$ command_here | highlight green "input" | highlight red "output"
Since you want matches highlighted, this is probably for human consumption (as opposed to piping to another program for instance), so a nice solution would be to use:
less -p <your-pattern> <your-file>
And if you don't care about case sensitivity:
less -i -p <your-pattern> <your-file>
This also has the advantage of having pages, which is nice when having to go through a long output
You can do it using only grep by:
reading the file line by line
matching a pattern in each line and highlighting pattern by grep
if there is no match, echo the line as is
which gives you the following:
while read line ; do (echo $line | grep PATTERN) || echo $line ; done < inputfile
If you want to print "all" lines, there is a simple working solution:
grep "test" -A 9999999 -B 9999999
A => After
B => Before
If you are doing this because you want more context in your search, you can do this:
cat BIG_FILE.txt | less
Doing a search in less should highlight your search terms.
Or pipe the output to your favorite editor. One example:
cat BIG_FILE.txt | vim -
Then search/highlight/replace.
If you are looking for a pattern in a directory recursively, you can either first save it to file.
ls -1R ./ | list-of-files.txt
And then grep that, or pipe it to the grep search
ls -1R | grep --color -rE '[A-Z]|'
This will look of listing all files, but colour the ones with uppercase letters. If you remove the last | you will only see the matches.
I use this to find images named badly with upper case for example, but normal grep does not show the path for each file just once per directory so this way I can see context.
Maybe this is an XY problem, and what you are really trying to do is to highlight occurrences of words as they appear in your shell. If so, you may be able to use your terminal emulator for this. For instance, in Konsole, start Find (ctrl+shift+F) and type your word. The word will then be highlighted whenever it occurs in new or existing output until you cancel the function.
I'd like to know if I can colour highlight the output of a shell command that matches certain strings.
For example, if I run myCommand, with the output below:
> myCommand
DEBUG foo bar
INFO bla bla
ERROR yak yak
I'd like all lines matching ^ERROR\s.* to be highlighted red.
Similarly, I'd like the same highlighting to be applied to the output of grep, less etc...
EDIT: I probably should mention that ideally I'd like to enable this feature globally via a 'profile' option in my .bashrc.
There is an answer in superuser.com:
your-command | grep -E --color 'pattern|$'
or
your-command | grep --color 'pattern\|$'
This will "match your pattern or the end-of-line on each line. Only the pattern is highlighted..."
You can use programs such as:
spc (Supercat)
grc (Generic Colouriser)
highlight
histring
pygmentize
grep --color
You can do something like this, but the commands won't see a tty (some will refuse to run or behave differently or do weird things):
exec > >(histring -fEi error) # Bash
If you want to enable this globally, you'll want a terminal feature, not a process that you pipe output into, because a pipe would be disruptive to some command (two problems are that stdout and stderr would appear out-of-order and buffered, and that some commands just behave differently when outputting to a terminal).
I don't know of any “conventional” terminal with this feature. It's easily done in Emacs, in a term buffer: configure font-lock-keywords for term-mode.
However, you should think carefully whether you really want that feature all the time. What if the command has its own colors (e.g. grep --color, ls --color)? Maybe it would be better to define a short alias to a colorizer command and run myCommand 2>&1|c when you want to colorize myCommand's output. You could also alias some specific always-colorize commands.
Note that the return status of a pipeline is its last command, so if you run myCommand | c, you'll get the status of c, not myCommand. Here's a bash wrapper that avoids this problem, which you can use as w myCommand:
w () {
"$#" | c
return $PIPESTATUS[0]
}
You could try (maybe needs a bit more escaping):
BLUE="$(tput setaf 4)"
BLACK="$(tput sgr0)"
command | sed "s/^ERROR /${BLUE}ERROR ${BLACK}/g"
Try
tail -f yourfile.log | egrep --color 'DEBUG|'
where DEBUG is the text you want to highlight.
You can use the hl command avalaible on github :
git clone http://github.com/mbornet-hl/hl
Then :
myCommand | hl -r '^ERROR.*'
You can use the $HOME/.hl.cfg configuration file to simplify the command line.
hl is written in C (source is available).
You can use up to 42 differents colors of text.
Use awk.
COLORIZE_AWK_COMMAND='{ print $0 }'
if [ -n "$COLORIZE" ]; then
COLORIZE_AWK_COMMAND='
/pattern1/ { printf "\033[1;30m" }
/pattern2/ { printf "\033[1;31m" }
// { print $0 "\033[0m"; }'
fi
then later you can pipe your output
... | awk "$COLORIZE_AWK_COMMAND"
printf is used in the patterns so we don't print a newline, just set the color.
You could probably enable it for specific commands using aliases and user defined shell functions wihtout too much trouble. If your coloring errors I assume you want to process stderr. Since stderr in unbuffered you would probably want to line buffer it by sending through a fifo.