I have to filter a very verbose log output that the only pattern between messages for warning or errors is the output text color
Is there a way to filter by the output color when using grep?
When I try to use the color code I have an error due to the [
$ echo -e "Default \e[94mLight blue" | grep \e[94m
grep: brackets ([ ]) not balanced
I managed to be able to filter it by using just <number color>m
echo -e "Default \e[94mLight blue" | grep 94m
Default 94mLight blue
But it removes all color formatting, changes the output and also has the potential to match a lot of other text that is not just color formatting.
How can I accomplish it?
There are several places where this goes wrong:
The character [ has a special meaning for grep, unless you use the --fixed-strings (-F in short) option. So it needs to be quoted:
grep '\[94m'
There is also (seems to be?) no way to specify the Escape character (\e) directly for grep with \e, \033 or something similar. But you can use zsh's $'…' quoting to let zsh create the Escape character:
grep $'\e\\[94m'
Strings inside $'…' will be process like arguments to the print builtin. So \e will be replaced by the actual Escape character and \ needs to be doubled in order to get a single, quoted backslash.
Be sure to use grep without any options that change formatting, for example --color=auto. By default Oh-My-Zsh creates an alias named grep that includes this option, if it is available.
You can force zsh to use the command instead of an alias by prepending the command precommand modifier:
command grep
All in all the following command should do the trick:
echo -e "Default \e[94mLight blue" | command grep $'\e\\[94m'
Note 1: This requires zsh due to $'…'. A more portable version would be
echo -e "Default \e[94mLight blue" | command grep $(printf '\033\\[94m')
Note 2: Unless the log output sets color codes (also) at the beginning of each line or resets them at the end of the line, the coloration of the output may still be broken. In that case you may have to use awk in order to get a better result:
echo "Default \e[94mLight blue\nDefault \e[94mLight blue" |
awk '/\033\[94m/ {printf "%s\033[0m\n", $0}'
This appends the reset coder \e[0m to the end of each matching line when printing it. Note that awk (at least the GNU version) does not use \e to signifiy the Escape character, you have to use 033 instead.
You can check, whether grep is an alias with this command:
whence -v grep
It is also possible that the environment variable GREP_OPTIONS is set to include such options. To check this, you can run this command:
echo "${(t)GREP_OPTIONS}: $GREP_OPTIONS}"
This shows the type of GREP_OPTIONS followed by its value (type: value). If the type does not contain export or the value is empty there should be no problem.
You can try this
echo -e "Default \e[94mLight blue" | sed -n '/\[94m/p'
This printed colors for me in zsh. I couldn't manage to include the '\e'
Related
I am trying to convert 2015-06-03_18-05-30 to 20150603180530 using sed.
I have this:
$ var='2015-06-03_18-05-30'
$ echo $var | sed 's/\-\|\_//g'
$ echo $var | sed 's/-|_//g'
None of these are working. Why is the alternation not working?
As long as your script has a #!/bin/bash (or ksh, or zsh) shebang, don't use sed or tr: Your shell can do this built-in without the (comparatively large) overhead of launching any external tool:
var='2015-06-03_18-05-30'
echo "${var//[-_]/}"
That said, if you really want to use sed, the GNU extension -r enables ERE syntax:
$ sed -r -e 's/-|_//g' <<<'2015-06-03_18-05-30'
20150603180530
See http://www.regular-expressions.info/posix.html for a discussion of differences between BRE (default for sed) and ERE. That page notes, in discussing ERE extensions:
Alternation is supported through the usual vertical bar |.
If you want to work on POSIX platforms -- with /bin/sh rather than bash, and no GNU extensions -- then reformulate your regex to use a character class (and, to avoid platform-dependent compatibility issues with echo[1], use printf instead):
printf '%s\n' "$var" | sed 's/[-_]//g'
[1] - See the "APPLICATION USAGE" section of that link, in particular.
Something like this ought to do.
sed 's/[-_]//g'
This reads as:
s: Search
/[-_]/: for any single character matching - or _
//: replace it with nothing
g: and do that for every character in the line
Sed operates on every line by default, so this covers every instance in the file/string.
I know you asked for a solution using sed, but I offer an alternative in tr:
$ var='2015-06-03_18-05-30'
$ echo $var | tr -d '_-'
20150603180530
tr should be a little faster.
Explained:
tr stands for translate and it can be used to replace certain characters with another ones.
-d option stands for delete and it removes the specified characters instead of replacing them.
'_-' specifies the set of characters to be removed (can also be specified as '\-_' but you need to escape the - there because it's considered another option otherwise).
Easy:
sed 's/[-_]//g'
The character class [-_] matches of the characters from the set.
sed 's/[^[:digit:]]//g' YourFile
Could you tell me what failed on echo $var | sed 's/\-\|\_//g', it works here (even if escapping - and _ are not needed and assuming you use a GNU sed due to \| that only work in this enhanced version of sed)
On OSX:
bash-3.2$ echo "abc" | sed 's/b/\x1b[31mz\x1b[m/'
ax1b[31mzx1b[mc
Whereas on Linux:
$ echo "abc" | sed 's/b/\x1b[31mz\x1b[m/'
azc
and the z correctly shows up red.
Is this a limitation of bash 3.2? My Linux test here runs bash 4.1.2.
The weird thing is on my linux environment at work the bash is version below 3.2, and it works there too.
Also, this might be related but is probably not:
bash-3.2$ echo "abc" | sed 's/b/^[[31mz^[[m/'
31mz$'m/'azc
Again, specific to BSD sed. It's pretty puzzling: Seems like something is causing the shell or sed to echo some mangled portion of the command to the terminal somehow? It is always preceding the correct output of the command, however. Where's that dollar sign coming from?
(don't be confused by colors in my commands (which come after the cyan unicode character that looks like a less bent > which is my prompt), I use syntax highlighting with zsh)
OS X's version of sed doesn't do the escape substitutions you're asking for. You can get around this by using $'...' to have bash do the substitution before handing the string to sed:
$ echo "abc" | sed 's/b/\x1b[31mz\x1b[m/'
ax1b[31mzx1b[mc
$ echo "abc" | sed $'s/b/\x1b[31mz\x1b[m/'
azc
(You'll have to trust me the "z" is red in the second one.) But note that this may require that in some cases you may have to double-escape things you want sed to do the escape substitution on.
Oh. right so the shell version does not affect this. No idea why I thought that.
The culprit is just that BSD sed doesn't do translation, so the solution is just the Ctrl+V approach of using the raw escape byte in the sed command string.
I've been trying to make tail a little more readable for server startups. My current command filters out most of the INFO and DEBUG messages from the startup:
tail -F ../server/durango/log/server.log | grep -e "ERROR" -e "WARN" -e "Shutdown" -e "MicroKernel" | grep --color=auto -E 'MicroKernel|$'
What I would like to do is craft something that would highlight WARN in yellow and ERROR in red, and MicroKernel in green. I tried just piping grep --color=auto multiple times, but the only color that survives is the last command in the pipe.
Is there a one liner to do this? Or even a many-liner?
yes, there is way to do this. That is, as long as your terminal supports ANSI escape sequences. This is most terminals that exist.
I think I don't need explain how to grep, sed etc. point is the color right?
see below, this will make
WARN yellow
ERROR red
foo green
here is example:
kent$ echo "WARN
ERROR
foo"|sed 's#WARN#\x1b[33m&#; s#ERROR#\x1b[31m&#; s#foo#\x1b[32m&#'
Note: \x1b is hexadecimal for the ESC character (^VEsc).
to see the result:
I wrote a script for this years ago. You can easily cover the case of multiple colors by piping successive invocations of highlight to each other.
From the README:
Usage: ./highlight [-i] [--color=COLOR_STRING] [--] <PATTERN0> [PATTERN1...]
This is highlight version 1.0.
This program takes text via standard input and outputs it with the given
perlre(1) pattern(s) highlighted with the given color. If no color option
is specified, it defaults to 'bold red'. Colors may be anything
that Perl's Term::ANSIColor understands. This program is similar to
"grep --color PATTERN" except both matching and non-matching lines are
printed.
The default color can be selected via the $HIGHLIGHT_COLOR environment
variable. The command-line option takes precedence.
Passing -i or --ignore-case will enable case-insensitive matching.
If your pattern begins with a dash ('-'), you can pass a '--' argument
after any options and before your pattern to distinguish it from an
option.
I have been using a tool called grc for this for years. works like a charm. It comes with some quite good templates for many standard log outputs and formats and it is easy to define your own.
A command I use often is
grc tail -f /var/log/syslog
It colorizes the syslog output so it is easy to spot errors (typically marked red.
Find the tool here:
https://github.com/garabik/grc
(it is also available as package for most common linux flavours).
I wrote TxtStyle, a small utility for colorising logs. You define regular expressions to highlight in ~/.txts.conf file:
[Style="example"]
!red: regex("error")
green: regex("\d{4}-\d\d-\d\d")
# ...
And then apply the styles:
txts -n example example.log
or you can also pipe the output
tail -f example.log | txts -n example
You can create a colored log instead of using a complex command.
For php is like this:
echo "^[[30;43m".$ip."^[[0m";
The key point is to use Ctrl-v ctrl-[ to input a green ^[ under insert mode in vim, direct input ^[ does not work.
More info here
My sample using awk. Match log format like: xxxx [debug] xxxxx xxxx xxxx
black=30m
red=31m
green=32m
yellow=33m
blue=34m
magenta=35m
cyan=36m
white=37m
blacklog="\"\033[$black\" \$0 \"\033[39m\""
redlog="\"\033[$red\" \$0 \"\033[39m\""
greenlog="\"\033[$green\" \$0 \"\033[39m\""
yellowlog="\"\033[$yellow\" \$0 \"\033[39m\""
bluelog="\"\033[$blue\" \$0 \"\033[39m\""
magentalog="\"\033[$magenta\" \$0 \"\033[39m\""
cyanlog="\"\033[$cyan\" \$0 \"\033[39m\""
whitelog="\"\033[$white\" \$0 \"\033[39m\""
trace="/\[trace\]/ {print $redlog}"
debug="/\[debug\]/ {print $magentalog}"
info="/\[info\]/ {print $greenlog}"
warning="/\[warning\]/ {print $bluelog}"
error="/\[error\]/ {print $yellowlog}"
yourcommand | awk "$trace $debug $info $warning $error"
I want to use echo to display(not content) directories that start with atleast 2 characters but can't begin with "an"
For example if had the following in the directory:
a as an23 an23 blue
I would only get
as blue back
I tried echo ^an* but that returns the directory with 1 charcter too.
Is there any way i can do this in the form of echo globalpattern
You can use the shells extended globbing feature, in bash:
bash$ setsh -s extglob
bash$ echo !(#(?|an*))
The !() construct inverts its internal expression, see this for more.
In zsh:
zsh$ setopt extendedglob
zsh$ print *~(?|an*)
In this case the ~ negates the pattern before the tilde. See the manual for more.
Since you want at least two characters in the names, you can use printf '%s\n' ??* to echo each such name on a separate line. You can then eliminate those names that start with an with grep -v '^an', leading to:
printf '%s\n' ??* | grep -v '^an'
The quotes aren't strictly necessary in the grep command with modern shells. Once upon a quarter of a century or so ago, the Bourne shell had ^ as a synonym for | so I still use quotes around carets.
If you absolutely must use echo instead of printf, then you'll have to map white space to newlines (assuming you don't have any names that contain white space).
I'm trying with just the echo command, no grep either?
What about:
echo [!a]?* a[!n]*
The first term lists all the two-plus character names not beginning with a; the second lists all the two-plus character names where the first is a and the second is not n.
This should do it, but you'd likely be better off with ls or even find:
echo * | tr ' ' '\012' | egrep '..' | egrep -v '^an'
Shell globbing is a form of regex, but it's not as powerful as egrep regex's.
This question already has answers here:
Colorized grep -- viewing the entire file with highlighted matches
(24 answers)
Closed 7 years ago.
When using grep, it will highlight any text in a line with a match to your regular expression.
What if I want this behaviour, but have grep print out all lines as well? I came up empty after a quick look through the grep man page.
Use ack. Checkout its --passthru option here: ack. It has the added benefit of allowing full perl regular expressions.
$ ack --passthru 'pattern1' file_name
$ command_here | ack --passthru 'pattern1'
You can also do it using grep like this:
$ grep --color -E '^|pattern1|pattern2' file_name
$ command_here | grep --color -E '^|pattern1|pattern2'
This will match all lines and highlight the patterns. The ^ matches every start of line, but won't get printed/highlighted since it's not a character.
(Note that most of the setups will use --color by default. You may not need that flag).
You can make sure that all lines match but there is nothing to highlight on irrelevant matches
egrep --color 'apple|' test.txt
Notes:
egrep may be spelled also grep -E
--color is usually default in most distributions
some variants of grep will "optimize" the empty match, so you might want to use "apple|$" instead (see: https://stackoverflow.com/a/13979036/939457)
EDIT:
This works with OS X Mountain Lion's grep:
grep --color -E 'pattern1|pattern2|$'
This is better than '^|pattern1|pattern2' because the ^ part of the alternation matches at the beginning of the line whereas the $ matches at the end of the line. Some regular expression engines won't highlight pattern1 or pattern2 because ^ already matched and the engine is eager.
Something similar happens for 'pattern1|pattern2|' because the regex engine notices the empty alternation at the end of the pattern string matches the beginning of the subject string.
[1]: http://www.regular-expressions.info/engine.html
FIRST EDIT:
I ended up using perl:
perl -pe 's:pattern:\033[31;1m$&\033[30;0m:g'
This assumes you have an ANSI-compatible terminal.
ORIGINAL ANSWER:
If you're stuck with a strange grep, this might work:
grep -E --color=always -A500 -B500 'pattern1|pattern2' | grep -v '^--'
Adjust the numbers to get all the lines you want.
The second grep just removes extraneous -- lines inserted by the BSD-style grep on Mac OS X Mountain Lion, even when the context of consecutive matches overlap.
I thought GNU grep omitted the -- lines when context overlaps, but it's been awhile so maybe I remember wrong.
You can use my highlight script from https://github.com/kepkin/dev-shell-essentials
It's better than grep cause you can highlight each match with it's own color.
$ command_here | highlight green "input" | highlight red "output"
Since you want matches highlighted, this is probably for human consumption (as opposed to piping to another program for instance), so a nice solution would be to use:
less -p <your-pattern> <your-file>
And if you don't care about case sensitivity:
less -i -p <your-pattern> <your-file>
This also has the advantage of having pages, which is nice when having to go through a long output
You can do it using only grep by:
reading the file line by line
matching a pattern in each line and highlighting pattern by grep
if there is no match, echo the line as is
which gives you the following:
while read line ; do (echo $line | grep PATTERN) || echo $line ; done < inputfile
If you want to print "all" lines, there is a simple working solution:
grep "test" -A 9999999 -B 9999999
A => After
B => Before
If you are doing this because you want more context in your search, you can do this:
cat BIG_FILE.txt | less
Doing a search in less should highlight your search terms.
Or pipe the output to your favorite editor. One example:
cat BIG_FILE.txt | vim -
Then search/highlight/replace.
If you are looking for a pattern in a directory recursively, you can either first save it to file.
ls -1R ./ | list-of-files.txt
And then grep that, or pipe it to the grep search
ls -1R | grep --color -rE '[A-Z]|'
This will look of listing all files, but colour the ones with uppercase letters. If you remove the last | you will only see the matches.
I use this to find images named badly with upper case for example, but normal grep does not show the path for each file just once per directory so this way I can see context.
Maybe this is an XY problem, and what you are really trying to do is to highlight occurrences of words as they appear in your shell. If so, you may be able to use your terminal emulator for this. For instance, in Konsole, start Find (ctrl+shift+F) and type your word. The word will then be highlighted whenever it occurs in new or existing output until you cancel the function.