For filtering errors in log files I have a command something like that
sudo grep -R --color=always -ri "err" *.log | grep -v "terry"
but the output isn't what I want. I still see lines like
mail.log:Mar 27 10:31:44 (removed) postfix/smtp[5449]: 4EB0822348: to=, relay=(removed), delay=6.6, delays=0.55/0.02/3.4/2.6, dsn=2.0.0, status=sent (250 OK id=1csFlH-00010k-6T)
Why is that line here when I have excluded "terry" from it?
Your "--color=always" is why you are still getting the the result. Remember that the pipe sends the stdout of one program to stdin of another. The output of your first grep command is outputting colors. In order to output color to the screen you have to send the color codes to the shell. In order for the shell to interpret these characters as colors it must use escape codes like this:
echo -e "This is \e[31mRed"
The word "Red" will be red when it is echo'ed. So grep is sending the escape characters to the second grep command. Go ahead and try it for yourself by redirecting your first grep command to a file and then examining the file.
grep -R --color=always -ri 'err' /tmp/log/syslog > /tmp/log/syslog2
Now open the file in a text editor (Don't cat the file out as you will just see the colors).
ar 26 10:30:59 zipmaster07 cinnamon-screensaver-dialog: pam_ecryptfs: seteuid ^[[01;31m^[[Kerr^[[m^[[Kor
Mar 26 14:27:19 zipmaster07 cinnamon-screensaver-dialog: pam_ecryptfs: seteuid ^[[01;31m^[[Kerr^[[m^[[Kor
t^[[01;31m^[[Kerr^[[m^[[Ky was here with an ^[[01;31m^[[Kerr^[[m^[[Kor.
mail.log:Mar 27 10:31:44 (removed) postfix/smtp[5449]: 4EB0822348: to=<t^[[01;31m^[[Kerr^[[m^[[Ky#(removed
The line "terry#...." is not terry anymore it is "t^[[01;31m^[[Kerr^...." and an inverted match of "terry" is not equal to "t^[[01;31m^[[Kerr^....", therefore grep includes it.
You need to remove the color option.
jschaeffer#zipmaster07 ~ $ grep -R -ri 'err' /tmp/log/sys2.log
pam_ecryptfs: seteuid err
pam_ecryptfs: seteuid err
terry was here with an error.
mail.log:Mar 27 10:31:44 (removed) postfix/smtp[5449]: 4EB0822348: to=<terry#(removed)>, relay=(removed), delay=6.6, delays=0.55/0.02/3.4/2.6, dsn=2.0.0, status=sent (250 OK id=1csFlH-00010k-6T)
Now with the second grep.
schaeffer#zipmaster07 ~ $ grep -R -ri 'err' /tmp/log/sys2.log | grep -v terry
pam_ecryptfs: seteuid err
pam_ecryptfs: seteuid err
Hopefully this all make sense.
Related
With sed I try to replace the value 0.1.233... On the command line there is no problem; however, when putting this command in a shell script, I get an error:
sed: couldn't open temporary file ../project/cas-dp-ap/sedwi3jVw: Permission denied
I don't understand where this temporary sedwi file comes from.
Do you have any idea why I have this temporary file and how I can pass it?
$(sed -i "s/$current_version/$version/" $PATHPROJET$CREATE_PACKAGE/Chart.yaml)
++ sed -i s/0.1.233/0.1.234/ ../project/cas-dp-ap/Chart.yaml
sed: couldn't open temporary file ../project/cas-dp-ap/sedwi3jVw: Permission denied
+ printf 'The version has been updated to : 0.1.234 \n\n \n\n'
The version has been updated to : 0.1.234
+ printf '***********************************'
sed -i is "in-place editing". However "in-place" isn't really. What happens is more like:
create a temporary file
run sed on original file and put changes into temporary file
delete original file
rename temporary file as original
For example, if we look at the inode of an edited file we can see that it is changed after sed has run:
$ echo hello > a
$ ln a b
$ ls -lai a b
19005916 -rw-rw-r-- 2 jhnc jhnc 6 Jan 31 12:25 a
19005916 -rw-rw-r-- 2 jhnc jhnc 6 Jan 31 12:25 b
$ sed -i 's/hello/goodbye/' a
$ ls -lai a b
19005942 -rw-rw-r-- 1 jhnc jhnc 8 Jan 31 12:25 a
19005916 -rw-rw-r-- 1 jhnc jhnc 6 Jan 31 12:25 b
$
This means that your script has to be able to create files in the folder where it is doing the "in-place" edit.
The proper syntax is identical on the command line and in a script. If you used $(...) at the prompt then you would have received the same error.
sed -i "s/$current_version/$version/" "$PATHPROJET$CREATE_PACKAGE/Chart.yaml"
(Notice also the quoting around the file name. Probably your private variables should use lower case.)
The syntax
$(command)
takes the output from command and tries to execute it as a command. Usually you would use this construct -- called a command substitution -- to interpolate the output of a command into a string, like
echo "Today is $(date)"
(though date +"Today is %c" is probably a better way to do that particular thing).
Currently i'm trying to catch a token from gitlab-runner list, which outputs something like that:
Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml
Ursain Bolt Executor=docker Token=abcdef678901234567890123456789 URL=https://my.gitlab.com
I am searching for the existence of a Token: Token=abcdef678901234567890123456789.
I tried several patterns (i'm familiar with regular expressions), the one i'd prefer to use looks as follows:
gitlab-runner list 2>&1 >/dev/null | grep -E 'Token=[a-f0-9]{30}'
I redirect stderr to stdout since it seems gitlab-runner prints to stdout. Not piping stdout to /dev/null makes no difference here.
This pattern does not match.
I tried sucessfully (grep returned 0) the following matches:
[a-f0-9]+
([a-f0-9])+'
([a-f0-9]){30}
Token
Token.
However, the following matches did not work (grep returned 1):
(Token.[a-f0-9]){30}
Token=[a-f0-9]{30}
Token=
Token\=
What am i missing, why does the regular combination of both patterns (Token and [a-f0-9]{30}) with an equal sign not work?
Raw Gitlab-Runner Output
bash-4.3# gitlab-runner list 2>&1 >/dev/null
Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml
Ursain Bolt Executor=docker Token=abcdef678901234567890123456789 URL=https://my.gitlab.com
Partial Hexdump Update
...
000000a0: 2045 7865 6375 746f 721b 5b30 3b6d 3d64 Executor.[0;m=d
000000b0: 6f63 6b65 7220 546f 6b65 6e1b 5b30 3b6d ocker Token.[0;m
000000c0: 3d65 6466 3834 3062 3436 6166 6434 3333 =edf840b46afd433
...
Version Numbers
bash-4.3# cat /etc/alpine-release
3.6.2
bash-4.3# bash --version
GNU bash, version 4.3.48(1)-release (x86_64-alpine-linux-musl)
bash-4.3# grep --version
grep (GNU grep) 3.0
Copyright (C) 2017 Free Software Foundation, Inc.
Trying to reproduce this by echoing the output doesn't seem to work, grep matches the pattern as intended.
If you are using GNU grep use the PCRE library that comes with it. The -P flag enables it and -o flag returns only the matched group than the entire line. The \K is a reserved character to indicate ignore the string in the matched string upto that part.
grep -Po 'Token=\K[a-f0-9]{30}'
The problem with grep -E 'Token=[a-f0-9]{30}' is it will return the entire line matching the regex and not the matching group alone, since the string you are looking for is part of a line with other words. You can of-course use the -o flag in your original expression, but it would still return Token=abcdef678901234567890123456789.
As a side note, you might want to ensure in which output stream your Token= string is available, because by your current re-direction 2>&1 > /dev/null you are suppressing the entire stdout to null stream and re-directing only stderr to console and grep is acting only on that.
Update
So if your problem is with ANSI escape sequences, you need to clean your output with sed or any tools of your choice and apply the grep regex filter, something like
gitlab-runner list 2>&1 >/dev/null | s/.\[0;m//g | grep -Po 'Token=\K[a-f0-9]{30}'
I fetched a hexdump of the output (using gitlab-runner list 2>&1 >/dev/null | xxd [xxd should be shipped with vim]. It turned out, that gitlab-runner adds Ansi Escape Sequences .[0;m before every equal sign:
...
000000a0: 2045 7865 6375 746f 721b 5b30 3b6d 3d64 Executor.[0;m=d
000000b0: 6f63 6b65 7220 546f 6b65 6e1b 5b30 3b6d ocker Token.[0;m
000000c0: 3d65 6466 3834 3062 3436 6166 6434 3333 =edf840b46afd433
...
The finally working pattern is Token.{5}=[a-f0-9]{30}.
I was working my way through a primer on Shell (Bash) Scripting and had the following doubt :
Why does not the following command print the contents of cp's directory : which cp | ls -l
Does not piping by definition mean that we pass the output of one command to another i.e. redirect the output ?
Can someone help me out ? I am a newbie ..
The output of which is being piped to the standard input of ls. However, ls doesn't take anything on standard input. You want it (I presume) to be passed as a parameter. There are a couple of ways of doing that:
which cp | xargs ls -l
or
ls -l `which cp`
or
ls -l $(which cp)
In the first example the xargs command takes the standard output of the previous previous command and makes each line a parameter to the command whose name immediately follows xargs. So, for instance
find / | xargs ls -l
will do an ls -l on each file in the filesystem (there are some issues with this with peculiarly named files but that's beyond the scope of this answer).
The remaining two are broadly equivalent and use the shell to do this, expanding the output from which into the command line for cp.
It would be,
$ ls -l $(which cp)
-rwxr-xr-x 1 root root 130304 Mar 24 2014 /bin/cp
OR
$ which cp | xargs ls -l
-rwxr-xr-x 1 root root 130304 Mar 24 2014 /bin/cp
To pass the output of one command as parameter of another command, you need to use xargs along with the pipe symbol.
From man xargs
xargs - build and execute command lines from standard input.xargs reads items
from the standard input, delimited by blanks (which can be protected
with double or single quotes or a backslash) or newlines, and executes
the command (default is /bin/echo) one or more times with any initial-
arguments followed by items read from standard input. Blank lines on
the standard input are ignored.
I have bunch of files containing the lines having label="some value". what I would like to do is to fetch those lines which does not have ", /> or as a matter of fact anything after that.
desired o/p
label="Relation From a Connector
I am doing something like this:
grep -rni "label="\" * | grep -v \"$
and getting this non desired o/p
label="Graceful Restart Helper"/>
label="BGP Route Reflector Policy"
This could be due to "\r\n" line endings in which case grep -v \"$ will not trigger because there is a carriage return character after the " and before the end of line. If this is the case then filter through
grep -E -v '"^M?$'
where ^M is control-M.
my grep command looks like this
zgrep -B bb -A aa "pattern" *
I would lke to have output as:
file1:line1
file1:line2
file1:line3
file1:pattern
file1:line4
file1:line5
file1:line6
</blank line>
file2:line1
file2:line2
file2:line3
file2:pattern
file2:line4
file2:line5
file2:line6
The problem is that its hard to distinguish when lines corresponding to the first found result end and the lines corresponding to the second found result start.
Note that although man grep says that "--" is added between contiguous group of matches. It works only when multiple matches are found in the same file. but in my search (as above) I am searching multiple files.
also note that adding a new blank line after every bb+aa+1 line won't work because what if a file has less than bb lines before the pattern.
pipe grep output through
awk -F: '{if(f!=$1)print ""; f=$1; print $0;}'
Pipe | any output to:
sed G
Example:
ls | sed G
If you man sed you will see
G Append's a newline character followed by the contents of the hold space to the pattern space.
The problem is that its hard to distinguish when lines corresponding to the first found result end and the lines corresponding to the second found result start.
Note that although man grep says that "--" is added between contiguous group of matches. It works only when multiple matches are found in the same file. but in my search (as above) I am searching multiple files.
If you don't mind a -- in lieu of a </blank line>, add the -0 parameter to your grep/zgrep command. This should allow for the -- to appear even when searching multiple files. You can still use the -A and -B flags as desired.
You can also use the --group-separator parameter, with an empty value, so it'd just add a new-line.
some-stream | grep --group-separator=
I can't test it with the -A and -B parameters so I can't say for sure but you could try using sed G as mentioned here on Unix StackEx. You'll loose coloring though if that's important.
There is no option for this in grep and I don't think there is a way to do it with xargs or tr (I tried), but here is a for loop that will do it (for f in *; do grep -H "1" $f && echo; done):
[ 11:58 jon#hozbox.com ~/test ]$ for f in *; do grep -H "1" $f && echo; done
a:1
b:1
c:1
d:1
[ 11:58 jon#hozbox.com ~/test ]$ ll
-rw-r--r-- 1 jon people 2B Nov 25 11:58 a
-rw-r--r-- 1 jon people 2B Nov 25 11:58 b
-rw-r--r-- 1 jon people 2B Nov 25 11:58 c
-rw-r--r-- 1 jon people 2B Nov 25 11:58 d
The -H is to display file names for grep matches. Change the * to your own file glob/path expansion string if necessary.
Try with -c 2; with printing a context I see grep is separating its found o/p