I know I can use grep -v '^#' to remove lines starting with #
now I run into issues when I try to do this to remove the [
ie. grep -v '^[' or even sed '/^[/ d'
Why is this happening and how can accomplish this?
Consider this test file:
$ cat brackets
keep
[remove]
Using grep:
$ grep -v '^\[' brackets
keep
Or:
$ grep -v '^[[]' brackets
keep
Using sed:
$ sed '/^\[/d' brackets
keep
Or:
$ sed '/^[[]/d' brackets
keep
Why
When a computer command fails to work as expected, it is important to look at the error message. Consider:
$ grep -vE '^[' brackets
grep: Invalid regular expression
The error message is reporting that an invalid regular expression was found. This is because [ is a regex-active character: [...] is used to define a character list. Thus, if a regex contains an unescaped [, it must also contain a matching ]. There are two ways to avoid this:
Escape it. If [ is a regex-active character, then \[ will generally be treated as a regular (inactive) character.
Put it in a character list. [[] is a character list that matches only one character: [.
Related
I want to trim a string from one character, the last /, to either : or #, which ever appears first. An example would be:
https://www.example.com/?client=safari/this-text:not-this:or_this
would be trimmed to:
this-text
and
https://www.example.com/?client=safari/this-text#not-this:or_this
would be trimmed to:
this-text
I know I can trim text in bash from a specific character to another character, but is there a way to trim from one character to either of 2 characters?
Use grep like so: grep -Po '^.*/\K[^:#]*'
Examples:
echo 'https://www.example.com/?client=safari/this-text:not-this:or_this' | grep -Po '^.*/\K[^:#]*'
or:
echo 'https://www.example.com/?client=safari/this-text#not-this:or_this' | grep -Po '^.*/\K[^:#]*'
Output:
this-text
Here, grep uses the following options:
-P : Use Perl regexes.
-o : Print the matches only, 1 match/line, not the entire lines.
The regex ^.*/\K[^:#]* does the following:
^.*/ : Match from the beginning of the string (^) all the way up to the last slash ('/').
\K : Pretend that the match started at this position.
[^:#]* : zero or more occurrences (greedy) of any characters except : or #. This matches either until the end of the line, or until the next : or #, whichever comes first.
SEE ALSO:
grep manual
NOTE:
This works with GNU grep, which may need to be installed, depending on your system. For example, to install GNU grep on macOS, see this answer: https://apple.stackexchange.com/a/357426/329079
With a little Bash function:
trim() {
local str=${1##*/}
printf '%s\n' "${str%%[:#]*}"
}
This first trims everything up to and including the last /, then everything starting from the first occurrence of : or #.
In use:
$ trim 'https://www.example.com/?client=safari/this-text:not-this:or_this'
this-text
$ trim 'https://www.example.com/?client=safari/this-text#not-this:or_this'
this-text
Another way is to use sed: sed -e 's,^.*/,,' -e 's,[:#].*$,,'.
First -e command (s/regex/replacement/) removes text from the start to the last /, then the second -e removes from : or # to the end of the text.
echo 'https://www.example.com/?client=safari/this-text:not-this:or_this' | sed -e 's,^.*/,,' -e 's,[:#].*$,,'
this-text
I am trying to match a whole string in a list of new line separated strings. Here is my example:
[hemanth.a#gateway ~]$ echo $snapshottableDirs
/user/hemanth.a/dummy1 /user/hemanth.a/dummy3
[hemanth.a#gateway ~]$ echo $snapshottableDirs | tr -s ' ' '\n'
/user/hemanth.a/dummy1
/user/hemanth.a/dummy3
[hemanth.a#gateway ~]$ echo $snapshottableDirs | tr -s ' ' '\n' | grep -w '/user/hemanth.a'
/user/hemanth.a/dummy1
/user/hemanth.a/dummy3
My aim is to only find a match if and only if the string /user/hemanth.a exists as a whole word(in a new line) in the list of strings. But the above command is also returning strings that contain /user/hemanth.a.
This is a sample scenario. There is no guarantee that all the strings that I would want to match will be in the form of /user/xxxxxx.x. Ideally I would want to match the exact string if it exists in a new line as a whole word in the list.
Any help would be appreciated. thank you.
Update: Using fgrep -x '/user/hemanth.a' is probably a better solution here, as it avoids having to escape characters such as $ to prevent grep from interpreting them as meta-characters. fgrep performs a literal string match as opposed to a regular expression match, and the -x option tells it to only match whole lines.
Example:
> cat testfile.txt
foo
foobar
barfoo
barfoobaz
> fgrep foo testfile.txt
foo
foobar
barfoo
barfoobaz
> fgrep -x foo testfile.txt
foo
Original answer:
Try adding the $ regex metacharacter to the end of your grep expression, as in:
echo $snapshottableDirs | tr -s ' ' '\n' | grep -w '/user/hemanth.a$'.
The $ metacharacter matches the end of the line.
While you're at it, you might also want to use the ^ metacharacter, which matches the beginning of the line, so that grep '/user/hemanth.a$' doesn't accidentally also match something like /user/foo/user/hemanth.a.
So you'd have this:
echo $snapshottableDirs | tr -s ' ' '\n' | grep '^/user/hemanth\.a$'.
Edit: You probably don't actually want the -w here, so I've removed that from my answer.
Edit 2: #U. Windl brings up a good point. The . character in a regular expression is a metacharacter that matches any character, so grep /user/hemanth.a might end up matching things you're not expecting, such as /user/hemanthxa, etc. Or perhaps more likely, it would also match the line /user/hemanth/a. To fix that, you need to escape the . character. I've updated the grep line above to reflect this.
Update: In response to your question in the comments about how to escape a string so that it can be used in a grep regular expression...
Yes, you can escape a string so that it should be able to be used in a regular expression. I'll explain how to do so, but first I should say that attempting to escape strings for use in a regex can become very complicated with lots of weird edge cases. For example, an escaped string that works with grep won't necessarily work with sed, awk, perl, bash's =~ operator, or even grep -e.
On top of that, if you change from single quotes to double quotes, you might then have to add another level of escaping so that bash will expand your string properly.
For example, if you wanted to search for the literal string 'foo [bar]* baz$'using grep, you'd have to escape the [, *, and $ characters, resulting in the regular expression:
'foo \[bar]\* baz\$'
But if for some reason you decided to pass that expression to grep as a double-quoted string, you would then have to escape the escapes. Otherwise, bash would interpret some of them as escapes. You can see this if you do:
echo "foo \[bar]\* baz\$"
foo \[bar]\* baz$
You can see that bash interpreted \$ as an escape sequence representing the character $, and thus swallowed the \ character. This is because normally, in double quoted strings $ is a special character that begins a parameter expansion. But it left \[ and \* alone because [ and * aren't special inside a double-quoted string, so it interpreted the backslashes as literal \ characters. To get this expression to work as an argument to grep in a double-quoted string, then, you would have to escape the last backslash:
# This command prints nothing, because bash expands `\$` to just `$`,
# which grep then interprets as an end-of-line anchor.
> echo 'foo [bar]* baz$' | grep "foo \[bar]\* baz\$"
# Escaping the last backslash causes bash to expand `\\$` to `\$`,
# which grep then interprets as matching a literal $ character
> echo 'foo [bar]* baz$' | grep "foo \[bar]\* baz\\$"
foo [bar]* baz$
But note that "foo \[bar]\* baz \\$" will not work with sed, because sed uses a different regex syntax in which escaping a [ causes it to become a meta-character, whereas in grep you have to escape it to prevent it from being interpreted as a meta-character.
So again, yes, you can escape a literal string for use as a grep regular expression. But if you need to match literal strings containing characters that will need to be escaped, it turns out there's a better way: fgrep.
The fgrep command is really just shorthand for grep -F, where the -F tells grep to match "fixed strings" instead of regular expression. For example:
> echo '[(*\^]$' | fgrep '[(*\^]$'
[(*\^]$
This works because fgrep doesn't know or care about regular expressions. It's just looking for the exact literal string '[(*\^]$'. However, this sort of puts you back at square one, because fgrep will match on substrings:
> echo '/users/hemanth/dummy' | fgrep '/users/hemanth'
/users/hemanth/dummy
Thankfully, there's a way around this, which it turns out was probably a better approach than my initial answer, considering your specific needs. The -x option to fgrep tells it to only match the entire line. Note that -x is not specific to fgrep (since fgrep is really just grep -F anyway). For example:
> echo '/users/hemanth/dummy' | fgrep -x '/users/hemanth' # prints nothing
This is equivalent to what you would have gotten by escaping the grep regex, and is almost certainly a better answer than my previous answer of enclosing your regex in ^ and $.
Now, as promised, just in case you want to go this route, here's how you would escape a fixed string to use as a grep regex:
# Suppose we want to match the literal string '^foo.\ [bar]* baz$'
# It contains lots of stuff that grep would normally interpret as
# regular expression meta-characters. We need to escape those characters
# so grep will interpret them as literals.
> str='^foo.\ [bar]* baz$'
> echo "$str"
^foo.\ [bar]* baz$
> regex=$(sed -E 's,[.*^$\\[],\\&' <<< "$str")
> echo "$regex"
\^foo\.\\ \[bar]\* baz\$
> echo "$str" | grep "$regex"
^foo.\ [bar]* baz$
# Success
Again, for the reasons cited above, I don't recommend this approach, especially not when fgrep -x exists.
Read "Anchoring" in man grep:
Anchoring
The caret ^ and the dollar sign $ are meta-characters that respectively
match the empty string at the beginning and end of a line.
Also be aware that . matches any character (from said manual page):
The period . matches any single character.
I'm trying to search into the files the word: "http://{$_SERVER["HTTP_HOST"]}"
and if I find some line with that item I want to add /suitecrm/ next to the closing curly brace "}"
or replace it with: 'http://{$_SERVER["HTTP_HOST"]}/suitecrm' , but I need to escape the brackets without losing them,
How can I do that? please!!!
this is my code:
sudo grep -lir "HTTP_HOST" /var/www/html/suitecrm/ | xargs sudo sed -i 's,http://{$_SERVER["HTTP_HOST"]},http://{$_SERVER["HTTP_HOST"]}/suitecrm,g’ ;
Try:
sed -i 's,http://{\$_SERVER\["HTTP_HOST"\]},http://{\$_SERVER\["HTTP_HOST"\]}/suitecrm,g’
Eg:
$ echo 'http://{$_SERVER["HTTP_HOST"]}' | sed 's,http://{\$_SERVER\["HTTP_HOST"\]},http://{\$_SERVER\["HTTP_HOST"\]}/suitecrm,'
http://{$_SERVER["HTTP_HOST"]}/suitecrm
Actually I just batch replaced $ to \$, [ to \[, and ] to \].
It's called Escaping, certain characters have special meanings in RE, when you refer to them literally, you have to Escape them.
Check this:
Escapes
I have something like this in a file named file.txt
AA.201610.pancake.Paul
AA.201610.hello.Robert
A.201610.hello.Mark
Now, i ONLY get the first three fields in 3 variables like:
field1="A"
field2="201610"
field3='hello'.
I'd like to remove a line, if it contains exactly the first 3 fields, like , in the case described above, i want only the third line to be removed from the file.txt . Is there a way to do that? And is there a way to do that in the same file?
I tried with:
sed -i /$field1"."$field2"."$field3"."/Id file.txt
but of course this removes both the second and the third line
I suggest using awk for this as sed can only do regex search and that requires escaping all special meta-chars and anchors, word boundaries etc to avoid false matches.
Suggested awk with non-regex matching:
awk -F '[.]' -v f1="$field1" -v f2="$field2" -v f3="$field3" '
!($1==f1 && $2==f2 && $3==f3)' file
AA.201610.pancake.Paul
AA.201610.hello.Robert
Use ^ to anchor the pattern at the beginning of the line. Also note that . in a regex means "any character" and not a literal peridio. You have to escape it: either \. (be careful with shell escaping and the difference between single and double quotes) or [.]
Sed cannot do string matches, only regexp matches which becomes horrendously complicated to work around when you simply want to match a literal string (see Is it possible to escape regex metacharacters reliably with sed). Just use awk:
$ awk -v str="${field1}.${field2}.${field3}." 'index($0,str)!=1' file
AA.201610.pancake.Paul
AA.201610.hello.Robert
The question was about bash so in bash:
#!/usr/bin/env bash
field1="A"
field2="201610"
field3='hello'
IFS=
while read -r i
do
case "$i" in
"${field1}.${field2}.${field3}."*) ;;
*) echo -E "$i"
esac
done < file.txt
I want to search
$GLOBALS["\x61\156\x75\156\x61"] using grep but " and / not working perfectly.
grep -rl "$GLOBALS["\x61\156\x75\156\x61"]" <filename>
$GLOBALS["\x61\156\x75\156\x61"] is a virus malware starting code, lots of files are affected. I have a script through I want search effected files and remove top line
Since you are looking for an exact match and you don't want the expression to be interpreted by grep, you have to use -F and single quotes to avoid the variable being expanded:
grep -Frl '$GLOBALS["\x61\156\x75\156\x61"]' <filename>
^ ^ ^
From man grep:
-F, --fixed-strings
Interpret PATTERN as a list of fixed strings, separated by
newlines, any of which is to be matched. (-F is specified by POSIX.)
See another example of the usage of -F together with single quotes:
We create a file like this:
$ cat a
hello
${myarray[0]}
bye
And an array:
$ myarray=('hello' 'how' 'are' 'you')
Let's use single quotes and look for the value:
$ grep '${myarray[0]}' a
$
Let's use fixed string with double quotes -> it gets interpreted!
$ grep -F "${myarray[0]}" a
hello
Let's use -F and single quotes:
$ grep -F '${myarray[0]}' a
${myarray[0]} #this works!