I want to trim a string from one character, the last /, to either : or #, which ever appears first. An example would be:
https://www.example.com/?client=safari/this-text:not-this:or_this
would be trimmed to:
this-text
and
https://www.example.com/?client=safari/this-text#not-this:or_this
would be trimmed to:
this-text
I know I can trim text in bash from a specific character to another character, but is there a way to trim from one character to either of 2 characters?
Use grep like so: grep -Po '^.*/\K[^:#]*'
Examples:
echo 'https://www.example.com/?client=safari/this-text:not-this:or_this' | grep -Po '^.*/\K[^:#]*'
or:
echo 'https://www.example.com/?client=safari/this-text#not-this:or_this' | grep -Po '^.*/\K[^:#]*'
Output:
this-text
Here, grep uses the following options:
-P : Use Perl regexes.
-o : Print the matches only, 1 match/line, not the entire lines.
The regex ^.*/\K[^:#]* does the following:
^.*/ : Match from the beginning of the string (^) all the way up to the last slash ('/').
\K : Pretend that the match started at this position.
[^:#]* : zero or more occurrences (greedy) of any characters except : or #. This matches either until the end of the line, or until the next : or #, whichever comes first.
SEE ALSO:
grep manual
NOTE:
This works with GNU grep, which may need to be installed, depending on your system. For example, to install GNU grep on macOS, see this answer: https://apple.stackexchange.com/a/357426/329079
With a little Bash function:
trim() {
local str=${1##*/}
printf '%s\n' "${str%%[:#]*}"
}
This first trims everything up to and including the last /, then everything starting from the first occurrence of : or #.
In use:
$ trim 'https://www.example.com/?client=safari/this-text:not-this:or_this'
this-text
$ trim 'https://www.example.com/?client=safari/this-text#not-this:or_this'
this-text
Another way is to use sed: sed -e 's,^.*/,,' -e 's,[:#].*$,,'.
First -e command (s/regex/replacement/) removes text from the start to the last /, then the second -e removes from : or # to the end of the text.
echo 'https://www.example.com/?client=safari/this-text:not-this:or_this' | sed -e 's,^.*/,,' -e 's,[:#].*$,,'
this-text
Related
I am trying to match a whole string in a list of new line separated strings. Here is my example:
[hemanth.a#gateway ~]$ echo $snapshottableDirs
/user/hemanth.a/dummy1 /user/hemanth.a/dummy3
[hemanth.a#gateway ~]$ echo $snapshottableDirs | tr -s ' ' '\n'
/user/hemanth.a/dummy1
/user/hemanth.a/dummy3
[hemanth.a#gateway ~]$ echo $snapshottableDirs | tr -s ' ' '\n' | grep -w '/user/hemanth.a'
/user/hemanth.a/dummy1
/user/hemanth.a/dummy3
My aim is to only find a match if and only if the string /user/hemanth.a exists as a whole word(in a new line) in the list of strings. But the above command is also returning strings that contain /user/hemanth.a.
This is a sample scenario. There is no guarantee that all the strings that I would want to match will be in the form of /user/xxxxxx.x. Ideally I would want to match the exact string if it exists in a new line as a whole word in the list.
Any help would be appreciated. thank you.
Update: Using fgrep -x '/user/hemanth.a' is probably a better solution here, as it avoids having to escape characters such as $ to prevent grep from interpreting them as meta-characters. fgrep performs a literal string match as opposed to a regular expression match, and the -x option tells it to only match whole lines.
Example:
> cat testfile.txt
foo
foobar
barfoo
barfoobaz
> fgrep foo testfile.txt
foo
foobar
barfoo
barfoobaz
> fgrep -x foo testfile.txt
foo
Original answer:
Try adding the $ regex metacharacter to the end of your grep expression, as in:
echo $snapshottableDirs | tr -s ' ' '\n' | grep -w '/user/hemanth.a$'.
The $ metacharacter matches the end of the line.
While you're at it, you might also want to use the ^ metacharacter, which matches the beginning of the line, so that grep '/user/hemanth.a$' doesn't accidentally also match something like /user/foo/user/hemanth.a.
So you'd have this:
echo $snapshottableDirs | tr -s ' ' '\n' | grep '^/user/hemanth\.a$'.
Edit: You probably don't actually want the -w here, so I've removed that from my answer.
Edit 2: #U. Windl brings up a good point. The . character in a regular expression is a metacharacter that matches any character, so grep /user/hemanth.a might end up matching things you're not expecting, such as /user/hemanthxa, etc. Or perhaps more likely, it would also match the line /user/hemanth/a. To fix that, you need to escape the . character. I've updated the grep line above to reflect this.
Update: In response to your question in the comments about how to escape a string so that it can be used in a grep regular expression...
Yes, you can escape a string so that it should be able to be used in a regular expression. I'll explain how to do so, but first I should say that attempting to escape strings for use in a regex can become very complicated with lots of weird edge cases. For example, an escaped string that works with grep won't necessarily work with sed, awk, perl, bash's =~ operator, or even grep -e.
On top of that, if you change from single quotes to double quotes, you might then have to add another level of escaping so that bash will expand your string properly.
For example, if you wanted to search for the literal string 'foo [bar]* baz$'using grep, you'd have to escape the [, *, and $ characters, resulting in the regular expression:
'foo \[bar]\* baz\$'
But if for some reason you decided to pass that expression to grep as a double-quoted string, you would then have to escape the escapes. Otherwise, bash would interpret some of them as escapes. You can see this if you do:
echo "foo \[bar]\* baz\$"
foo \[bar]\* baz$
You can see that bash interpreted \$ as an escape sequence representing the character $, and thus swallowed the \ character. This is because normally, in double quoted strings $ is a special character that begins a parameter expansion. But it left \[ and \* alone because [ and * aren't special inside a double-quoted string, so it interpreted the backslashes as literal \ characters. To get this expression to work as an argument to grep in a double-quoted string, then, you would have to escape the last backslash:
# This command prints nothing, because bash expands `\$` to just `$`,
# which grep then interprets as an end-of-line anchor.
> echo 'foo [bar]* baz$' | grep "foo \[bar]\* baz\$"
# Escaping the last backslash causes bash to expand `\\$` to `\$`,
# which grep then interprets as matching a literal $ character
> echo 'foo [bar]* baz$' | grep "foo \[bar]\* baz\\$"
foo [bar]* baz$
But note that "foo \[bar]\* baz \\$" will not work with sed, because sed uses a different regex syntax in which escaping a [ causes it to become a meta-character, whereas in grep you have to escape it to prevent it from being interpreted as a meta-character.
So again, yes, you can escape a literal string for use as a grep regular expression. But if you need to match literal strings containing characters that will need to be escaped, it turns out there's a better way: fgrep.
The fgrep command is really just shorthand for grep -F, where the -F tells grep to match "fixed strings" instead of regular expression. For example:
> echo '[(*\^]$' | fgrep '[(*\^]$'
[(*\^]$
This works because fgrep doesn't know or care about regular expressions. It's just looking for the exact literal string '[(*\^]$'. However, this sort of puts you back at square one, because fgrep will match on substrings:
> echo '/users/hemanth/dummy' | fgrep '/users/hemanth'
/users/hemanth/dummy
Thankfully, there's a way around this, which it turns out was probably a better approach than my initial answer, considering your specific needs. The -x option to fgrep tells it to only match the entire line. Note that -x is not specific to fgrep (since fgrep is really just grep -F anyway). For example:
> echo '/users/hemanth/dummy' | fgrep -x '/users/hemanth' # prints nothing
This is equivalent to what you would have gotten by escaping the grep regex, and is almost certainly a better answer than my previous answer of enclosing your regex in ^ and $.
Now, as promised, just in case you want to go this route, here's how you would escape a fixed string to use as a grep regex:
# Suppose we want to match the literal string '^foo.\ [bar]* baz$'
# It contains lots of stuff that grep would normally interpret as
# regular expression meta-characters. We need to escape those characters
# so grep will interpret them as literals.
> str='^foo.\ [bar]* baz$'
> echo "$str"
^foo.\ [bar]* baz$
> regex=$(sed -E 's,[.*^$\\[],\\&' <<< "$str")
> echo "$regex"
\^foo\.\\ \[bar]\* baz\$
> echo "$str" | grep "$regex"
^foo.\ [bar]* baz$
# Success
Again, for the reasons cited above, I don't recommend this approach, especially not when fgrep -x exists.
Read "Anchoring" in man grep:
Anchoring
The caret ^ and the dollar sign $ are meta-characters that respectively
match the empty string at the beginning and end of a line.
Also be aware that . matches any character (from said manual page):
The period . matches any single character.
How can I remove lines that contain more than 5 "." or less than 5 dots (simply put: 5 dots per line?
How can I write a regex that will detect it in bash using grep?
INPUT:
yGEtfWYBCBKtvxTbHxMK,126.221.42.321.0.147.30,10,Bad stuff is happening,http://mystuff.com/file.json
yGEtfWYBCBKtvxTbHxwK,126.221.42.21,10,Bad stuff is happening,http://mystuff.com/file.json
EXPECTED OUTPUT:
yGEtfWYBCBKtvxTbHxwK,176.221.42.21,10,Bad stuff is happening,http://mystuff.com/file.json
Tried:
grep -P '[.]{5}' stuff.txt
grep -P '[\.]{5}' stuff.txt
grep -P '([\.]{5})' stuff.txt
grep -P '\.{5}' stuff.txt
grep -E '([\.]{5}' stuff.txt
You can display only the lines that contain exactly 5 dots as follow :
grep '^[^.]*\.[^.]*\.[^.]*\.[^.]*\.[^.]*\.[^.]*$' stuff.txt
or if you want to factor it :
grep -E '^([^.]*\.){5}[^.]*$' stuff.txt
Using -ERE in this second one is helpful to avoid having to escape the \(\) and \{\}, in the first one grep's default BRE regex flavour is sufficient.
^ and $ are anchors representing respectively the start and end of the line that make sure we match the whole line and not just a part of it that contains 5 dots.
[^.] is a negated character class that will match anything but a dot.
They are quantified with * so that any number of non-dot characters can happen between each dot (you might want to change that to + if consecutive dots shouldn't be matched).
\. matches a literal dot (rather than any character, which the meta-character . outside of a character class would).
To detect specifically the bad IP address
Can you be certain that the IP address is always surrounded by commas and does not contain spaces - i.e. is never the first or last field?
Then, you might get away with:
grep -E ',\w+((\.\w+){2,3}|(\.\w+){5,}),'
If not, it is quite difficult to distinguish between a broken IP form with spaces and an ordinary sentence, so you might have to specify the column.
Using Perl one-liner to print only if number of "." exceeds 5
> cat five_dots.txt
yGEtfWYBCBKtvxTbHxMK,126.221.42.321.0.147.30,10,Bad stuff is happening,http://mystuff.com/file.json
yGEtfWYBCBKtvxTbHxwK,126.221.42.21,10,Bad stuff is happening,http://mystuff.com/file.json
> perl -ne '{ while(/\./g){$count++} print if $count > 5; $count=0 } ' five_dots.txt
yGEtfWYBCBKtvxTbHxMK,126.221.42.321.0.147.30,10,Bad stuff is happening,http://mystuff.com/file.json
>
I have a text file that contain a lot of mess text.
I used grep to get all the text that contains the string prod like this
cat textfile | grep "<host>prod*"
The result
<host>prod-reverse-proxy01</host>
<host>prod-reverse-proxy01</host>
<host>prod-reverse-proxy01</host>
Continually, i used sed with the intention to remove all the "host" part
cat textfile | grep "<host>prod*" | sed "s/<host>//g"; "s/</host>//g"
But only the first "host" was removed.
prod-reverse-proxy01</host>
prod-reverse-proxy01</host>
prod-reverse-proxy01</host>
How can i remove the other "/host" part?
sed -n -e "s/^<host>\(.*\)<\/host>/\1/p" textfile
sed can process your file directly. No need to grep or cat.
-n is there to suppress any lines that do not match. Last 'p' in the script will print all matching files.
Script dissection:
s/.../.../...
is the search/replace form. The bit between the first and the second '/' is what you search for. The bit between the second and third is what you replace it with. The last part is any commands you want to apply to the replacement.
Search:
^<host>\(.*\)<\/host>
finds all lines beginning with <host> followed by any text (.*) followed by </host>. Any text between <host> and </host> is stored into internal variable '1' using '(' and ')'. Note that (, ) and / (in </host>) have to be escaped.
Replace:
\1
Replace found text with contents of variable 1 (1 has to be escaped, otherwise, everything is replaced by character '1'.
Commands:
p
Print resulting line (after replacement).
Note: Your search involves removing two similar but not identical strings (<host> and </host>).
I think this sed is enough
sed 's/<[/]*host>//g' infile
I have a very long string, with over 2000 occurrences, and it look like that:
Input:
1a2a3a4a5a6a7a8absoad8ryaa90thneas... and more than
I want replace mutiple occurrences at the 3rd to the 450th occurrence in string to output:
1a2a3A4A5A6A7A8AbsoAd8ryAA90thneAs... and more than
I replaced "a" to "A", it replaced from the 3rd occurrence to ending, but I only want to replace from 3rd to 450th, this is my old script:
echo "1a2a3a4a5a6a7a8absoad8ryaa90thneas..." | sed 's/a/A/3g';
Does anyone help me? Or is there any other way? Thanks!
Save the string to a variable and then use brace expansion to target the positional character in the string you want to replace using bash global replace. Consider the following example:
## Our sample string. ##
string="abcde01234
## New, changed string. ##
echo "${string//${string:0:1}/${string:5:1}}"
In this example, when run as a bash script, the first character of $string is replaced with the sixth character of $string. Knowing this little trick, I am sure you will figure out a way to do what you need without having to use sed. Or, you can use the brace expansion similarly in sed.
## Our sample string ##
string="abcde01234"
## New, changed, string. Output is the same as above example. ##
sed -e "s/${string:0:1}/${string:5:1}/g" <(echo "$string")
This should be enough to get you headed in the right direction as far as figuring out the best way for your needs.
choose a character that you know for sure is not present in your input.. for example the ASCII NUL character, then you could do
$ # replacing 3rd to 4th
$ echo "a1a23a5a62a34a235a" | sed 's/a/\x0/3g; s/\x0/a/3g; s/\x0/A/g'
a1a23A5A62a34a235a
$ # replacing 3rd to 5th
$ echo "a1a23a5a62a34a235a" | sed 's/a/\x0/3g; s/\x0/a/4g; s/\x0/A/g'
a1a23A5A62A34a235a
$ # replacing 3rd to 6th
$ echo "a1a23a5a62a34a235a" | sed 's/a/\x0/3g; s/\x0/a/5g; s/\x0/A/g'
a1a23A5A62A34A235a
This might work for you (GNU sed):
sed -r 's/a/\n&/3;s//&\n/450;h;y/a/A/;G;s/.*\n(.*)\n.*\n(.*)\n.*\n/\2\1/' file
Split the string up into pieces using a marker (newline cannot occur in an unadulterated string because it is how sed splits up files), make a copy and then make changes, append the copy and rearrange the pieces.
Alternatively use GNU sed global replace flag:
sed 's/a/\n/451g;s//A/3g;s/\n/a/g' file
sed -E ':again ; s/^(.{4,449})a/\1A/ ; t again' file
you can use t command to branch to label. This is a conditional go to :label when a replacement has happened. So for a character range [x:y], regexp .*{x-1,y-1}a will match greedy for first time, but we loop until there are no more matches.
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 6 years ago.
I'm trying to replace text in a file with the output of another command. Unfortunately, the outputted text contains characters bash expands. For example, I'm running the following script to change the file (somestring references output that would break the sed command):
#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt
sed fails with this error:
sed: -e expression #1, char 30: unknown option to `s'
I think bash is substuting the contents of $somestring when building the sed command, but is then trying to expand the resulting text. I can't put the entire sed script in single quotes, I need bash to expand it the first time, just not the second. Any suggestions? Thanks
here the forward slash / is the problem. If it's the only issue you can set sed to use a different delimiter.
for example
$ somestring="abc/def"; echo xxx | sed 's/xxx/'"$somestring"'/'
sed: -e expression #1, char 11: unknown option to `s'
$ somestring="abc/def"; echo xxx | sed 's_xxx_'"$somestring"'_'
abc/def
you also need to worry about & and \ chars and escape them if can appear in the replacement text.
If you can't control the the replacement string, either you have to sanitize with another sed script or, alternatively use r command to read it from a file. For example,
$ seq 5 | sed -e '/3/{r replace' -e 'd}'
1
2
3slashes///1ampersand&and2backslashes\\end
4
5
where
$ cat replace
3slashes///1ampersand&and2backslashes\\end
You have several errors here:
the string somestring has characters that are significative for sed command (the most important being '/' that you are using as a delimiter) You can escape it, by substituting it with a previous
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
that will convert your / chars to \/ sequences.
you are using sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt which is looking to substitute the string titl followed by any number of e characters by that $somestring value, followed by a new line and the original one. Unfortunately, sed(1) doesn't allow you to use newline characters in the pattern substitution side of the s command, but you can afford the result by using the i command with a text consisting of you pattern (preceding any new line by a \ to interpret it as literal):
Finally the script leads to:
#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
sed '/^title/i\
'"$somestring\\
" $HOME/example.txt
If your shell is Bash, you can use parameter substitution to replace the problematic /:
somestring="{somestring//\//\\/}"
That looks scary, but is easier to understand if you look at the version that replaces x with __:
somestring="${somestring//x/__}"
It might be easier to use (say) underscore as the delimiter for your sed s command, and then the substitution above would be
somestring="${somestring//_/\\_}"
If you already have backslashes, you'll need to first replace those:
somestring="${somestring//\\/\\\\}"
somestring="{somestring//\//\\/}"
If there were other characters that needed escaping (e.g. on the search side of s///), then you could extend the above appropriately.
This URL provides the cleanest answer:
Command to escape a string in bash
printf "%q" "$someVariable"
will escape any characters you need escaped for you.