I have something like this in a file named file.txt
AA.201610.pancake.Paul
AA.201610.hello.Robert
A.201610.hello.Mark
Now, i ONLY get the first three fields in 3 variables like:
field1="A"
field2="201610"
field3='hello'.
I'd like to remove a line, if it contains exactly the first 3 fields, like , in the case described above, i want only the third line to be removed from the file.txt . Is there a way to do that? And is there a way to do that in the same file?
I tried with:
sed -i /$field1"."$field2"."$field3"."/Id file.txt
but of course this removes both the second and the third line
I suggest using awk for this as sed can only do regex search and that requires escaping all special meta-chars and anchors, word boundaries etc to avoid false matches.
Suggested awk with non-regex matching:
awk -F '[.]' -v f1="$field1" -v f2="$field2" -v f3="$field3" '
!($1==f1 && $2==f2 && $3==f3)' file
AA.201610.pancake.Paul
AA.201610.hello.Robert
Use ^ to anchor the pattern at the beginning of the line. Also note that . in a regex means "any character" and not a literal peridio. You have to escape it: either \. (be careful with shell escaping and the difference between single and double quotes) or [.]
Sed cannot do string matches, only regexp matches which becomes horrendously complicated to work around when you simply want to match a literal string (see Is it possible to escape regex metacharacters reliably with sed). Just use awk:
$ awk -v str="${field1}.${field2}.${field3}." 'index($0,str)!=1' file
AA.201610.pancake.Paul
AA.201610.hello.Robert
The question was about bash so in bash:
#!/usr/bin/env bash
field1="A"
field2="201610"
field3='hello'
IFS=
while read -r i
do
case "$i" in
"${field1}.${field2}.${field3}."*) ;;
*) echo -E "$i"
esac
done < file.txt
Related
I need to extract path from a string. I found examples in another post, but missing additional steps.
I have a string as below:
title="test test good dskgkdh hdfyr /rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL kdlkfg nsfgf trhrnrt"
cobsrc=$(awk '{match($0,/\/[^"]*/,a);print a[0]}' <<< $title)
echo $cobsrc
Output is
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL kdlkfg nsfgf trhrnrt
I need only
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
What modification is required?
An existing post on similar query:
how to extract path from string in shell script
Four solutions, in order of my own preference.
First option would be simple parameter expansion, in two steps:
$ title="/${title#*/}"
$ title="${title%% *}"
$ echo "$title"
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
The first line removes everything up to the first slash (while prepending a slash to replace the one that's stripped", the second line removes everything from the first bit of whitespace that remains.
Or, if you prefer, use a regex:
$ [[ $title =~ ^[^/]*(/[^ ]+)\ ]]
$ echo ${BASH_REMATCH[1]}
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
The regex translates as:
null at the beginning of the line,
a run of zero or more non-slashes,
an atom:
a slash followed by non-space characters
a space, to end the previous atom.
The $BASH_REMATCH array contains the content of the bracketed atom.
Next option might be grep -o:
$ grep -o '/[^ ]*' <<<"$title"
(Result redacted -- you know what it'll be.)
You could of course assign this output to a variable using command substitution, which you already know about.
Last option is another external tool...
$ sed 's:^[^/]*::;s/ .*//' <<<"$title"
This is the same functionality as is handled by the parameter expansion (at the top of the answer) only in a sed script, which requires a call to an external program. Included only for pedantry. :)
Could you please try following.
echo "$title" | awk 'match($0,/\/.*\/[^ ]*/){print substr($0,RSTART,RLENGTH)}'
Output will be as follows.
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
Solution 2nd: Considering that your variable don't have space in between its value then following may help you too.
echo "$title" | awk '{sub(/[^/]* /,"");sub(/ .*/,"")} 1'
I trying to find and replace items using bash. I was able to use sed to grab out some of the characters, but I think I might be using it in the wrong matter.
I am basically trying to remove the characters after ";" and before "," including removing ","
sed -e 's/\(;\).*\(,\)/\1\2/'
That is what I used to replace it with nothing. However, it ends up replacing everything in the middle so my output came out like this:
cmd2="BMC,./socflash_x64 if=B600G3_BMC_V0207.ima;,reboot -f"
This is the original text of what I need to replace
cmd2="BMC,./socflash_x64 if=B600G3_BMC_V0207.ima;X,sleep 120;after_BMC,./run-after-bmc-update.sh;hba_fw,./hba_fw.sh;X,sleep 5;DB,2;X,reboot -f"
Is there any way to make it look like this output?
./socflash_x64 if=B600G3_BMC_V0207.ima;sleep 120;./run-after-bmc-update.sh;./hba_fw.sh;sleep 5;reboot -f
Ff there is any way to make this happen other than bash I am fine with any type of language.
Non-greedy search can (mostly) be simulated in programs that don't support it by replacing match-any (dot .) with a negated character class.
Your original command is
sed -e 's/\(;\).*\(,\)/\1\2/'
You want to match everything in between the semi-colon and the comma, but not another comma (non-greedy). Replace .* with [^,]*
sed -e 's/\(;\)[^,]*\(,\)/\1\2/'
You may also want to exclude semi-colons themselves, making the expression
sed -e 's/\(;\)[^,;]*\(,\)/\1\2/'
Note this would treat a string like "asdf;zxcv;1234,qwer" differently, since one would match ;zxcv;1234, and the other would match only ;1234,
In perl:
perl -pe 's/;.*?,/;/g;' -pe 's/^[^,]*,//' foo.txt
will output:
./socflash_x64 if=B600G3_BMC_V0207.ima;sleep 120;./run-after-bmc-update.sh;./hba_fw.sh;sleep 5;2;reboot -f
The .*? is non greedy matching before the comma. The second command is to remove from the beginning to the comma.
Something like:
echo $cmd2 | tr ';' '\n' | cut -d',' -f2- | tr '\n' ';' ; echo
result is:
./socflash_x64 if=B600G3_BMC_V0207.ima;sleep 120;./run-after-bmc-update.sh;./hba_fw.sh;sleep 5;2;reboot -f;
however, I thing your requirements are a few more complex, because 'DB,2' seems a particular case. After "tr" command, insert a "grep" or "grep -v" to include/exclude these cases.
I have a file like
key\0value\n
akey\0value\n
key2\0value\n
I have to create a script that take as argument a word. I have to return every lines having a key exactly the same than the argument.
I tried
grep -aF "$key\x0"
but grep seems to do not understand the \x0 (\0 same result). Futhermore, I have to check that the line begins with "$key\0"
I only can use sed grep and tr and other no maching commands
To have the \0 taken into account try :
grep -Pa "^key\x0"
it works for me.
Using sed
sed will work:
$ sed -n '/^key1\x00/p' file
key1value
The use of \x00 to represent a hex character is a GNU extension to sed. Since this question is tagged linux, that is not a problem.
Since the null character does not display well, one might (or might not) want to improve the display with something like this:
$ sed -n 's/^\(akey\)\x00/\1-->/p' file
akey-->value
Using sed with keys that contain special characters
If the key itself can contain sed or shell active characters, then we must escape them first and then run sed against the input file:
#!/bin/bash
printf -v script '/^%s\\x00/p' "$(sed 's:[]\[^$.*/]:\\&:g' <<<"$1")"
sed -n "$script" file
To use this script, simply supply the key as the first argument on the command line, enclosed in single-quotes, of course, to prevent shell processing.
To see how it works, let's look at the pieces in turn:
sed 's:[]\[^$.*/]:\\&:g' <<<"$1"
This puts a backslash escape in front of all sed-active characters.
printf -v script '/^%s\\x00/p' "$(sed 's:[]\[^$.*/]:\\&:g' <<<"$1")"
This creates a sed command using the escaped key and stores it in the shell variable script.
sed -n "$script" file
This runs sed using the shell variable script as the sed command.
Using awk
The question states that awk is not an acceptable tool. For completeness, though, here is an awk solution:
$ awk -F'\x00' -v k=key1 '$1 == k' file
key1value
Explanation:
-F'\x00'
awk divides the input up into records (lines) and divides the records up into fields. Here, we set the field separator to the null character. Consequently, the first field, denoted $1, is the key.
-v k=key1
This creates an awk variable, called k, and sets it to the key that we are looking for.
$1 == k
This statement looks for records (lines) for which the first field matches our specified key. If a match is found, the line is printed.
I have a string variable x=tmp/variable/custom-sqr-sample/test/example
in the script, what I want to do is to replace all the “-” with the /,
after that,I should get the following string
x=tmp/variable/custom/sqr/sample/test/example
Can anyone help me?
I tried the following syntax
it didnot work
exa=tmp/variable/custom-sqr-sample/test/example
exa=$(echo $exa|sed 's/-///g')
sed basically supports any delimiter, which comes in handy when one tries to match a /, most common are |, # and #, pick one that's not in the string you need to work on.
$ echo $x
tmp/variable/custom-sqr-sample/test/example
$ sed 's#-#/#g' <<< $x
tmp/variable/custom/sqr/sample/test/example
In the commend you tried above, all you need is to escape the slash, i.e.
echo $exa | sed 's/-/\//g'
but choosing a different delimiter is nicer.
The tr tool may be a better choice than sed in this case:
x=tmp/variable/custom-sqr-sample/test/example
echo "$x" | tr -- - /
(The -- isn't strictly necessary, but keeps tr (and humans) from mistaking - for an option.)
In bash, you can use parameter substitution:
$ exa=tmp/variable/custom-sqr-sample/test/example
$ exa=${exa//-/\/}
$ echo $exa
tmp/variable/custom/sqr/sample/test/example
If I run these commands from a script:
#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla
it is fine.
But, if I run:
#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s'
I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.
How can I get sed to recognize a $var as an environment variable as it is defined in the shell?
Your two examples look identical, which makes problems hard to diagnose. Potential problems:
You may need double quotes, as in sed 's/xxx/'"$PWD"'/'
$PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.
To nail both issues at once, perhaps
sed 's#xxx#'"$PWD"'#'
In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).
So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.
sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"
In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.
Another easy alternative:
Since $PWD will usually contain a slash /, use | instead of / for the sed statement:
sed -e "s|xxx|$PWD|"
You can use other characters besides "/" in substitution:
sed "s#$1#$2#g" -i FILE
一. bad way: change delimiter
sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's#xxx#'"$PWD"'#'
maybe those not the final answer,
you can not known what character will occur in $PWD, / : OR #.
if delimiter char in $PWD, they will break the expression
the good way is replace(escape) the special character in $PWD.
二. good way: escape delimiter
for example:
try to replace URL as $url (has : / in content)
x.com:80/aa/bb/aa.js
in string $tmp
URL
A. use / as delimiter
escape / as \/ in var (before use in sed expression)
## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js #escape fine
echo ${url//\//\/}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
URL
echo $tmp | sed "s/URL/${url//\//\/}/"
URL
OR
B. use : as delimiter (more readable than /)
escape : as \: in var (before use in sed expression)
## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
x.com:80/aa/bb/aa.js
With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:
sed 's/xxx/'$PWD'/'
will be expanded to
sed `s/xxx//home/yourname//
which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.
Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.
Dealing with VARIABLES within sed
[root#gislab00207 ldom]# echo domainname: None > /tmp/1.txt
[root#gislab00207 ldom]# cat /tmp/1.txt
domainname: None
[root#gislab00207 ldom]# echo ${DOMAIN_NAME}
dcsw-79-98vm.us.oracle.com
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'
--- Below is the result -- very funny.
domainname: ${DOMAIN_NAME}
--- You need to single quote your variable like this ...
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'
--- The right result is below
domainname: dcsw-79-98vm.us.oracle.com
VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1'
where VAR contains what you want to replace the field with
I had similar problem, I had a list and I have to build a SQL script based on template (that contained #INPUT# as element to replace):
for i in LIST
do
awk "sub(/\#INPUT\#/,\"${i}\");" template.sql >> output
done
If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:
PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected
for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:
sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt
BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working.
THEN use different delimiter:
sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt