I am trying to use sed to delete a string from a given file. The string to be removed is stored in password. Here is what I have
sed -i "s/$password//g" shadowCopy.txt
It keeps telling me
sed: -e expression #1, char 0: no previous regular expression
I understand that it is yelling at me for the blank in the spot designated for the replacement regular expression, but I don't want to replace it with anything. Lots of looking online says that I'm doing it the way i should be. Any guesses?
I have also tried blank=""
sed -i "s/$password/$blank/g" shadowCopy.txt which gives me the same error. I have also looked at maybe using awk somehow, but cant figure out a way. Really any suggestion as to a way to delete a string in a file will satisfy
EDIT: Using the suggestion below my code is as follows
#Retrieve the root password hash from shadowCopy
password= awk -F: '$1=="root" {print $2}' shadowCopy.txt
#Remove the root password
pw="$password" perl -pe 'BEGIN { $search = quotemeta($ENV{pw}); } s/$search//g' shadowCopy.txt
and my output is
$1$aj/Pot/V$H4A7chbz7rfsRIrdL7GO61 //This is the password
root:$1$aj/Pot/V$H4A7chbz7rfsRIrdL7GO61:16469:0:99999:7:::
bin:*:16229:0:99999:7:::
Why is the password being printed out? And obviously when the file contents are dumped to show the changes, the password is still there. If i add echo $password nothing prints so clearly it isn't being properly intialized. Obviously the code in my assignment line works somewhat as it prints out what i'm wanting but it isnt storing it in the variable
RESOLVED: cp shadowCopy.txt shadowCopy.txt~ && awk 'BEGIN{FS=OFS=":"} $1=="root"{$2=""}1' shadowCopy.txt~ > shadowCopy.txt && rm shadowCopy.txt~ is what actually ended up doing it for me. In case anyone in the future cares to know
The problem is that sed cannot operate on strings, only on regexps with some additional restrictions. This will work:
awk -v tgt="$password" 's=index($0,tgt){$0 = substr($0,1,s-1) substr($0,s+length(tgt)}1' file
because it's using only string operations.
Given your newly posted information, you want this:
awk 'BEGIN{FS=OFS=":"} $1=="root"{$2=""} 1' shadowCopy.txt > tmp$$ &&
mv tmp$$ shadowCopy.txt
or if you have GNU awk and care about not naming the tmp file:
awk -i inplace 'BEGIN{FS=OFS=":"} $1=="root"{$2=""} 1' shadowCopy.txt
It happens when your variable is resolved to an empty string. Look:
p="a" && sed "s/$p/?/g" infile
That yields:
J?n 16 08:33:18 m?il.
And:
p="" && sed "s/$p//" infile
That yields:
sed: -e expression #1, char 0: no previous regular expression
Related
I am try to update a file and doing some transformation using any linux tool.
For example, here I am trying with awk.
Would be great to know how to get the rest of the pattern?
awk -F '/' '{print $1"/raw"$2}' <<< "string1/string2/string3/string4/string5"
string1,rawstring2
here I dont know how many "/" is there and I want to get the output:
string1/rawstring2/string3/string4/string5
Something like
awk -F/ -v OFS=/ '{ $2 = "raw" $2 } 1' <<< "string1/string2/string3/string4/string5"
Just modify the desired field, and print out the changed line (Have to set OFS so it uses a slash instead of a space to separate fields on output, and a pattern of 1 uses the default action of printing $0. It's an idiom you'll see a lot of with awk.)
Also possible with sed:
sed -E 's|([^/]*/)|\1raw|' <<< "string1/string2/string3/string4/string5"
The \1 in the replacement string reproduces the bit inside parenthesis and appends raw to it.
Equivalent to
sed 's|\([^/]*/\)|\1raw|' <<< "string1/string2/string3/string4/string5"
I want to know how to use two 'grep' and 'sed' utilities or something else in order to replace the substring. I will explain what I want to do below.
We have the file 'test.txt' with the following string:
A1='AA1', A2='AA2', A3='AA3', A4='AA4', A5{ATTR}='AA5', A6='keyword_A'
After searching 'keyword_A' using grep, I want to replace the value of A5 with other string, for example, "NEW".
A1='AA1', A2='AA2', A3='AA3', A4='AA4', A5{ATTR}='NEW', A6='keyword_A'
I tried to use two commands like
grep keyword_A test.txt | sed -e 's/blabla/blabla/'
After trying all I know, I gave up at all.
Please let me know the right solution.
First, you never need grep and sed. Sed has a full regular-expression search engine, so it is a superset of grep. This command will read test.txt, change the lines that you've indicated, and print the entire result on standard output:
sed "/keyword_A/s/A5{ATTR}='[A-Z0-9]*'/A5{ATTR}='NEW'/g" < test.txt
If you want to store the results back into the file test.txt, use the -i (in-place editing) switch to sed:
sed "/keyword_A/s/A5{ATTR}='[A-Z0-9]*'/A5{ATTR}='NEW'/g" -i.bak test.txt
If you want to select only the indicated lines, modify those, and print only those lines to standard out, use a combination of the p (print) command and the -n (no output) switch.
sed "/keyword_A/s/A5{ATTR}='[A-Z0-9]*'/A5{ATTR}='NEW'/gp" -n test.txt
Using grep+sed is always the wrong approach. Here's one way to do it with GNU awk:
$ awk '/keyword_A/{ $0=gensub(/(A5({[^}]+})?=\047)[^\047]+/,"\\1NEW",1) } 1' file
A1='AA1', A2='AA2', A3='AA3', A4='AA4', A5{ATTR}='NEW', A6='keyword_A'
Using a couple variables you could define the keyword and replacement ( if they change at all ):
q="keyword_A"
r="NEW"
Then with sed:
sed -r "s/^(.+\{.+\}=')(.+)('.+"${q}".+)$/\1"${r}"\3/" file
Result:
A1='AA1', A2='AA2', A3='AA3', A4='AA4', A5{ATTR}='NEW', A6='keyword_A'
A5="NEW"
A6="keyword_A"
# with sed
sed "s/='[^']*\(',[[:blank:]]*A6='${A6}'\)/='${A5}\1/" YourFile
# with awk
awk -F "'" -v A5="${A5}" -v A6="${A6}" '
BEGIN { OFS="\047" }
$12 == A6 { $10 = A5; $0 = $0 }
7
' YourFile
Change by the end of the string, for sed and using ' as field separator in awk instead of traditional space.
assuming there is no ' in value (or need to treat the escaping method) for awk version
We can just directly replace the fifth column when the sting keyword_A is found as shown below:
awk -F, 'BEGIN{OFS=",";}/keyword_A/{$5="A5{ATTR}='"'"NEW"'"'"}1' filename
Couple of slight alternatives:
sed -r "/keyword_A/s/(A5[^']*')[^']*/\1NEW/"
awk -F"'" '/keyword_A/{$10 = "NEW"}1' OFS="'"
Of course the negative with awk is afterwards you would have to rename the new file.
I have the following sample file:
evtlog.161202.002609.debugevtlog.161201.162408.debugevtlog.161202.011046.debugevtlog.161202.002809.debugevtlog.161201.160035.debugevtlog.161201.155140.debugevtlog.161201.232156.debugevtlog.161201.145017.debugevtlog.161201.154816.debug
I want to separate the string and add a newline after matching "debug" like this:
evtlog.161202.002609.debug
evtlog.161201.162408.debug
So far I tried almost everything with sed, but it doesn't seem to do what I want.
sed 's/debug/{G}' latest_evtlogs.out
sed '/debug/i "SAD"' latest_evtlogs.out
etc...
sed 's/debug/\n/g' latest_evtlogs.out doesn't work when I add it as a pipe in the script , but it does when I run it manually.
Here's how I generate the file:
printf $(ls -l $EVTLOG_PATH/evtlog|tail -n 10|awk '{printf $8 , "%s\n\n"}'|sed 's/debug/\n/g') >> latest_evtlogs.out
Initially I wanted to just add newline with awk, but it doesn't work either.
Any ideas why I can't separate the string with a newline ?
I'm using :
Distributor ID: Debian
Description: Debian GNU/Linux 5.0.10 (lenny)
Release: 5.0.10
Codename: lenny
Just add a new line after debug:
sed 's/debug/&\n/g' file
Note & prints back the matched text, so it is a way to print "debug" back.
This returns:
evtlog.161202.002609.debug
evtlog.161201.162408.debug
evtlog.161202.011046.debug
evtlog.161202.002809.debug
evtlog.161201.160035.debug
evtlog.161201.155140.debug
evtlog.161201.232156.debug
evtlog.161201.145017.debug
evtlog.161201.154816.debug
The problem is, that you are using the output of sed in a command expansion. In this context your shell will replace all newlines with spaces. The spaces are then used to do the word splitting, so that printf sees each line as a separate argument, interpreting the first line as the format argument and ignoring the rest as there are printf-placeholders in the format.
It should work if you drop the outer printf $() from your command and just redirect the output from your pipeline to your file:
ls -l $EVTLOG_PATH/evtlog|tail -n 10|awk '{printf $8 , "%s\n\n"}'|sed 's/debug/\n/g' >> latest_evtlogs.out
Maybe Perl is "happier" than sed on your system:
perl -pe 's/debug/&\n/g' < YourLogFile
Get will append what is in the hold buffer unto the pattern space (Usually just the current line read from the input file) So this cannot be used.
insert will print the specified text to standard output. So this cannot be used.
What you you want to to replace all debug with debug^J, where ^J is a newline, dependent on the sed version, you can either do:
sed 's/debug/&\n/g' input_file
But \n is - afaik - not strictly specified in POSIX sed. One can however use c strings:
sed 's/debug/&'$'\n''/g' input_file
Or a multi line string:
sed 's/debug/&\
/g' input_file
Thank you all for the answers.I finally did it like this :
echo $(ls -l $EVTLOG_PATH/evtlog|tail -n 10|awk '{printf $8 , "%s\n\n"}'|sed 's/debug/&\n/g') > temp.out
sed 's/ /\n/g' /share/sqa/dumps/5314577631/checks/temp.out > latest_evtlogs.out
It's not at all elegant, but it finally works.
I have a file that contains strings in this format:
MD5 (TestImages/IMG_0627.JPG) = 6ed611b3e777c5f7b729fa2f2412d656
I am trying to figure out a way to extract the file path, so that I would get a string like this:
TestImages/IMG_0627.JPG
For a different part of my script, I am using this code to remove everything before and after the brackets, and I could of course do something similar, however I'm sure there is a better way?
shortFile=${line#*MD5 }
shortFile=${shortFile%%)*}
Anyone have any suggestions?
You could use sed but that has the overhead of starting a new process.
echo $line | sed -r 's/MD5 \((.*)\).*/\1/'
Just to throw a non-sed answer onto the pile. (Also slightly cheaper since it avoids the pipeline and sub-shell.)
awk -F '[()]' '{print $2}' <<<"$line"
That said the substring expansion option is a reasonable one if it does what you need. (Though it looks like you missed the ( in the first expansion.)
Another way with cut can be :
echo $line|cut -d "(" -f2|cut -d ")" -f1
sed -e 's/^.*(\([^)]*\)).*$/\1/' < infile.txt
I want to check inside a file if it matches a binary pattern.
For that, I'm using clamAV signature database
Trojan.Bancos-166:1:*:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139
I code this to retrieve the hex string representation signature
signature=$(echo "$line" |awk -F':' '{ print $4 }')
Moreover I change hex string to binary
printf -v variable $(sed 's/\(..\)/\\x\1/g;' <<< "$signature")
Until here It works perfectly.
Finally I would like to check if my file ( *$raw_file_path* ) matches my binary pattern (now in $variable)
I try this
test_var=$(grep -qU "$variable" "$raw_file_path")
or
test_var=$(grep -qU --regexp="$variable" "$raw_file_path")
I don't know why it doesn't work, Grep doesn't match anything
.
And sometimes some errors:
grep: Trailing backslash
grep: Invalid regular expression
I know it deals with pattern matching problems.
In my test I don't want use regular expression.
If you have any idea, or other bash tool.
Thanks.
You are currently using the --quiet option for grep by specifying q in -qU. This prevents grep from printing anything to stdout, therefore nothing will be saved to test_var.
Change your code to:
test_var=$(grep -UE "$variable" "$raw_file_path")
First the extra sub-shell can be avoided:
#!/bin/bash
signature="Trojan.Bancos-166:1:*:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139"
variable=$(echo "${signature//*:/}" | sed 's/\(..\)/\\x\1/g;')
Require only confirmation of a match:
if grep -qU "$variable" "$raw_file_path"; then
# matches
fi
Or require the result for further processing:
test_var=$(grep -U "$variable" "$raw_file_path")
# contents of match in test_var
When returning to a variable, greps -q opt suppresses stdout
Edit
Tested working example
> signature="Trojan.Bancos-166:1:All_text before-the last : should be trimed:3d415d736715ab5ee347238cacac61c7123fe35427224d25253c7b035558baf19e54e8d1a82742d6a7b37afc6d91015f751de1102d0a31e66ec33b74034b1ab471cc1381884dfdf0bb3e4233bd075fef235f342302ffd72ecabfa5aedf1b3dc99b3348346db4d9001026aef44c592fee61493f7262ad2bd1bce8a7ce60d81022533f6473ae184935f25cf6cc07c3aebfdf70a5a09139" \
> hex_string=$( echo "${signature//*:/}" | sed 's/\(..\)/\\x\1/g;' ) \
> echo "$hex_string"
\x3d\x41\x5d\x73\x67\x15\xab\x5e\xe3\x47\x23\x8c\xac\xac\x61\xc7\x12\x3f\xe3\x54\x27\x22\x4d\x25\x25\x3c\x7b\x03\x55\x58\xba\xf1\x9e\x54\xe8\xd1\xa8\x27\x42\xd6\xa7\xb3\x7a\xfc\x6d\x91\x01\x5f\x75\x1d\xe1\x10\x2d\x0a\x31\xe6\x6e\xc3\x3b\x74\x03\x4b\x1a\xb4\x71\xcc\x13\x81\x88\x4d\xfd\xf0\xbb\x3e\x42\x33\xbd\x07\x5f\xef\x23\x5f\x34\x23\x02\xff\xd7\x2e\xca\xbf\xa5\xae\xdf\x1b\x3d\xc9\x9b\x33\x48\x34\x6d\xb4\xd9\x00\x10\x26\xae\xf4\x4c\x59\x2f\xee\x61\x49\x3f\x72\x62\xad\x2b\xd1\xbc\xe8\xa7\xce\x60\xd8\x10\x22\x53\x3f\x64\x73\xae\x18\x49\x35\xf2\x5c\xf6\xcc\x07\xc3\xae\xbf\xdf\x70\xa5\xa0\x91\x39