Found this error in other questions, but I can't see how the solutions relate to this.
Assume a file test containing:
one
twoX
three
I can correct twoX with:
perl -0777 -i -pe 's/twoX/two/igm' test
I can make a function to do this:
replace_str(){ perl -0777 -i -pe 's/'$2'/'$3'/igm' $1; }
replace_str test twoX two
But this fails when the replacement contains a space (possibly other chars):
replace_str test two 'two frogs'
Substitution replacement not terminated at -e line 1.
The perl line works with the space. Why not when called in a function? I've tried with other quotes and e.g. $(echo two frogs) (with and without quotes).
It's because you end the string you pass as argument to Perl for your variable expansions. That makes the regex become two arguments.
Instead just put the whole regex, including variables, inside double-quotes and the shell should expand the variables properly.
So use "s/$2/$3/igm" instead.
Related
I'm trying to replace a string VCC_TGL to $G_CORNER_SQL_DETAILS($corner+vcc_tgl)
I have tried using the perl one liner, but it is missing the $ character.
my $i="VCC_TGL";
my $test="\$G_CORNER_SQL_DETAILS(\$corner+vcc_tgl)";
print "replace $i with $test\n";
`perl -pi.back -e 's/$i/$test/' configure.tcl`;
The output im getting on executing the above perl script is ,
set vccio G_CORNER_SQL_DETAILS(corner+vcc_tgl)
You are passing a string containing special shell characters, e.g. $, through shell parsing, i.e. they are processed by the shell and therefore get lost:
$ perl -e '$test="XXX\$YYY"; qx/set -x; echo $test/'
+ echo XXX
Back-ticks, qx// and system("...") are prone to such problems and are unsafe in general.
You should instead use the safe array version of system(), i.e. bypass the shell completely so that parameters are passed into the called command as-is:
$ perl -e '$i="AAA"; $test="XXX\$YYY"; system(qw(echo perl -pi.back -e), "s/$i/$test/", qw(configure.tcl));'
perl -pi.back -e s/AAA/XXX$YYY/ configure.tcl
What I want to do is that i want to remove all prepositions in a text file in CentOS. Things like 'on of to the in at ....'. Here is my script:
!/bin/bash
list='i me my myself we our ours ourselves you your yours yourself ..... '
cat Hamlet.txt | for item in $list
do
sed 's/$item//g'
done > newHam.txt
but at the end when i open newHam.txt nothing changes! It's the same as Ham.txt. I don't know whether this is a good approach or not. Any suggestion? Any approach??
Assuming your sed understands \< and \> for word boundaries,
sed 's/\<\(i\|me\|my\|myself|\we|\our|\ours|\ourselves|\you|\your|\yours|\yourself\)\> \?//g' Hamlet.txt >newHam.txt
You want to make sure you include word boundaries; your original attempt would replace e.g. i everywhere n the nput.
If you already have the words in a string, you can interpolate it in Bash with
sed "s/\\<\\(${list// /\\|}\\)\\> \\?//g" Hamlet.txt >newHam.txt
but the ${variable//pattern/substitution} parameter expansion is not portable to e.g. /bin/sh. Notice also how double quotes instead of single are necessary for the shell to be allowed to perform variable substitutions within the script, and how all literal backslashes need to be escaped with another backslash within double quotes.
Unfortunately, many details of sed are poorly standardized. Ironically, switching to a tool which isn't standard at all might be the most portable solution.
perl -pe 'BEGIN {
#list = qw(i me my myself we our ours ourselves you your yours yourself .....);
$re = join("|", #list); }
s/\b($re)\b ?//go' Hamlet.txt >newHam.txt
If you want this as a standalone script,
#!/usr/bin/perl
BEGIN {
#list = qw(i me my myself we our ours ourselves you your yours yourself .....);
$re = join("|", #list);
}
while (<>) {
s/\b($re)\b ?//go;
print
}
These words are pronouns, not prepositions.
Finally, take care to fix the shebang of your script; the first line of the script needs to start with exactly the two characters #! because that's what makes it a shebang. You'll also want to avoid the useless cat in the future.
This question already has answers here:
Replace whole line when match found with sed
(4 answers)
Closed 4 years ago.
How do I find line starts with and replace complete line?
File output:
xyz
abc
/dev/linux-test1/
Code:
output=/dev/sda/windows
sed 's/^/dev/linux*/$output/g' file.txt
I am getting below Error:
sed: -e expression #1, char 9: unknown option to `s'
File Output expected after replacement:
xyz
abc
/dev/sda/windows
Let's take this in small steps.
First we try changing "dev" to "other":
sed 's/dev/other/' file.txt
/other/linux-test1/
(Omitting the other lines.) So far, so good. Now "/dev/" => "/other/":
sed 's//dev///other//' file.txt
sed: 1: "s//dev///other//": bad flag in substitute command: '/'
Ah, it's confused, we're using '/' as both a command delimiter and literal text. So we use a different delimiter, like '|':
sed 's|/dev/|/other/|' file.txt
/other/linux-test1/
Good. Now we try to replace the whole line:
sed 's|^/dev/linux*|/other/|' file.txt
/other/-test1/
It didn't replace the whole line... Ah, in sed, '*' means the previous character repeated any number of times. So we precede it with '.', which means any character:
sed 's|^/dev/linux.*|/other/|' file.txt
/other/
Now to introduce the variable:
sed 's|^/dev/linux.*|$output|' file.txt
$output
The shell didn't expand the variable, because of the single quotes. We change to double quotes:
sed "s|^/dev/linux.*|$output|" file.txt
/dev/sda/windows
This might work for you (GNU sed):
output="/dev/sda/windows"; sed -i '\#/dev/linux.*/#c'"$output" file
Set the shell variable and change the line addressed by /dev/linux.*/ to it.
N.B. The shell variable needs to interpolated hence the ; i.e. the variable may be set on a line on its own. Also the the delimiter for the sed address must be changed so as not to interfere with the address, hence \#...#, and finally the shell variable should be enclosed in double quotes to allow full interpolation.
I'd recommend not doing it this way. Here's why.
Sed is not a programming language. It's a stream editor with some constructs that look and behave like a language, but it offers very little in the way of arbitrary string manipulation, format control, etc.
Sed only takes data from a file or stdin (also a file). Embedding strings within your sed script is asking for errors -- constructs like s/re/$output/ are destined to fail at some point, almost regardless of what workarounds you build into your sed script. The best solutions for making sed commands like this work is to do your input sanitization OUTSIDE of sed.
Which brings me to ... this may be the wrong tool for this job, or might be only one component of the toolset for the job.
The error you're getting is obviously because the sed command you're using is horribly busted. The substitute command is:
s/pattern/replacement/flags
but the command you're running is:
s/^/dev/linux*/$output/g
The pattern you're searching for is ^, the null at the beginning of the line. Your replacement pattern is dev, then you have a bunch of text that might be interpreted as flags. This plainly doesn't work, when your search string contains the same character that you're using as a delimiter to the options for the substitute command.
In regular expressions and in sed, you can escape things. You while you might get some traction with s/^\/dev\/linux.*/$output/, you'd still run into difficulty if $output contained slashes. If you're feeding this script to sed from bash, you could use ${output//\//\\\/}, but you can't handle those escapes within sed itself. Sed has no variables.
In a proper programming language, you'd have better separation of variable content and the commands used for the substitution.
output="/dev/sda/windows"
awk -v output="$output" '$1~/\/dev\/linux/ { $0=output } 1' file.txt
Note that I've used $1 here because in your question, your input lines (and output) appear to have a space at the beginning of each line. Awk automatically trims leading and trailing space when assigning field (positional) variables.
Or you could even do this in pure bash, using no external tools:
output="/dev/sda/windows"
while read -r line; do
[[ "$line" =~ ^/dev/linux ]] && line="$output"
printf '%s\n' "$line"
done < file.txt
This one isn't resilient in the face of leading whitespace. Salt to taste.
So .. yes, you can do this with sed. But the way commands get put together in sed makes something like this risky, and despite the available workarounds like switching your substitution command delimiter to another character, you'd almost certainly be better off using other tools.
I'm new to shell scripting. My requirement is to retrieve lines between two pattern, its working fine if I run it from the terminal without using variables inside sed cmd. But the problem arises when I put all those below cmd in a file and tried to execute it.
#!/bin/sh
word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
fileC=`cat test.log`
output=`echo $fileC | sed -e "n/\$word/$upto/p"`
printf '%s\n' "$output"
If I use the below cmd in the terminal it works fine
sed -n '/ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34/,/2017-01-03 23:00/ p' test.log
Please suggest a workaround.
If we put aside for a moment the fact you shouldn't cat a file to a variable and then echo it for sed filtering, the reason why your command is not working is because you're not quoting the file content variable, fileC when echoing. This will munge together multiple whitespace characters and turn them into a single space. So, you're losing newlines from the file, as well as multiple spaces, tabs, etc.
To fix it, you can write:
fileC=$(cat test.log)
output=$(echo "$fileC" | sed -n "/$word/,/$upto/p")
Note the double-quotes around fileC (and a fixed sed expression, similar to your second example). Without the quotes (try echo $fileC), your fileC is expanded (with the default IFS) into a series of words, each being one argument to echo, and echo will just print those words separated with a single space. Additionally, if the file contains some of the globbing characters (like *), those patterns are also expanded. This is a common bash pitfall.
Much better would be to write it like this:
output=$(sed -n "/$word/,/$upto/p" test.log)
And if your patterns include some of the sed metacharacters, you should really escape them before using with sed, like this:
escape() {
sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$1";
}
output=$(sed -n "/$(escape "$word")/,/$(escape "$upto")/ p" test.log)
The correct approach will be something like:
word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
awk -v beg="$word" -v end="$upto" '$0==beg{f=1} f{print; if ($0==end) exit}' file
but until we see your sample input and output we can't know for sure what it is you need to match on (full lines, partial lines, all text on one line, etc.) or what you want to print (include delimiters, exclude one, exclude both, etc.).
I am using sed -e "s/foo/$bar/" -e "s/some/$text/" file.whatever to replace a phrase in a certain file. The problem is that the $bar string contains multiple special characters like /. So when I try to replace something in a text file using the following code...
#!/bin/bash
bar="http://stackoverflow.com/"
sed -e "s/foo/$bar/" -e "s/some/$text/ file.whatever
...then I get an error saying : sed: unknown option to s is there anything I can do about it?
You can use any delimiter. s#some#SOME# for example. Another good delimiter is vertical-bar. Other chars can work but have special significance for some contexts such as regular expressions.
You can get this difficulty in sed regardless of what delimiters you use, especially if you don't know what the string contains. I'd pick a different method for passing the shell variables into the helper interpreter.
awk -v rep1="$bar" -v rep2="$text" '{sub(/foo/, rep1); sub(/some/, rep2); print}'
or
perl -spe 's/foo/$rep1/; s/some/$rep2/' -- -rep1="$bar" -rep2="$text"
Correctness trumps brevity in this case.
(reference for Perl example)