This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I want to ask 2 questions about sed.
For example, when I try to put a string to sed which contains special character like (.\n*.+) and sed cannot run properly.
set a = ".\n*.\+"
set input = ".\n*.\+adsdfasdf"
Then execute:
echo "$input" | sed 's/'$a'/hi/g' # It will give: sed: No match
but
echo "$input" | sed "s#${a}#hi#g" # It will run but not true
My questions are:
What is the difference between these commands: sed 's///' and sed
"s###"
How to treat input just as it is purely string?
1. What is the difference between these commands: sed 's///' and sed "s###"
--- In your case, / or # is a separator of the crucial "sections":
The s command (as in substitute) is probably the most important in sed
and has a lot of different options. The syntax of the s command is
‘s/regexp/replacement/flags’.
...
The / characters may be uniformly replaced by any other single character within any given s command.
2. How to treat input just as it is purely string?
--- To expand/interpolate a variables within sed command, those variables OR the whole expression should be enclosed with double quotes:
set a = ".\n*.\+"
echo "$input" | sed "s/$a/hi/g"
Well, you have a \n in the input string, which you substitute without quote delimiters in 's/'$a'/hi/g' and is parsed as a space, so you pass actually two parameters to sed(1) and you substitute as only one parameter (with the \n included as one character) in only one parameter in "s#$a#hi#g" (in which double quotes include the variable substitution). There is actually no difference in the character used as delimiter in sed(1), but you have called it differently in the two calls.
Finally, I found out that the separator MUST BE different with any characters in the string!
Or it will cause error!
Thus, now I will change the separator to #. It's like comment character.
Related
This question already has answers here:
How to pass a variable containing slashes to sed
(7 answers)
Is it possible to escape regex metacharacters reliably with sed
(4 answers)
Closed 3 months ago.
I have a set of ip addresses declared to a variable IP
IP=["https://127.0.0.1", "https://127.0.0.2", "https://127.0.0.3"]
Need to subtitute the value of IP in some file so i use the sed command
sed -i 's/#hosts: \["https:\/\/localhost:80"\]/hosts: ['$IP']/g' someFile
This errors out with
sed: -e expression #1, char 84: unknown option to `s'
So i tried, which works fine, notice the \ escapes on the IP addresses
sed -i 's/#hosts: \["https:\/\/localhost:80"\]/hosts:["https:\/\/127.0.0.1", "https:\/\/127.0.0.2", "https:\/\/127.0.0.3"]/g'
expected results:
hosts: ["https://127.0.0.1", "https://127.0.0.2", "https://127.0.0.3"]
I can't really influence the value of IP variable because it's gotten from a parameter store with other applications using it. I'm assuming I need to write a function to do the escapes after getting the value from the parameter store? Thanks
The primary issue is you're using sed's default script delimiter of /, which also shows up in the data/strings you're processing, with the net result being that sed can't tell which / are script delimiters vs data.
One solution, as you've figured out, is to escape the / that show up as data.
Another solution is to use a different character (that doesn's show up in the data) as the sed script delimiter.
Addressing a couple other issues with the current code, and using | as the sed script delimiter:
IP='["https://127.0.0.1", "https://127.0.0.2", "https://127.0.0.3"]' # wrap value in single quotes so the whole line is treated as part of the assignment to IP
sed -i "s|#hosts: \[https://localhost:80\]|hosts: $IP|g" someFile # wrap script in double quotes to allow for expansion of $IP
This generates:
$ cat someFile
hosts: ["https://127.0.0.1", "https://127.0.0.2", "https://127.0.0.3"]
This question already has answers here:
What special characters must be escaped in regular expressions?
(13 answers)
Closed 2 years ago.
I want to remove a line from a file and understand this can be done with sed.
The command below fails because of unterminated address regex
sed -i '/$settings['file_temp_path'] = '../tmp';/d' file.php
Having read the the answer at unterminated address regex while using sed . I now understand this is because characters [ and / must be escaped.
Having tried this, the code below is still unsuccessful
sed -i '/$settings\['file_temp_path'] = '..\/tmp';/d' file.php
What is wrong with this? What am I missing?
This might work for you (GNU sed):
sed -i '\#\$settings\['\''file_temp_path'\''\] = '\''\.\./tmp'\'';#d' file
When the regexp/replacement contains the match delimiter (usually /), it can either be escaped/quoted or the delimiters can be altered i.e. /\/tmp/ or \#/tmp#. Note that in the case of the substitution command s/\/tmp/replacement/ can also be s#/tmp#replacement# and leading delimiter does not need to escaped/quoted.
Meta characters i.e. ^,$,[,],*,.,\ and & must be escaped/quoted by \or placed in a character class e.g. . should be \. or [.].
As a rule of thumb, sed commands should be enclosed in single quotes ' and for single quotes to be included in the regexp they should be replaced by '\'' which closes off the existing commands, shell escapes/quotes a ' and reopens the next sed command.
Using double quotes " may also be used but may have unexpected side effects as they are open to shell interpolation.
N.B. If the regexp/substitution delimiter is put inside a character class it does not need to be escaped/quoted i.e. if / is the delimiter then [/] is the same as \/. Also note, that {,},|,? and + should not be escaped/quoted if they are to represent their literal value unless the -E or -r sed command line option is invoked, in which case they should be i.e + represents the plus sign as does \+ when the -E is invoked, whereas \+ and + when the -E or -r is invoked represent one or more of the preceding character/group.
You need to escape several special characters in your pattern, including $.
Example content of file.php:
foo
$settings['file_temp_path'] = '../tmp';
bar
Example code:
$ sed -i "s/\$settings\['file_temp_path'\] = '..\/tmp';//" file.php
$ cat file.php
foo
bar
This question already has answers here:
Replace whole line when match found with sed
(4 answers)
Closed 4 years ago.
How do I find line starts with and replace complete line?
File output:
xyz
abc
/dev/linux-test1/
Code:
output=/dev/sda/windows
sed 's/^/dev/linux*/$output/g' file.txt
I am getting below Error:
sed: -e expression #1, char 9: unknown option to `s'
File Output expected after replacement:
xyz
abc
/dev/sda/windows
Let's take this in small steps.
First we try changing "dev" to "other":
sed 's/dev/other/' file.txt
/other/linux-test1/
(Omitting the other lines.) So far, so good. Now "/dev/" => "/other/":
sed 's//dev///other//' file.txt
sed: 1: "s//dev///other//": bad flag in substitute command: '/'
Ah, it's confused, we're using '/' as both a command delimiter and literal text. So we use a different delimiter, like '|':
sed 's|/dev/|/other/|' file.txt
/other/linux-test1/
Good. Now we try to replace the whole line:
sed 's|^/dev/linux*|/other/|' file.txt
/other/-test1/
It didn't replace the whole line... Ah, in sed, '*' means the previous character repeated any number of times. So we precede it with '.', which means any character:
sed 's|^/dev/linux.*|/other/|' file.txt
/other/
Now to introduce the variable:
sed 's|^/dev/linux.*|$output|' file.txt
$output
The shell didn't expand the variable, because of the single quotes. We change to double quotes:
sed "s|^/dev/linux.*|$output|" file.txt
/dev/sda/windows
This might work for you (GNU sed):
output="/dev/sda/windows"; sed -i '\#/dev/linux.*/#c'"$output" file
Set the shell variable and change the line addressed by /dev/linux.*/ to it.
N.B. The shell variable needs to interpolated hence the ; i.e. the variable may be set on a line on its own. Also the the delimiter for the sed address must be changed so as not to interfere with the address, hence \#...#, and finally the shell variable should be enclosed in double quotes to allow full interpolation.
I'd recommend not doing it this way. Here's why.
Sed is not a programming language. It's a stream editor with some constructs that look and behave like a language, but it offers very little in the way of arbitrary string manipulation, format control, etc.
Sed only takes data from a file or stdin (also a file). Embedding strings within your sed script is asking for errors -- constructs like s/re/$output/ are destined to fail at some point, almost regardless of what workarounds you build into your sed script. The best solutions for making sed commands like this work is to do your input sanitization OUTSIDE of sed.
Which brings me to ... this may be the wrong tool for this job, or might be only one component of the toolset for the job.
The error you're getting is obviously because the sed command you're using is horribly busted. The substitute command is:
s/pattern/replacement/flags
but the command you're running is:
s/^/dev/linux*/$output/g
The pattern you're searching for is ^, the null at the beginning of the line. Your replacement pattern is dev, then you have a bunch of text that might be interpreted as flags. This plainly doesn't work, when your search string contains the same character that you're using as a delimiter to the options for the substitute command.
In regular expressions and in sed, you can escape things. You while you might get some traction with s/^\/dev\/linux.*/$output/, you'd still run into difficulty if $output contained slashes. If you're feeding this script to sed from bash, you could use ${output//\//\\\/}, but you can't handle those escapes within sed itself. Sed has no variables.
In a proper programming language, you'd have better separation of variable content and the commands used for the substitution.
output="/dev/sda/windows"
awk -v output="$output" '$1~/\/dev\/linux/ { $0=output } 1' file.txt
Note that I've used $1 here because in your question, your input lines (and output) appear to have a space at the beginning of each line. Awk automatically trims leading and trailing space when assigning field (positional) variables.
Or you could even do this in pure bash, using no external tools:
output="/dev/sda/windows"
while read -r line; do
[[ "$line" =~ ^/dev/linux ]] && line="$output"
printf '%s\n' "$line"
done < file.txt
This one isn't resilient in the face of leading whitespace. Salt to taste.
So .. yes, you can do this with sed. But the way commands get put together in sed makes something like this risky, and despite the available workarounds like switching your substitution command delimiter to another character, you'd almost certainly be better off using other tools.
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 6 years ago.
I'm trying to replace text in a file with the output of another command. Unfortunately, the outputted text contains characters bash expands. For example, I'm running the following script to change the file (somestring references output that would break the sed command):
#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt
sed fails with this error:
sed: -e expression #1, char 30: unknown option to `s'
I think bash is substuting the contents of $somestring when building the sed command, but is then trying to expand the resulting text. I can't put the entire sed script in single quotes, I need bash to expand it the first time, just not the second. Any suggestions? Thanks
here the forward slash / is the problem. If it's the only issue you can set sed to use a different delimiter.
for example
$ somestring="abc/def"; echo xxx | sed 's/xxx/'"$somestring"'/'
sed: -e expression #1, char 11: unknown option to `s'
$ somestring="abc/def"; echo xxx | sed 's_xxx_'"$somestring"'_'
abc/def
you also need to worry about & and \ chars and escape them if can appear in the replacement text.
If you can't control the the replacement string, either you have to sanitize with another sed script or, alternatively use r command to read it from a file. For example,
$ seq 5 | sed -e '/3/{r replace' -e 'd}'
1
2
3slashes///1ampersand&and2backslashes\\end
4
5
where
$ cat replace
3slashes///1ampersand&and2backslashes\\end
You have several errors here:
the string somestring has characters that are significative for sed command (the most important being '/' that you are using as a delimiter) You can escape it, by substituting it with a previous
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
that will convert your / chars to \/ sequences.
you are using sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt which is looking to substitute the string titl followed by any number of e characters by that $somestring value, followed by a new line and the original one. Unfortunately, sed(1) doesn't allow you to use newline characters in the pattern substitution side of the s command, but you can afford the result by using the i command with a text consisting of you pattern (preceding any new line by a \ to interpret it as literal):
Finally the script leads to:
#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
sed '/^title/i\
'"$somestring\\
" $HOME/example.txt
If your shell is Bash, you can use parameter substitution to replace the problematic /:
somestring="{somestring//\//\\/}"
That looks scary, but is easier to understand if you look at the version that replaces x with __:
somestring="${somestring//x/__}"
It might be easier to use (say) underscore as the delimiter for your sed s command, and then the substitution above would be
somestring="${somestring//_/\\_}"
If you already have backslashes, you'll need to first replace those:
somestring="${somestring//\\/\\\\}"
somestring="{somestring//\//\\/}"
If there were other characters that needed escaping (e.g. on the search side of s///), then you could extend the above appropriately.
This URL provides the cleanest answer:
Command to escape a string in bash
printf "%q" "$someVariable"
will escape any characters you need escaped for you.
How can I fix this:
abc="a/b/c"; echo porc | sed -r "s/^/$abc/"
sed: -e expression #1, char 7: unknown option to `s'
The substitution of variable $abc is done correctly, but the problem is that $abc contains slashes, which confuse sed. Can I somehow escape these slashes?
Note that sed(1) allows you to use different characters for your s/// delimiters:
$ abc="a/b/c"
$ echo porc | sed -r "s|^|$abc|"
a/b/cporc
$
Of course, if you go this route, you need to make sure that the delimiters you choose aren't used elsewhere in your input.
The GNU manual for sed states that "The / characters may be uniformly replaced by any other single character within any given s command."
Therefore, just use another character instead of /, for example ::
abc="a/b/c"; echo porc | sed -r "s:^:$abc:"
Do not use a character that can be found in your input. We can use : above, since we know that the input (a/b/c/) doesn't contain :.
Be careful of character-escaping.
If using "", Bash will interpret some characters specially, e.g. ` (used for inline execution), ! (used for accessing Bash history), $ (used for accessing variables).
If using '', Bash will take all characters literally, even $.
The two approaches can be combined, depending on whether you need escaping or not, e.g.:
abc="a/b/c"; echo porc | sed 's!^!'"$abc"'!'
You don't have to use / as pattern and replace separator, as others already told you. I'd go with : as it is rather rarely used in paths (it's a separator in PATH environment variable). Stick to one and use shell built-in string replace features to make it bullet-proof, e.g. ${abc//:/\\:} (which means replace all : occurrences with \: in ${abc}) in case of : being the separator.
$ abc="a/b/c"; echo porc | sed -r "s:^:${abc//:/\\:}:"
a/b/cporc
backslash:
abc='a\/b\/c'
space filling....
As for the escaping part of the question I had the same issue and resolved with a double sed that can possibly be optimized.
escaped_abc=$(echo $abc | sed "s/\//\\\AAA\//g" | sed "s/AAA//g")
The triple A is used because otherwise the forward slash following its escaping backslash is never placed in the output, no matter how many backslashes you put in front of it.