Using a variable to replace lines in a file with backslashes - linux

I want to add the string %%% to the beginning of some specific lines in a text file.
This is my script:
#!/bin/bash
a="c:\Temp"
sed "s/$a/%%%$a/g" <File.txt
And this is my File.txt content:
d:\Temp
c:\Temp
e:\Temp
But nothing changes when I execute it.
I think the 'sed' command is not finding the pattern, possibly due to the \ backslashes in the variable a.
I can find the c:\Temp line if I use grep with -F option (to not interpret strings):
cat File.txt | grep -F "$a"
But sed seems not to implement such '-F` option.
Not working neither:
sed 's/$a/%%%$a/g' <File.txt
sed 's/"$a"/%%%"$a"/g' <File.txt
I have found similar threads about replacing with sed, but they don't refer to variables.
How can I replace the desired lines by using a variable adding them the %%% char string?
EDIT: It would be fine that the $a variable could be entered via parameter when calling the script, so it will be assigned like:
a=$1

Try it like this:
#!/bin/sh
a='c:\\Temp' # single quotes
sed "s/$a/%%%$a/g" <File.txt # double quotes
Output:
Johns-MacBook-Pro:sed jcreasey$ sh x.sh
d:\Temp
e:\Temp
%%%c:\Temp
You need the double slash '\' to escape the '\'.
The single quotes won't expand the variables.
So you escape the slash in single quotes and pass it into the double quotes.
Of course you could also just do this:
#!/bin/sh
sed 's/\(.*Temp\)/%%%&/' <File.txt
If you want to get input from the command line you have to allow for the fact that \ is an escape character there too. So the user needs to type 'c:\\' or the interpreter will just wait for another character. Then once you get it, you will need to escape it again. (printf %q).
#!/bin/sh
b=`printf "%q" $1`
sed "s/\($b\)/%%% &/" < File.txt

The issue you are having has to do with substitution of your variable providing a regular expression looking for a literal c:Temp with the \ interpreted as an escape by the shell. There are a number of workarounds. Seeing the comments and having worked through the possibilities, the following will allow the unquoted entry of the search term:
#!/bin/bash
## validate that needed input is given on the command line
[ -n "$1" -a "$2" ] || {
printf "Error: insufficient input. Usage: %s <term> <file>\n" "${0//*\//}" >&2
exit 1
}
## validate that the filename given is readable
[ -r "$2" ] || {
printf "Error: file not readable '%s'\n" "$2" >&2
exit 1
}
a="$1" # assign a
filenm="$2" # assign filename
## test and fix the search term entered
[[ "$a" =~ '/' ]] || a="${a/:/:\\}" # test if \ removed by shell, if so replace
a="${a/\\/\\\\}" # add second \
sed -e "s/$a/%%%$a/g" "$filenm" # call sed with output to stdout
Usage:
$ bash sedwinpath.sh c:\Temp dat/winpath.txt
d:\Temp
%%%c:\Temp
e:\Temp
Note: This allows both single-quoted or unquoted entry of the dos path search term. To edit in place use sed -i. Additionally, the [[ operator and =~ operator are limited to bash.
I could have sworn the original question said replace, but to append, just as you suggest in the comments. I have updated the code with:
sed -e "s/$a/%%%$a/g" "$filenm"
Which provides the new output:
$ bash sedwinpath.sh c:\Temp dat/winpath.txt
d:\Temp
%%%c:\Temp
e:\Temp
Remember: If you want to edit the file in place use sed -i or sed -i.bak which will edit the actual file (and if -i.bak is given create a backup of the original in originalname.bak). Let me know if that is not what you intended and I'm happy to edit again.

Creating your script with a positional parameter of $1
#!/bin/bash
a="$1"
cat <file path>|sed "s/"$1"/%%%"$1"/g" > "temporary file"
Now whenever you want sed to find "c:\Temp" you need to use your script command line as follows
bash <my executing script> c:\\\\Temp
The first backslash will make bash interpret any backslashes that follows therefore what will be save in variable "a" in your executing script is "c:\\Temp". Now substituting this variable in sed will cause sed to interpret 1 backlash since the first backslash in this variable will cause sed to start interpreting the other backlash.
when you Open your temporary file you will see:
d:\Temp
%%%c:\Temp
e:\Temp

Related

Replacing placeholders with SED

I'm trying to replace [[****]] placeholders in a html file with values from property file.
Sample content in the input file:
<html>
<host>[[my_host]]</host>
<port>[[my_port]]</port>
</html>
Sample content in property file:
my_host=linkcmb.com
my_port=8080
My current script:
#/bin/sh
property_file=$1
input_html=$2
output_html=$3
IFS="="
while read k v || [[ -n "$k" ]]; do
test -z "$k" && continue
declare $k=$v
done <"$property_file"
eval "$(sed 's/\[\[\([^]]\+\)\]\]/${\1}/g' $input_html) >$output_html";
Error: Html tags are getting evaluated too causing errors.
./some.sh: line 32: html: No such file or directory
./some.sh: line 33: host: No such file or directory
./some.sh: line 35: /host: No such file or directory
....
....
Any advises will be much appreciated. Thanks.
You can replace your while loop with
. "$property_file"
However, I do not like the eval, and you do not need to declare these settings.
You want sed commands like
sed '/=/ s/\([^=]*\)=\(.*\)/s#\\\[\\\[\1\\\]\\\]#\2#g/' "$property_file"
A lot of backslashes, the [[]] were a difficult choice.
You can use these commands using process substitution:
sed -f <(
sed '/=/ s/\([^=]*\)=\(.*\)/s#\\\[\\\[\1\\\]\\\]#\2#g/' "$property_file"
) "${input_html}"
There is a regex option which is missing from sed, but it is present in perl. If you can use perl, anything between \Q and \E is escaped and taken literally.
The script needs to change to create a temporary perl file that contains all the substitution commands. It would be something like this:
#/bin/sh
property_file=$1
input_html=$2
output_html=$3
perlfile="$$.perl"
IFS="="
while read k v || [[ -n "$k" ]]; do
test -z "$k" && continue
echo "s/\Q[[${k}]]\E/${v}/g;" >> $perlfile
done <"$property_file"
perl -p $perlfile $input_html >$output_html
rm $perlfile
EDIT:
If one of your properties contains a slash (e.g., a path name), you may escape it directly in the properties file:
# input
<path>[[my_path]]</path>
# properties
mypath=dir\\/filename
# output
<path>dir/filename</path>
Same thing with backslash:
# input
<path>[[my_path]]</path>
# properties
mypath=dir\\\\filename
# output
<path>dir\filename</path>
Otherwise, you may need to add logic to the script to do so.

Bash Script - Nested $(..) Commands - Not working correctly

I was trying to do these few operations/commands on a single line and assign it to a variable. I have it working about 90% of the way except for one part of it.
I was unaware you could do this, but I read that you can nest $(..) inside other $(..).... So I was trying to do that to get this working, but can't seem to get it the rest of the way.
So basically, what I want to do is:
1. Take the output of a file and assign it to a variable
2. Then pre-pend some text to the start of that output
3. Then append some text to the end of the output
4. And finally remove newlines and replace them with "\n" character...
I can do this just fine in multiple steps but I would like to try and get this working this way.
So far I have tried the following:
My 1st attempt, before reading about nested $(..):
MY_VAR=$(echo -n "<pre style=\"display:inline;\">"; cat input.txt | sed ':a;N;$!ba;s/\n/\\n/g'; echo -n "</pre>")
This one worked 99% of the way except there was a newline being added between the cat command's output and the last echo command. I'm guessing this is from the cat command since sed removed all newlines except for that one, maybe...?
Other tries:
MY_VAR=$( $(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo -n "</pre>") | sed ':a;N;$!ba;s/\n/\\n/g')
MY_VAR="$( echo $(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo "</pre>") | sed ':a;N;$!ba;s/\n/\\n/g' )"
MY_VAR="$( echo "$(echo -n "<pre style=\"display:inline;\">"; cat input.txt; echo "</pre>")" | sed ':a;N;$!ba;s/\n/\\n/g' )"
*Most these others were tried with and without the extra double-quotes surrounding the different $(..) parts...
I had a few other attempts, but they didn't have any luck either... On a few of the other attempts above, it seemed to work except sed was NOT inserting the replacement part of it. The output was correct for the most part, except instead of seeing "\n" between lines it just showed each of the lines smashed together into one line without anything to separate them...
I'm thinking there is something small I am missing here if anyone has any idea..?
*P.S. Does Bash have a name for the $(..) structure? It's hard trying to Google for that since it doesn't really search symbols...
You have no need to nest command substitutions here.
your_var='<pre style="display:inline;">'"$(<input.txt)"'</pre>'
your_var=${your_var//$'\n'/'\n'}
"$(<input.txt)" expands to the contents of input.txt, but without any trailing newline. (Command substitution always strips trailing newlines; printf '%s' "$(cat ...)" has the same effect, albeit less efficiently as it requires a subshell, whereas cat ... alone does not).
${foo//bar/baz} expands to the contents of the shell variable named foo, with all instances of bar replaced with baz.
$'\n' is bash syntax for a literal newline.
'\n' is bash syntax for a two-character string, beginning with a backslash.
Thus, tying all this together, it first generates a single string with the prefix, the contents of the file, and the suffix; then replaces literal newlines inside that combined string with '\n' two-character sequences.
Granted, this is multiple lines as implemented above -- but it's also much faster and more efficient than anything involving a command substitution.
However, if you really want a single, nested command substitution, you can do that:
your_var=$(printf '%s' '<pre style="display:inline;">' \
"$(sed '$ ! s/$/\\n/g' <input.txt | tr -d '\n')" \
'</pre>')
The printf %s combines its arguments without any delimiter between them
The sed operation adds a literal \n to the end of each line except the last
The tr -d '\n' operation removes literal newlines from the file
However, even this approach could be done more efficiently without the nesting:
printf -v your_var '%s' '<pre style="display:inline;">' \
"$(sed '$ ! s/$/\\n/g' <input.txt | tr -d '\n')" \
'</pre>')
...which has the printf assign its results directly to your_var, without any outer command substitution required (and thus saving the expense of the outer subshell).

bash script to replace all occurrences of placeholders in file

I'm trying to write a bash script to replace all occurrences of a placeholder in a file with an environment variable of the same name. As an example, if I have a file like the following...
This is an {{VAR1}} {{VAR2}}.
It should work across multiple lines in this {{VAR2}}.
... and I have the following environment variables set:
VAR1='example'
VAR2='file'
after running the script on my file, I should get the output:
This is an example file.
It should work across multiple lines in this file.
I'm sure there must be a solution using awk/sed, but so far the closest I've come can't handle if there's more than one variable on a line. Here's my attempt so far:
cat example.txt | grep -o '{{.*}}' > temp
while read placeholder; do
varName=$(echo "$placeholder" | tr -d '{}')
value="${!varName}"
sed -i "s/$placeholder/$value/g" "$file"
done < temp
rm -rf temp
I'd use Perl:
perl -pe 's/{{(.*?)}}/$ENV{$1}/g' filename
This assumes that VAR1 and VAR2 are environment variables (i.e., are exported), so that Perl can pick them out of its environment. This would be required of any approach that isn't pure shell; I just mention it to avoid confusion.
This works as follows:
s/pattern/replacement/g is a substitution command; you may recognize it from sed. The difference is that here we can use Perl's more powerful regex engine and variables. The g flag makes it so that all matches are replaced; without it, it would apply only to the first.
In the pattern, .*? matches non-greedily, so that in a line that contains foo {{VAR1}} bar {{VAR2}} baz, the pattern {{.*?}} matches only {{VAR1}} instead of {{VAR1}} bar {{VAR2}}.
The part between {{ and }} is captured because it is between () and can be reused as $1
$ENV{$1} in the replacement uses the special %ENV hash that contains the environment of the Perl process. $ENV{$1} is the value of the environment variable that has the name $1, which is the captured group from before.
Only bash and sed:
$ VAR1='example'
$ VAR2='file'
$ export VAR1 VAR2
$ sed -e '{s/{{\([^{]*\)}}/${\1}/g; s/^/echo "/; s/$/";/}' -e e filename
This is an example file.
It should work across multiple lines in this file.
sed -e '{s/{{\([^{]*\)}}/${\1}/g;}' filename:
This is an ${VAR1} ${VAR2}.
It should work across multiple lines in this ${VAR2}.
{{\([^{]*\)}} - Search for {{..}}
[^{] - Non greedy match
\1 - Access to the bracketed values \(...\).
sed -e '{s/{{\([^{]*\)}}/${\1}/g; s/^/echo "/; s/$/";/}' filename:
echo "This is an ${VAR1} ${VAR2}.";
echo "It should work across multiple lines in this ${VAR2}.";
s/^/echo "/ - Replace the beginning of the line with echo "
s/$/";/ - Replace the end of the line with ";
I was just playing with your original approach. Wouldn't adding another loop on $varName work?
cat example.txt | grep -o '{{.*}}' > temp
while read placeholder; do
varName=$(echo "$placeholder" | tr -d '{}')
for i in $varName; do
value="${!i}"
sed -i "s/{{$i}}/$value/g" example.txt
done
done < temp
rm -rf temp

why sed not working properly in script

Hi I have following line in shellscript
sed 's_$org_$repl_g' $i > $temp_file
In this $org denotes the name to be change and $repl denotes replacement. I have done echo for both and both are write. $i represent the file name. when I echo below
echo $(sed "s/$org/$repl_g" $i)
Then also it does not replace the word . while when I try this with terminal directly as below
sed 's_Dilip_Agarwal_g' test.txt
then it give correct output by replacing the original one.
Can any body help me where I am wrong in this.
Thanks
Do not use command substitution on it. And use double quotes instead of single quotes. Single quotes do not expand parameters. You also have to fix your parameter expansion. _ is also a valid parameter character so you need to use braces to isolate the only characters that would identify the parameters.
sed "s_${org}_${repl}_g" "$i" > "$temp_file"
You can also just use another delimeter that's not a parameter character:
sed "s|$org|$repl|g" "$i" > "$temp_file"

Environment variable substitution in sed

If I run these commands from a script:
#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla
it is fine.
But, if I run:
#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s'
I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.
How can I get sed to recognize a $var as an environment variable as it is defined in the shell?
Your two examples look identical, which makes problems hard to diagnose. Potential problems:
You may need double quotes, as in sed 's/xxx/'"$PWD"'/'
$PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.
To nail both issues at once, perhaps
sed 's#xxx#'"$PWD"'#'
In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).
So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.
sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"
In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.
Another easy alternative:
Since $PWD will usually contain a slash /, use | instead of / for the sed statement:
sed -e "s|xxx|$PWD|"
You can use other characters besides "/" in substitution:
sed "s#$1#$2#g" -i FILE
一. bad way: change delimiter
sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's#xxx#'"$PWD"'#'
maybe those not the final answer,
you can not known what character will occur in $PWD, / : OR #.
if delimiter char in $PWD, they will break the expression
the good way is replace(escape) the special character in $PWD.
二. good way: escape delimiter
for example:
try to replace URL as $url (has : / in content)
x.com:80/aa/bb/aa.js
in string $tmp
URL
A. use / as delimiter
escape / as \/ in var (before use in sed expression)
## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js #escape fine
echo ${url//\//\/}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
URL
echo $tmp | sed "s/URL/${url//\//\/}/"
URL
OR
B. use : as delimiter (more readable than /)
escape : as \: in var (before use in sed expression)
## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
x.com:80/aa/bb/aa.js
With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:
sed 's/xxx/'$PWD'/'
will be expanded to
sed `s/xxx//home/yourname//
which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.
Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.
Dealing with VARIABLES within sed
[root#gislab00207 ldom]# echo domainname: None > /tmp/1.txt
[root#gislab00207 ldom]# cat /tmp/1.txt
domainname: None
[root#gislab00207 ldom]# echo ${DOMAIN_NAME}
dcsw-79-98vm.us.oracle.com
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'
--- Below is the result -- very funny.
domainname: ${DOMAIN_NAME}
--- You need to single quote your variable like this ...
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'
--- The right result is below
domainname: dcsw-79-98vm.us.oracle.com
VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1'
where VAR contains what you want to replace the field with
I had similar problem, I had a list and I have to build a SQL script based on template (that contained #INPUT# as element to replace):
for i in LIST
do
awk "sub(/\#INPUT\#/,\"${i}\");" template.sql >> output
done
If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:
PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected
for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:
sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt
BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working.
THEN use different delimiter:
sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt

Resources