I have a config file where i need to replace the entire line with new values. It can be either a word or a URL. I am trying to write a script to replace only this particular parameter with new values.
I have tried using grep to extract the line using the parameter and divided the values separately and saved in two different variables.
Now I am trying to replace the whole line with the parameter along with new value or url usind sed
jaram=`grep -i "$a" app.properties`;
param=`grep -i "$a" app.properties |sed 's/\'$a'=*//'`;
sed -e 's~'$jaram'~'$a=''$changed_param'~g' app.properties
The config file contains:
abc1=http://howareyou:scema=olk
abc2=http://howareyou:scema=olk
Here I am trying to replace the url of only abc1. though both have same value. I need to replace the entire url with something different url or a word.
Here I am trying to find the line which contains abc1 and this line url after = is saved in a different variable.
I tried to replace the url with new one using sed:
sed -i 's~'$jaram'~'$a=''$param'~g' app.properties
sed: -e expression #1, char 0: no previous regular expression
Seems like I am doing wrong at some syntax when using sed
trying to replace something like
sed 's/jaram/'{$a=$param}/'
Expecting an output like
abc1=http://jalkek:kj/iuwerj
abc2=http://howareyou:scema=olk
The following script with comments:
# recrete the input config file
cat <<EOF >input
abc1=http://howareyou:scema=olk
abc2=http://howareyou:scema=olk
EOF
# some input variables
name="abc1"
newvalue='http://jalkek:kj/iuwerj'
# the sed script
sed -i 's'$'\01''^'"$name"'=.*'$'\01'"$name"'='"$newvalue"$'\x01' input
# and the output
cat input
produces the following output:
abc1=http://jalkek:kj/iuwerj
abc2=http://howareyou:scema=olk
Notes:
I used the 0x01 byte as the separator for s command inside sed. So it should work with all printable characters.
Remember about quoting. All variables should be inside " double quotes, but all the rest would be best if inside ' single quotes.
I match the name= with ^, so from the beginning of the line. So that for example blablaabc1= doesn't match.
I use ANSI-C Quoting from bash to generate the unredable 0x01 bytes to delimit sed command.
A little more readable version of the sed script could be this one, that just uses " for quoting and uses ~ as a separator:
sed "s~^${name}=.*~${name}=${newvalue}~"
Related
I am new to shell scripting. I am using ksh.
I have this particular line in my script which I use to append text in a variable q to the end of a particular line given by the variable a
containing the line number .
sed -i ''$a's#$#'"$q"'#' test.txt
Now the variable q can contain a large amount of text, with all sorts of special characters, such as !##$%^&*()_+:"<>.,/;'[]= etc etc, no exceptions. For now, I use a couple of sed commands in my script to remove any ' and " in this text (sed "s/'/ /g" | sed 's/"/ /g'), but still when I execute the above command I get the following error
sed: -e expression #1, char 168: unterminated `s' command
Any sed, awk, perl, suggestions are very much appreciated
The difficulty here is to quote (escape) the substitution separator characters # in the sed command:
sed -i ''$a's#$#'"$q"'#' test.txt
For example, if q contains # it will not work. The # will terminate the replacement pattern prematurely. Example: q='a#b', a=2, and the command expands to
sed -i 2s#$#a#b# test.txt
which will not append a#b to the end of line 2, but rather a#.
This can be solved by escaping the # characters in q:
sed -i 2s#$#a\#b# test.txt
However, this escaping could be cumbersome to do in shell.
Another approach is to use another level of indirection. Here is an example of using a Perl one-liner. First q is passed to the script in quoted form. Then, within the script the variable assigned to a new internal variable $q. Using this approach there is no need to escape the substitution separator characters:
perl -pi -E 'BEGIN {$q = shift; $a = shift} s/$/$q/ if $. == $a' "$q" "$a" test.txt
Do not bother trying to sanitize the string. Just put it in a file, and use sed's r command to read it in:
echo "$q" > tmpfile
sed -i -e ${a}rtmpfile test.txt
Ah, but that creates an extra newline that you don't want. You can remove it with:
sed -e ${a}rtmpfile test.txt | awk 'NR=='$a'{printf $0; next}1' > output
Another approach is to use the patch utility if present in your system.
patch test.txt <<-EOF
${a}c
$(sed "${a}q;d" test.txt)$q
.
EOF
${a}c will be replaced with the line number followed by c which means the operation is a change in line ${a}.
The second line is the replacement of the change. This is the concatenated value of the original text and the added text.
The sole . means execute the commands.
How can i replace the following text in a file in linux with a different line
Current :
0 22 * * * /scripts/application_folder_backup.sh >> /var/log/application_folder_backup.log
Replacement line : #line_removed
I tried using sed but my text in the file already has a / which is causing problems. I tried storing the string in a variable too. But it doesn't work
#!/bin/bash
var="0 22 * * * /scripts/application_folder_backup.sh >> /var/log/application_folder_backup.log"
sed -i -e 's/$var/#line_removed/g' /tmp/k1.txt
exit
Just / is not a problem here, even * or all the special regex meta characters will be a problem for sed since it uses only regex for search patterns.
Better to use this non-regex based awk command:
awk -v var="$var" 'index($0, var) { $0 = "#line_removed" } 1' file
#line_removed
index function in awk uses plain text search instead of a regex based search.
I suggest previous to feed the contents of $var to the sed script, to escape all the / chars to be \/, as in:
var_esc=$(echo "$var" | sed 's/\//\\\//g')
But the sed expression gets very complicated (you must use single quotes, if you don't want to double \ chars, as double quotes do interpret also the backslashes.
Another thing that could simplify the expression is to use the possibility of change the regexp delimiter (with using a different char to begin it) as in:
var_esc=$(echo "$var" | sed 's:/:\/:g')
The idea is to, before substituting the $var variable, to escape all the possible interferring chars you can have (you don't know a priory if there are going to be, as you are using a variable for that purpose) and then substitute them in the final command:
sed "s/$var_esc/#line_removed/g"
Finally, if you don't want to substitute the line, but just to erase it, you can do something like this:
sed "/$var_esc/d"
and that will erase all lines that match your $var_esc contents. (this time I believe you cannot change the regexp delimiter, as the / introduces the matching line operator --- d is the command to delete line in the output)
Another way to delete the line in your file is to call ex directly and pass the editor command as input:
ex file <<EOF <-- edit "file" with the commands that follow until EOF is found.
/$var_esc <-- search for first occurrence of $var_esc
d <-- delete line
w <-- write file
EOF <-- eof marker.
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 6 years ago.
I'm trying to replace text in a file with the output of another command. Unfortunately, the outputted text contains characters bash expands. For example, I'm running the following script to change the file (somestring references output that would break the sed command):
#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt
sed fails with this error:
sed: -e expression #1, char 30: unknown option to `s'
I think bash is substuting the contents of $somestring when building the sed command, but is then trying to expand the resulting text. I can't put the entire sed script in single quotes, I need bash to expand it the first time, just not the second. Any suggestions? Thanks
here the forward slash / is the problem. If it's the only issue you can set sed to use a different delimiter.
for example
$ somestring="abc/def"; echo xxx | sed 's/xxx/'"$somestring"'/'
sed: -e expression #1, char 11: unknown option to `s'
$ somestring="abc/def"; echo xxx | sed 's_xxx_'"$somestring"'_'
abc/def
you also need to worry about & and \ chars and escape them if can appear in the replacement text.
If you can't control the the replacement string, either you have to sanitize with another sed script or, alternatively use r command to read it from a file. For example,
$ seq 5 | sed -e '/3/{r replace' -e 'd}'
1
2
3slashes///1ampersand&and2backslashes\\end
4
5
where
$ cat replace
3slashes///1ampersand&and2backslashes\\end
You have several errors here:
the string somestring has characters that are significative for sed command (the most important being '/' that you are using as a delimiter) You can escape it, by substituting it with a previous
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
that will convert your / chars to \/ sequences.
you are using sed '0,/^title/s/^title*/'"$somestring"'\n&/' $HOME/example.txt which is looking to substitute the string titl followed by any number of e characters by that $somestring value, followed by a new line and the original one. Unfortunately, sed(1) doesn't allow you to use newline characters in the pattern substitution side of the s command, but you can afford the result by using the i command with a text consisting of you pattern (preceding any new line by a \ to interpret it as literal):
Finally the script leads to:
#!/bin/bash
somestring='$6$sPnfj/lnXwZVrec7$fCnL9uy1oWIMZduInKTHBAxhsQxGCsBpm2XfVFFqDPHKidrd93yfjbYvKgYexXHVcvkKdu9lbfy16Ek5GvKy/1'
somestring=$(echo "$somestring" | sed -e 's/\//\\\//g')
sed '/^title/i\
'"$somestring\\
" $HOME/example.txt
If your shell is Bash, you can use parameter substitution to replace the problematic /:
somestring="{somestring//\//\\/}"
That looks scary, but is easier to understand if you look at the version that replaces x with __:
somestring="${somestring//x/__}"
It might be easier to use (say) underscore as the delimiter for your sed s command, and then the substitution above would be
somestring="${somestring//_/\\_}"
If you already have backslashes, you'll need to first replace those:
somestring="${somestring//\\/\\\\}"
somestring="{somestring//\//\\/}"
If there were other characters that needed escaping (e.g. on the search side of s///), then you could extend the above appropriately.
This URL provides the cleanest answer:
Command to escape a string in bash
printf "%q" "$someVariable"
will escape any characters you need escaped for you.
I am trying to change the values in a text file using sed in a Bash script with the line,
sed 's/draw($prev_number;n_)/draw($number;n_)/g' file.txt > tmp
This will be in a for loop. Why is it not working?
Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:
# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp
# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command
Further Reading:
Difference between single and double quotes in Bash
Is it possible to escape regex metacharacters reliably with sed
Using different delimiters for sed substitute command
Unless you need it in a different file you can use the -i flag to change the file in place
Variables within single quotes are not expanded, but within double quotes they are. Use double quotes in this case.
sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
You could also make it work with eval, but don’t do that!!
This may help:
sed "s/draw($prev_number;n_)/draw($number;n_)/g"
You can use variables like below. Like here, I wanted to replace hostname i.e., a system variable in the file. I am looking for string look.me and replacing that whole line with look.me=<system_name>
sed -i "s/.*look.me.*/look.me=`hostname`/"
You can also store your system value in another variable and can use that variable for substitution.
host_var=`hostname`
sed -i "s/.*look.me.*/look.me=$host_var/"
Input file:
look.me=demonic
Output of file (assuming my system name is prod-cfm-frontend-1-usa-central-1):
look.me=prod-cfm-frontend-1-usa-central-1
I needed to input github tags from my release within github actions. So that on release it will automatically package up and push code to artifactory.
Here is how I did it. :)
- name: Invoke build
run: |
# Gets the Tag number from the release
TAGNUMBER=$(echo $GITHUB_REF | cut -d / -f 3)
# Setups a string to be used by sed
FINDANDREPLACE='s/${GITHUBACTIONSTAG}/'$(echo $TAGNUMBER)/
# Updates the setup.cfg file within version number
sed -i $FINDANDREPLACE setup.cfg
# Installs prerequisites and pushes
pip install -r requirements-dev.txt
invoke build
Retrospectively I wish I did this in python with tests. However it was fun todo some bash.
Another variant, using printf:
SED_EXPR="$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)"
sed "${SED_EXPR}" file.txt
or in one line:
sed "$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)" file.txt
Using printf to build the replacement expression should be safe against all kinds of weird things, which is why I like this variant.
With the help of SO I have been able to correct my sed command so I can use variables (was using ' instead of " and wasn't using g for global).
However, I am struggling to get the command to work correctly.
To give you some context, I have a file containing numerous lines of text and some line contain one or more tags. these tags are in the following format:
[#$key#] - i.e #[#] is used to indicate the presence of a key
Then I have an internal array object which stores a string value which contains each key and a corresponding value in the following format:
$key=$value
What I am trying to do is to cat my file containing the tags at the same time as using a global sed replace to swap the $key for the corresponding value within the array object.
The problem I have is that my grep/sed command is not replacing any of the text and I can't figure out why.
here is my code:
for x in "${prop[#]}"
do
key=`echo "${x}" | cut -d '=' -f 1`
value=`echo "${x}" | cut -d '=' -f 2`
# global replace on the $MY_FILE
cat $MY_FILE | sed "s|\[#${key}#\]|${value}|g" > ${TEMP_MY_FILE}
cat $TEMP_MY_FILE > $MY_FILE
done
I thought about using sed -i, but my version doesn't seem to support-i
Although [ doesn't have any special meaning in a double-quoted bash string, \[ still evaluates to [ since backslashes are processed to allow for escaping dollar signs. Try
sed "s|\\[#${key}#\\]|${value}|g"
as your sed command. The double backslash will causes a literal backslash to be sent to sed, which will use it to escape the [ to treat it literally as well.