I need to replace -Xmx1024m (changing 1024 to 2048) in standalone.conf from a shell command.
I try do this with sed:
echo $(sed 's/1024/2048/g' standalone.conf) > standalone.conf.
The code works, but the text saved loses tabs and newlines.
Passing an expansion to echo unquoted subjects it to string-splitting and glob-expansion, passing every individual word produced by those processes as a separate argument to echo (which echo then combines with a single space between each).
Consider instead:
sed 's/1024/2048/g' <standalone.conf >standalone.conf.new && mv standalone.conf{.new,}
...and, in general, always use echo "$foo" or instead of echo $foo -- it's the lack of quotes here which was most immediately responsible for your bug.
Do not use echo, but let sed change the file:
sed -i 's/1024/2048/g' standalone.conf
sed 's/1024/2048/g' will translate 1024 to 2048 globally, throughout the file, which might be unwise. It would be much better to limit the translation somehow. If your goal is in fact the one you stated, namely changing -Xmx1024m to -Xmx2048m, then the following would at least be a reasonable start (assuming your sed supports the -i option):
sed -i -e 's/-Xmx1024m/-Xmx2048m/' standalone.conf
(If your sed does not support -i, then make the obvious changes.)
If the timestamp of the file is useful, and if you only want to update the file if it needs updating, then consider:
grep -q -e -Xmx1024m standalone.conf && sed -i -e 's/-Xmx1024m/-Xmx2048m/' standalone.conf
Related
I have two sed command which includes in my cook.sh script. One command is
sed -E -i "s/^(\\\$mainDomain=\")[^\"]+(\";)$/\1$MainDomain\2/" /var/config.php
This is working fine.
But the below command which is almost same. But it is not working.
sed -E -i "s/^(\\\$authURI=\")[^\"]+(\";)$/\1$duo_auth\2/" /var/config.php
That give the below error message
sed: -e expression #1, char 36: unknown option to `s'
Any idea on this ?
The issue is likely due to your replacement variable $duo_auth having a un-escaped /, change the default sed separator from / to ~ as
sed -E -i "s~^(\\\$authURI=\")[^\"]+(\";)$~\1$duo_auth\2~" /var/config.php
Try it without -i for seeing if the replacement is as expected and put it back after confirmation.
Example:-
cat /var/config.php
<?php
$authURI="dev.digin.io";
now setting the variable
duo_auth="http://auth.uri.digin.io:3048/"
Now the replacement, without -i
sed -E "s~^(\\\$authURI=\")[^\"]+(\";)$~\1$duo_auth\2~" /var/config.php
<?php
$authURI="http://auth.uri.digin.io:3048/";
The problem is probably due to $duo_auth containing an unescaped /. This means that the sed editing script will have a syntax error.
You may pick any other character to use as a delimiter in the s/.../.../ command, for example #:
sed "s#....#....#"
Just make sure that you pick a character that is not ever going to turn up in either $duo_auth or $authURI.
While testing, I'd recommend that you avoid using in-place-editing (-i) with sed. Also, the -i switch is horribly non-portable between sed implementations (some requires an argument).
Instead, do the slightly more cumbersome
sed -e "s#...#...#" data.in >data.in.tmp && mv -f data.in.tmp data.in
While testing, check the data.in.tmp file before moving it.
What's the simplest way to do a find and replace for a given input string, say abc, and replace with another string, say XYZ in file /tmp/file.txt?
I am writting an app and using IronPython to execute commands through SSH — but I don't know Unix that well and don't know what to look for.
I have heard that Bash, apart from being a command line interface, can be a very powerful scripting language. So, if this is true, I assume you can perform actions like these.
Can I do it with bash, and what's the simplest (one line) script to achieve my goal?
The easiest way is to use sed (or perl):
sed -i -e 's/abc/XYZ/g' /tmp/file.txt
Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.
If you really really want to use just bash, then the following can work:
while IFS='' read -r a; do
echo "${a//abc/XYZ}"
done < /tmp/file.txt > /tmp/file.txt.t
mv /tmp/file.txt{.t,}
This loops over each line, doing a substitution, and writing to a temporary file (don't want to clobber the input). The move at the end just moves temporary to the original name. (For robustness and security, the temporary file name should not be static or predictable, but let's not go there.)
For Mac users:
sed -i '' 's/abc/XYZ/g' /tmp/file.txt
(See the comment below why)
File manipulation isn't normally done by Bash, but by programs invoked by Bash, e.g.:
perl -pi -e 's/abc/XYZ/g' /tmp/file.txt
The -i flag tells it to do an in-place replacement.
See man perlrun for more details, including how to take a backup of the original file.
I was surprised when I stumbled over this...
There is a replace command which ships with the "mysql-server" package, so if you have installed it try it out:
# replace string abc to XYZ in files
replace "abc" "XYZ" -- file.txt file2.txt file3.txt
# or pipe an echo to replace
echo "abcdef" |replace "abc" "XYZ"
See man replace for more on this.
This is an old post but for anyone wanting to use variables as #centurian said the single quotes mean nothing will be expanded.
A simple way to get variables in is to do string concatenation since this is done by juxtaposition in bash the following should work:
sed -i -e "s/$var1/$var2/g" /tmp/file.txt
Bash, like other shells, is just a tool for coordinating other commands. Typically you would try to use standard UNIX commands, but you can of course use Bash to invoke anything, including your own compiled programs, other shell scripts, Python and Perl scripts etc.
In this case, there are a couple of ways to do it.
If you want to read a file, and write it to another file, doing search/replace as you go, use sed:
sed 's/abc/XYZ/g' <infile >outfile
If you want to edit the file in place (as if opening the file in an editor, editing it, then saving it) supply instructions to the line editor 'ex'
echo "%s/abc/XYZ/g
w
q
" | ex file
Example is like vi without the fullscreen mode. You can give it the same commands you would at vi's : prompt.
I found this thread among others and I agree it contains the most complete answers so I'm adding mine too:
sed and ed are so useful...by hand.
Look at this code from #Johnny:
sed -i -e 's/abc/XYZ/g' /tmp/file.txt
When my restriction is to use it in a shell script, no variable can be used inside in place of "abc" or "XYZ". The BashFAQ seems to agree with what I understand at least. So, I can't use:
x='abc'
y='XYZ'
sed -i -e 's/$x/$y/g' /tmp/file.txt
#or,
sed -i -e "s/$x/$y/g" /tmp/file.txt
but, what can we do? As, #Johnny said use a while read... but, unfortunately that's not the end of the story. The following worked well with me:
#edit user's virtual domain
result=
#if nullglob is set then, unset it temporarily
is_nullglob=$( shopt -s | egrep -i '*nullglob' )
if [[ is_nullglob ]]; then
shopt -u nullglob
fi
while IFS= read -r line; do
line="${line//'<servername>'/$server}"
line="${line//'<serveralias>'/$alias}"
line="${line//'<user>'/$user}"
line="${line//'<group>'/$group}"
result="$result""$line"'\n'
done < $tmp
echo -e $result > $tmp
#if nullglob was set then, re-enable it
if [[ is_nullglob ]]; then
shopt -s nullglob
fi
#move user's virtual domain to Apache 2 domain directory
......
As one can see if nullglob is set then, it behaves strangely when there is a string containing a * as in:
<VirtualHost *:80>
ServerName www.example.com
which becomes
<VirtualHost ServerName www.example.com
there is no ending angle bracket and Apache2 can't even load.
This kind of parsing should be slower than one-hit search and replace but, as you already saw, there are four variables for four different search patterns working out of one parse cycle.
The most suitable solution I can think of with the given assumptions of the problem.
You can use sed:
sed -i 's/abc/XYZ/gi' /tmp/file.txt
You can use find and sed if you don't know your filename:
find ./ -type f -exec sed -i 's/abc/XYZ/gi' {} \;
Find and replace in all Python files:
find ./ -iname "*.py" -type f -exec sed -i 's/abc/XYZ/gi' {} \;
Be careful if you replace URLs with "/" character.
An example of how to do it:
sed -i "s%http://domain.com%http://www.domain.com/folder/%g" "test.txt"
Extracted from: http://www.sysadmit.com/2015/07/linux-reemplazar-texto-en-archivos-con-sed.html
If the file you are working on is not so big, and temporarily storing it in a variable is no problem, then you can use Bash string substitution on the whole file at once - there's no need to go over it line by line:
file_contents=$(</tmp/file.txt)
echo "${file_contents//abc/XYZ}" > /tmp/file.txt
The whole file contents will be treated as one long string, including linebreaks.
XYZ can be a variable eg $replacement, and one advantage of not using sed here is that you need not be concerned that the search or replace string might contain the sed pattern delimiter character (usually, but not necessarily, /). A disadvantage is not being able to use regular expressions or any of sed's more sophisticated operations.
You may also use the ed command to do in-file search and replace:
# delete all lines matching foobar
ed -s test.txt <<< $'g/foobar/d\nw'
See more in "Editing files via scripts with ed".
To edit text in the file non-interactively, you need in-place text editor such as vim.
Here is simple example how to use it from the command line:
vim -esnc '%s/foo/bar/g|:wq' file.txt
This is equivalent to #slim answer of ex editor which is basically the same thing.
Here are few ex practical examples.
Replacing text foo with bar in the file:
ex -s +%s/foo/bar/ge -cwq file.txt
Removing trailing whitespaces for multiple files:
ex +'bufdo!%s/\s\+$//e' -cxa *.txt
Troubleshooting (when terminal is stuck):
Add -V1 param to show verbose messages.
Force quit by: -cwq!.
See also:
How to edit files non-interactively (e.g. in pipeline)? at Vi SE
Try the following shell command:
find ./ -type f -name "file*.txt" | xargs sed -i -e 's/abc/xyz/g'
You can use python within the bash script too. I didn't have much success with some of the top answers here, and found this to work without the need for loops:
#!/bin/bash
python
filetosearch = '/home/ubuntu/ip_table.txt'
texttoreplace = 'tcp443'
texttoinsert = 'udp1194'
s = open(filetosearch).read()
s = s.replace(texttoreplace, texttoinsert)
f = open(filetosearch, 'w')
f.write(s)
f.close()
quit()
Simplest way to replace multiple text in a file using sed command
Command -
sed -i 's#a/b/c#D/E#g;s#/x/y/z#D:/X#g;' filename
In the above command s#a/b/c#D/E#g where I am replacing a/b/c with D/E and then after the ; we again doing the same thing
You can use rpl command. For example you want to change domain name in whole php project.
rpl -ivRpd -x'.php' 'old.domain.name' 'new.domain.name' ./path_to_your_project_folder/
This is not clear bash of cause, but it's a very quick and usefull. :)
For MAC users in case you don't read the comments :)
As mentioned by #Austin, if you get the Invalid command code error
For the in-place replacements BSD sed requires a file extension after the -i flag to save to a backup file with given extension.
sed -i '.bak' 's/find/replace' /file.txt
You can use '' empty string if you want to skip backup.
sed -i '' 's/find/replace' /file.txt
All merit to #Austin
Open file using vim editor. In command mode
:%s/abc/xyz/g
This is the simplest
In case of doing changes in multiple files together we can do in a single line as:-
user_name='whoami'
for file in file1.txt file2.txt file3.txt; do sed -i -e 's/default_user/${user_name}/g' $file; done
Added if in case could be useful.
I am trying to convert 2015-06-03_18-05-30 to 20150603180530 using sed.
I have this:
$ var='2015-06-03_18-05-30'
$ echo $var | sed 's/\-\|\_//g'
$ echo $var | sed 's/-|_//g'
None of these are working. Why is the alternation not working?
As long as your script has a #!/bin/bash (or ksh, or zsh) shebang, don't use sed or tr: Your shell can do this built-in without the (comparatively large) overhead of launching any external tool:
var='2015-06-03_18-05-30'
echo "${var//[-_]/}"
That said, if you really want to use sed, the GNU extension -r enables ERE syntax:
$ sed -r -e 's/-|_//g' <<<'2015-06-03_18-05-30'
20150603180530
See http://www.regular-expressions.info/posix.html for a discussion of differences between BRE (default for sed) and ERE. That page notes, in discussing ERE extensions:
Alternation is supported through the usual vertical bar |.
If you want to work on POSIX platforms -- with /bin/sh rather than bash, and no GNU extensions -- then reformulate your regex to use a character class (and, to avoid platform-dependent compatibility issues with echo[1], use printf instead):
printf '%s\n' "$var" | sed 's/[-_]//g'
[1] - See the "APPLICATION USAGE" section of that link, in particular.
Something like this ought to do.
sed 's/[-_]//g'
This reads as:
s: Search
/[-_]/: for any single character matching - or _
//: replace it with nothing
g: and do that for every character in the line
Sed operates on every line by default, so this covers every instance in the file/string.
I know you asked for a solution using sed, but I offer an alternative in tr:
$ var='2015-06-03_18-05-30'
$ echo $var | tr -d '_-'
20150603180530
tr should be a little faster.
Explained:
tr stands for translate and it can be used to replace certain characters with another ones.
-d option stands for delete and it removes the specified characters instead of replacing them.
'_-' specifies the set of characters to be removed (can also be specified as '\-_' but you need to escape the - there because it's considered another option otherwise).
Easy:
sed 's/[-_]//g'
The character class [-_] matches of the characters from the set.
sed 's/[^[:digit:]]//g' YourFile
Could you tell me what failed on echo $var | sed 's/\-\|\_//g', it works here (even if escapping - and _ are not needed and assuming you use a GNU sed due to \| that only work in this enhanced version of sed)
I want to do some simple string replace in Bash with sed. I am Ubuntu 10.10.
Just see the following code, it is self-explanatory:
name="A%20Google.."
echo $name|sed 's/\%20/_/'|sed 's/\.+/_/'
I want to get A_Google_ but I get A_Google..
The sed 's/\.+/_/' part is obviously wrong.
BTW, sed 's/\%20/_/' and sed 's/%20/_/' both work. Which is better?
sed speaks POSIX basic regular expressions, which don't include + as a metacharacter. Portably, rewrite to use *:
sed 's/\.\.*/_/'
or if all you will ever care about is Linux, you can use various GNU-isms:
sed -r 's/\.\.*/_/' # turn on POSIX EREs (use -E instead of -r on OS X)
sed 's/\.\+/_/' # GNU regexes invert behavior when backslash added/removed
That last example answers your other question: a character which is literal when used as is may take on a special meaning when backslashed, and even though at the moment % doesn't have a special meaning when backslashed, future-proofing means not assuming that \% is safe.
Additional note: you don't need two separate sed commands in the pipeline there.
echo $name | sed -e 's/\%20/_/' -e 's/\.+/_/'
(Also, do you only need to do that once per line, or for all occurrences? You may want the /g modifier.)
The sed command doesn't understand + so you'll have to expand it by hand:
sed 's/\.\.*/_/'
Or tell sed that you want to use extended regexes:
sed -r 's/\.+/_/' # GNU
sed -E 's/\.+/_/' # OSX
Which switch, -r or -E, depends on your sed and it might not even support extended regexes so the portable solution is to use \.\.* in place of \.+. But, since you're on Linux, you should have GNU sed so sed -r should do the trick.
I have a program that writes to fd3 and I want to process that data with grep and sed. Here is how the code looks so far:
exec 3> >(grep "good:"|sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3
Nothing is output until I do a
exec 3>&-
Then, everything that I wanted finally arrives as I expected:
I got: data2
It seems to reply immediately if I use only a grep or only a sed, but mixing them seems to cause some sort of buffering. How can I get immediate output from fd3?
I think I found it. For some reason, grep doesn't automatically do line buffering. I added a --line-buffered option to grep and now it responds immediately.
You only need to tell grep and sed to not bufferize lines:
grep --line-buffered
and
sed -u
An alternate means to stop sed from buffering is to run it through the s2p sed-to-Perl translator and insert a directive to have it command-buffered, perhaps like
BEGIN { $| = 1 }
The other reason to do this is that it gives you the more convenient notation from EREs instead of the backslash-annoying legacy BREs. You also get the full complement of Unicode properties, which is often critical.
But you don’t need the translator for such a simple sed command. And you do not need both grep and sed, either. These all work:
perl -nle 'BEGIN{$|=1} if (/good:/) { s/.*:(.*)/I got: $1/; print }'
perl -nle 'BEGIN{$|=1} next unless /good:/; s/.*:(.*)/I got: $1/; print'
perl -nle 'BEGIN{$|=1} next unless /good:/; s/.*:/I got: /; print'
Now you also have access to the minimal quantifier, *?, +?, ??, {N,}?, and {N,M}?. These now allow things like .*? or \S+? or [\p{Pd}.]??, which may well be preferable.
You can merge the grep into the sed like so:
exec 3> >(sed -une '/^good:/s//I got: /p')
echo "bad:data1">&3
echo "good:data2">&3
Unpacking that a bit: You can put a regexp (between slashes as usual) before any sed command, which makes it only be applied to lines that match that regexp. If the first regexp argument to the s command is the empty string (s//whatever/) then it will reuse the last regexp that matched, which in this case is the prefix, so that saves having to repeat yourself. And finally, the -n option tells sed to print only what it is specifically told to print, and the /p suffix on the s command tells it to print the result of the substitution.
The -e option is not strictly necessary but is good style, it just means "the next argument is the sed script, not a filename".
Always put sed scripts in single quotes unless you need to substitute a shell variable in there, and even then I would put everything but the shell variable in single quotes (the shell variable is, of course, double-quoted). You avoid a bunch of backslash-related grief that way.
On a Mac, brew install coreutils and use gstdbuf to control buffering of grep and sed.
Turn off buffering in pipe seems to be the easiest and most generic answer. Using stdbuf (coreutils) :
exec 3> >(stdbuf -oL grep "good:" | sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3
I got: data2
Buffering has other dependencies, for example depending on mawk either gawk reading this pipe :
exec 3> >(stdbuf -oL grep "good:" | awk '{ sub(".*:", "I got: "); print }')
In that case, mawk would retain the input, gawk wouldn't.
See also How to fix stdio buffering