What's the simplest way to do a find and replace for a given input string, say abc, and replace with another string, say XYZ in file /tmp/file.txt?
I am writting an app and using IronPython to execute commands through SSH — but I don't know Unix that well and don't know what to look for.
I have heard that Bash, apart from being a command line interface, can be a very powerful scripting language. So, if this is true, I assume you can perform actions like these.
Can I do it with bash, and what's the simplest (one line) script to achieve my goal?
The easiest way is to use sed (or perl):
sed -i -e 's/abc/XYZ/g' /tmp/file.txt
Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.
If you really really want to use just bash, then the following can work:
while IFS='' read -r a; do
echo "${a//abc/XYZ}"
done < /tmp/file.txt > /tmp/file.txt.t
mv /tmp/file.txt{.t,}
This loops over each line, doing a substitution, and writing to a temporary file (don't want to clobber the input). The move at the end just moves temporary to the original name. (For robustness and security, the temporary file name should not be static or predictable, but let's not go there.)
For Mac users:
sed -i '' 's/abc/XYZ/g' /tmp/file.txt
(See the comment below why)
File manipulation isn't normally done by Bash, but by programs invoked by Bash, e.g.:
perl -pi -e 's/abc/XYZ/g' /tmp/file.txt
The -i flag tells it to do an in-place replacement.
See man perlrun for more details, including how to take a backup of the original file.
I was surprised when I stumbled over this...
There is a replace command which ships with the "mysql-server" package, so if you have installed it try it out:
# replace string abc to XYZ in files
replace "abc" "XYZ" -- file.txt file2.txt file3.txt
# or pipe an echo to replace
echo "abcdef" |replace "abc" "XYZ"
See man replace for more on this.
This is an old post but for anyone wanting to use variables as #centurian said the single quotes mean nothing will be expanded.
A simple way to get variables in is to do string concatenation since this is done by juxtaposition in bash the following should work:
sed -i -e "s/$var1/$var2/g" /tmp/file.txt
Bash, like other shells, is just a tool for coordinating other commands. Typically you would try to use standard UNIX commands, but you can of course use Bash to invoke anything, including your own compiled programs, other shell scripts, Python and Perl scripts etc.
In this case, there are a couple of ways to do it.
If you want to read a file, and write it to another file, doing search/replace as you go, use sed:
sed 's/abc/XYZ/g' <infile >outfile
If you want to edit the file in place (as if opening the file in an editor, editing it, then saving it) supply instructions to the line editor 'ex'
echo "%s/abc/XYZ/g
w
q
" | ex file
Example is like vi without the fullscreen mode. You can give it the same commands you would at vi's : prompt.
I found this thread among others and I agree it contains the most complete answers so I'm adding mine too:
sed and ed are so useful...by hand.
Look at this code from #Johnny:
sed -i -e 's/abc/XYZ/g' /tmp/file.txt
When my restriction is to use it in a shell script, no variable can be used inside in place of "abc" or "XYZ". The BashFAQ seems to agree with what I understand at least. So, I can't use:
x='abc'
y='XYZ'
sed -i -e 's/$x/$y/g' /tmp/file.txt
#or,
sed -i -e "s/$x/$y/g" /tmp/file.txt
but, what can we do? As, #Johnny said use a while read... but, unfortunately that's not the end of the story. The following worked well with me:
#edit user's virtual domain
result=
#if nullglob is set then, unset it temporarily
is_nullglob=$( shopt -s | egrep -i '*nullglob' )
if [[ is_nullglob ]]; then
shopt -u nullglob
fi
while IFS= read -r line; do
line="${line//'<servername>'/$server}"
line="${line//'<serveralias>'/$alias}"
line="${line//'<user>'/$user}"
line="${line//'<group>'/$group}"
result="$result""$line"'\n'
done < $tmp
echo -e $result > $tmp
#if nullglob was set then, re-enable it
if [[ is_nullglob ]]; then
shopt -s nullglob
fi
#move user's virtual domain to Apache 2 domain directory
......
As one can see if nullglob is set then, it behaves strangely when there is a string containing a * as in:
<VirtualHost *:80>
ServerName www.example.com
which becomes
<VirtualHost ServerName www.example.com
there is no ending angle bracket and Apache2 can't even load.
This kind of parsing should be slower than one-hit search and replace but, as you already saw, there are four variables for four different search patterns working out of one parse cycle.
The most suitable solution I can think of with the given assumptions of the problem.
You can use sed:
sed -i 's/abc/XYZ/gi' /tmp/file.txt
You can use find and sed if you don't know your filename:
find ./ -type f -exec sed -i 's/abc/XYZ/gi' {} \;
Find and replace in all Python files:
find ./ -iname "*.py" -type f -exec sed -i 's/abc/XYZ/gi' {} \;
Be careful if you replace URLs with "/" character.
An example of how to do it:
sed -i "s%http://domain.com%http://www.domain.com/folder/%g" "test.txt"
Extracted from: http://www.sysadmit.com/2015/07/linux-reemplazar-texto-en-archivos-con-sed.html
If the file you are working on is not so big, and temporarily storing it in a variable is no problem, then you can use Bash string substitution on the whole file at once - there's no need to go over it line by line:
file_contents=$(</tmp/file.txt)
echo "${file_contents//abc/XYZ}" > /tmp/file.txt
The whole file contents will be treated as one long string, including linebreaks.
XYZ can be a variable eg $replacement, and one advantage of not using sed here is that you need not be concerned that the search or replace string might contain the sed pattern delimiter character (usually, but not necessarily, /). A disadvantage is not being able to use regular expressions or any of sed's more sophisticated operations.
You may also use the ed command to do in-file search and replace:
# delete all lines matching foobar
ed -s test.txt <<< $'g/foobar/d\nw'
See more in "Editing files via scripts with ed".
To edit text in the file non-interactively, you need in-place text editor such as vim.
Here is simple example how to use it from the command line:
vim -esnc '%s/foo/bar/g|:wq' file.txt
This is equivalent to #slim answer of ex editor which is basically the same thing.
Here are few ex practical examples.
Replacing text foo with bar in the file:
ex -s +%s/foo/bar/ge -cwq file.txt
Removing trailing whitespaces for multiple files:
ex +'bufdo!%s/\s\+$//e' -cxa *.txt
Troubleshooting (when terminal is stuck):
Add -V1 param to show verbose messages.
Force quit by: -cwq!.
See also:
How to edit files non-interactively (e.g. in pipeline)? at Vi SE
Try the following shell command:
find ./ -type f -name "file*.txt" | xargs sed -i -e 's/abc/xyz/g'
You can use python within the bash script too. I didn't have much success with some of the top answers here, and found this to work without the need for loops:
#!/bin/bash
python
filetosearch = '/home/ubuntu/ip_table.txt'
texttoreplace = 'tcp443'
texttoinsert = 'udp1194'
s = open(filetosearch).read()
s = s.replace(texttoreplace, texttoinsert)
f = open(filetosearch, 'w')
f.write(s)
f.close()
quit()
Simplest way to replace multiple text in a file using sed command
Command -
sed -i 's#a/b/c#D/E#g;s#/x/y/z#D:/X#g;' filename
In the above command s#a/b/c#D/E#g where I am replacing a/b/c with D/E and then after the ; we again doing the same thing
You can use rpl command. For example you want to change domain name in whole php project.
rpl -ivRpd -x'.php' 'old.domain.name' 'new.domain.name' ./path_to_your_project_folder/
This is not clear bash of cause, but it's a very quick and usefull. :)
For MAC users in case you don't read the comments :)
As mentioned by #Austin, if you get the Invalid command code error
For the in-place replacements BSD sed requires a file extension after the -i flag to save to a backup file with given extension.
sed -i '.bak' 's/find/replace' /file.txt
You can use '' empty string if you want to skip backup.
sed -i '' 's/find/replace' /file.txt
All merit to #Austin
Open file using vim editor. In command mode
:%s/abc/xyz/g
This is the simplest
In case of doing changes in multiple files together we can do in a single line as:-
user_name='whoami'
for file in file1.txt file2.txt file3.txt; do sed -i -e 's/default_user/${user_name}/g' $file; done
Added if in case could be useful.
I'm writing a bash script to non-interactively enable hibernation on a linux system. To this end, I need to insert a shell variable that contains slashes on a specific line of a while, inside quotes that are on that line.
The relevant part of the file to b edited looks like this:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
I need to change it to this:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Manjaro"
GRUB_CMDLINE_LINUX_DEFAULT="quiet resume=/dev/disk/by-partuuid/c5a552c2-fe8f-423a-9037-c35bf090d9c3"
GRUB_CMDLINE_LINUX=""
The added parameter is provided by a shell variable.
I tried this:
sed -i '\*^GRUB_CMDLINE_LINUX_DEFAULT* s*"$* '"$(grub_resume_boot_option)"'"*' /etc/default/grub
Logic that I was aiming for was: "Look for a line that starts with pattern "GRUB_CMDLINE_LINUX_DEFAULT", and replace the last " with the given pattern. Use * as delimiter to preserve the slashes in the expanded variable."
However, the command fails if there are any spaces at the end of the line. Is there any way to make it not take spaces at the end of the line into account?
Also, if there is more simple or readable solution to this, I would be very interested.
You can do:
sed -i 's#^\(GRUB_CMDLINE_LINUX_DEFAULT="quiet\)"$#\1 resume=/dev/disk/by-partuuid/c5a552c2-fe8f-423a-9037-c35bf090d9c3"#' /etc/default/grub
Example:
$ sed 's#^\(GRUB_CMDLINE_LINUX_DEFAULT="quiet\)"$#\1 resume=/dev/disk/by-partuuid/c5a552c2-fe8f-423a-9037-c35bf090d9c3"#' <<<'GRUB_CMDLINE_LINUX_DEFAULT="quiet"'
GRUB_CMDLINE_LINUX_DEFAULT="quiet resume=/dev/disk/by-partuuid/c5a552c2-fe8f-423a-9037-c35bf090d9c3"
From: https://serverfault.com/questions/885684/editing-the-value-of-grub-cmdline-linux-default-thru-bash-script
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="/&newstuff /' /etc/default/grub
This will add "newstuff" to the beginning of the list of params. Examples:
sed 's/^GRUB_CMDLINE_LINUX_DEFAULT="/&newstuff /' <<< 'GRUB_CMDLINE_LINUX_DEFAULT="quiet"'
GRUB_CMDLINE_LINUX_DEFAULT="newstuff quiet"
sed 's/^GRUB_CMDLINE_LINUX_DEFAULT="/&newstuff /' <<< 'GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"'
GRUB_CMDLINE_LINUX_DEFAULT="newstuff quiet splash"
sed -i "s/GRUB_CMDLINE_LINUX_DEFAULT="[^"]*/& $(grub_resume_boot_option)/" /etc/default/grub
I'm a newbie at bash script and I'm trying to write a bash script to edit a configuration file. i want to remove " foo" from the file.
Using the code
echo
sed -e "s/ foo//" ./test > ./test2
will create a new file with the content i want. But if i try to edit the same file using this
echo
sed -e "s/ foo//" ./test > ./test
the file will just become empty. Is this not the way? how can i simply edit a file? thanks.
This is what the -i switch is for: edit-in-place:
sed -i -e "s/ foo//" ./test
The -i switch is documented in man sed:
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied)
For GNU sed (linux), no back-up is created when the suffix is omitted. For other versions of sed, it might be mandatory to supply a backup suffix.
Empty file
Shell opens the file test for writing -> clearing all its previous contents.
How to edit same file
Solution 1:- inline change
sed -i.bk "s/ foo//" ./test
Solution 2:- use intermediate file (manual version of previous cmd)
sed -e "s/ foo//" ./test > test.bk && mv test.bk test
Sed is a [s]tream [ed]itor, not a text editor. Using the '-i' parameter is not POSIX, plus it adds the overhead of creating a temporary file before replacing the original file.
If you want to edit your files truly in place, you should use ed/ex instead.
ex -sc '%s/ foo//g|xit' ./test
See the manual pages sed(1p), ed(1p) and ex(1p).
I know this is a simple answer and I could probably keep digging around on Google before I stroll across the answer. But I am on a tight schedule and I was hoping for an easy response.
I need to update a variable in ifcfg-eth0 upon an installation. So in other words, this is what needs to happen:
The following variables need to change from:
ONBOOT=no
BOOTPROTO=dhcp
to
ONBOOT=yes
BOOTPROTO=static
Thanks in advance!
Cheers.
sed -i -e '/^ONBOOT=/s|.*|ONBOOT=yes|; /^BOOTPROTO=/s|.*|BOOTPROTO=static|' file
Also try:
sed -i -re 's|^(ONBOOT=).*|\1yes|; s|^(BOOTPROTO=).*|\1static|' file
Or
sed -i -e 's|^\(ONBOOT=\).*|\1yes|; s|^\(BOOTPROTO=\).*|\1static|' file
You can make use of something like this, which lets you define exactly what values you want to add in the file:
$ bootproto="static"
$ sed -r "s/(BOOTPROTO\s*=\s*).*/\1$bootproto/" file
ONBOOT=no
BOOTPROTO=static
And to make the two of them together:
$ onboot="yes"
$ bootproto="static"
$ sed -r -e "s/(ONBOOT\s*=\s*).*/\1$onboot/" -e "s/(BOOTPROTO\s*=\s*).*/\1$bootproto/" file
ONBOOT=yes
BOOTPROTO=static
Explanation
(BOOTPROTO\s*=\s*).* catches a group of text containing BOOTPROTO + any_number_of_spaces + = + any_number_of_spaces. Then, matches the rest of the text, that we want to remove. \1$var prints it back together with the given variable.
\s* keeps the current spaces as they were and then replaces the same line changing the current text with the bash variable $bootproto.
-r is used to catch groups with () instead of \( and \).
To make it edit in place, use sed -i.bak. This will create a backup of the file, file.bak, and the file will be updated with the new content.
All together:
sed -ir -e "s/(ONBOOT\s*=\s*).*/\1$onboot/" -e "s/(BOOTPROTO\s*=\s*).*/\1$bootproto/" file
The similar as the sed solutons with perl
perl -i.bak -pe 's/(ONBOOT)=no/$1=yes/;s/(BOOTPROTO)=dhcp/$1=static/' files....
I have a file BookDB.txt which stores information in the following manner :
C++ for dummies:Jared:10.67:4:5
Java for dummies:David:10.45:3:6
PHP for dummies:Sarah:10.47:2:7
Assuming that during runtime, the scipt asks the user for the title he wants to delete. This is then stored in the TITLE variable. How do I then delete the line containing the string in question? I've tried the following command but to no avail :
sed '/$TITLE/' BookDB.txt >> /dev/null
You can for example do:
$ title="C++ for dummies"
$ sed -i "/$title/d" a
$ cat a
**Java for dummies**:David:10.45:3:6
**PHP for dummies**:Sarah:10.47:2:7
Note few things:
Double quotes in sed are needed to have you variable expanded. Otherwise, it will look for the fixed string "$title".
With -i you make in-place replacement, so that your file gets updated once sed has performed.
d is the way to indicate sed that you want to delete such matching line.
Your command should be,
sed "/$TITLE/d" file
To save the changes, you need to add -i inline edit parameter.
sed -i "/$TITLE/d" file
For variable expansion in sed, you need to put the code inside double quotes.