I want to find a string say 'foo' in a file sat '1.txt' using shell script and replace 'foo' with 'bar' and store the output into other file say '2.txt' without any modification in 1.txt.
so '1.txt' would contain 'foo' itself but '2.txt' will now have all the contents of '1.txt' with 'foo' replaced by 'bar'
I am using this command in bash
sed -i "s/foo/bar/g" "1.txt" > 2.txt
but its not working.
Remove the -i option as it stands for in-place operation.
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied)
(sed(1) man page)
Remove the -i switch, or provide a backup extension (-i.bak), so the modified file will be 1.txt and 1.txt.bak the original backup.
You don't need quotes on "1.txt", unless the filename contains spaces.
sed "s/foo/bar/g" 1.txt > 2.txt
or
sed -i.bak "s/foo/bar/g" 1.txt
Take a look at sed manual
Related
I have a directory with multiple fasta file named as followed:
BC-1_bin_1_genes.faa
BC-1_bin_2_genes.faa
BC-1_bin_3_genes.faa
BC-1_bin_4_genes.faa
etc. (about 200 individual files)
The fasta header look like this:
>BC-1_k127_3926653_6 # 4457 # 5341 # -1 # ID=2_6;partial=01;start_type=Edge;rbs_motif=None;rbs_spacer=None;gc_cont=0.697
I now want to add the filename to the header since I want to annotate the sequences for each file.I tried the following:
for file in *.faa;
do
sed -i "s/>.*/${file%%.*}/" "$file" ;
done
It worked partially but it removed the ">" from the header which is essential for the fasta file. I tried to modify the "${file%%.*}" part to keep the carrot but it always called me out on bad substitutions.
I also tried this:
awk '/>/{sub(">","&"FILENAME"_");sub(/\.faa/,x)}1' *.faa
This worked in theory but only printed everything on my terminal rather than changing it in the respective files.
Could someone assist with this?
It's not clear whether you want to replace the earlier header, or add to it. Both scenarios are easy to do. Don't replace text you don't want to replace.
for file in ./*.faa;
do
sed -i "s/^>.*/>${file%%.*}/" "$file"
done
will replace the header, but include a leading > in the replacement, effectively preserving it; and
for file in ./*.faa;
do
sed -i "s/^>.*/&${file%%.*}/" "$file"
done
will append the file name at the end of the header (& in the replacement string evaluates to the string we are replacing, again effectively preserving it).
For another variation, try
for file in *.faa;
do
sed -i "/^>/s/\$/ ${file%%.*}/" "$file"
done
which says on lines which match the regex ^>, replace the empty string at the end of the line $ with the file name.
Of course, your Awk script could easily be fixed, too. Standard Awk does not have an option to parallel the -i "in-place" option of sed, but you can easily use a temporary file:
for file in ./*.faa;
do
awk '/>/{ $0 = $0 " " FILENAME);sub(/\.faa/,"")}1' "$file" >"$file.tmp" &&
mv "$file.tmp" "$file"
done
GNU Awk also has an -i inplace extension which you could simply add to the options of your existing script if you have GNU Awk.
Since FASTA files typically contain multiple headers, adding to the header rather than replacing all headers in a file with the same string seems more useful, so I changed your Awk script to do that instead.
For what it's worth, the name of the character ^ is caret (carrot is 🥕). The character > is called greater than or right angle bracket, or right broket or sometimes just wedge.
You just need to detect the pattern to replace and use regex to implement it:
fasta_helper.sh
location=$1
for file in $location/*.faa
do
full_filename=${file##*/}
filename="${full_filename%.*}"
#scape special chars
filename=$(echo $filename | sed 's_/_\\/_g')
echo "adding file name: $filename to: $full_filename"
sed -i -E "s/^[^#]+/>$filename /" $location/$full_filename
done
usage:
Just pass the folder with fasta files:
bash fasta_helper.sh /foo/bar
test:
lectures
Regex: matching up to the first occurrence of a character
Extract filename and extension in Bash
https://unix.stackexchange.com/questions/78625/using-sed-to-find-and-replace-complex-string-preferrably-with-regex
Locating your files
Suggesting to first identify your files with find command or ls command.
find . -type f -name "*.faa" -printf "%f\n"
A find command to print only file with filenames extension .faa. Including sub directories to current directory.
ls -1 "*.faa"
An ls command to print files and directories with extension .faa. In current directory.
Processing your files
Once you have the correct files list, iterate over the list and apply sed command.
for fileName in $(find . -type f -name "*.faa" -printf "%f\n"); do
stripedFileName=${fileName/.*/} # strip extension .faa
sed -i "1s|\$| $stripedFileName|" "fileName" # append value of stripedFileName at end of line 1
done
I have a folder lets say located here:
/Users/spotter/Downloads
and within the root folder there are two files:
test1.txt and test2.txt.
I want to write a shell script to save all the files to a list with a line like this:
file_list="$(ls /Users/spotter/Downloads)"
and echo $file_list will return:
/Users/spotter/Downloads/test1.txt
/Users/spotter/Downloads/test2.txt
However I want to change part of the dirname. Particularly I want to remove the /Users/spotter part.
I tried this like so:
file_list="$(ls /Users/spotter/Downloads |
while read path; do dirname "$path" | sed 's/users/spotter///'; done)"
which returns:
sed: 1: "s/users/spotter/Downloa ...": bad flag in substitute command: 'D'
sed: 1: "s/users/spotter/Downloa ...": bad flag in substitute command: 'D'
when I do echo $file_list I want this to be the output:
Downloads/test1.txt
Downloads/test2.txt
The problem is that sed thinks '/' is the delimiter between the RE and the substitution, so sed is not reading the other '/'s the way you want it to. You can use other characters as a delimiter. For instance 's~/Users/spotter/~~'.
I need to edit the file saved in a string using sed. As per the below example, I want to change some pattern in the php.ini file. Below example will allow only to change what is saved in the string.
[root#server ~]# PHPINI=/etc/php.ini
[root#server ~]# sed "s/somepattern/changedpattern/" <<< "$PHPINI"
Can somebody help? Thanks in advance.
You're using the herestring syntax which passes a string on standard input. You should instead pass the name of the file as an argument:
sed 's/somepattern/changedpattern/' /etc/php.ini
To overwrite the existing file, the standard method is to write to a temporary file and then overwrite the original:
sed 's/somepattern/changedpattern/' /etc/php.ini > tmp && mv tmp /etc/php.ini
Some versions of sed support "in-place" editing (which does more or less the same in the background):
sed -i.bak 's/somepattern/changedpattern/' /etc/php.ini
This creates a backup of the original file with a .bak suffix.
If the filename is contained within a variable, then simply use that instead
php_ini=/etc/php.ini
sed -i.bak 's/somepattern/changedpattern/' "$php_ini"
Note my use of lowercase variable names. Uppercase ones should be reserved for use by the shell.
If I wanted to search for a line in a file and append a string to the end of that line, how can I go about it? I.E.:
file=?
I want to search for file=? and replace the question mark with a file path. The file path is located in a variable $FILEPATH
file=$FILEPATH
Thanks!
EDIT
sed -i -f "s,file=\?,file=$FILEPATH,g"
The above works well and is what I'm looking for but is there a way to replace the question mark? With the code above if I have the following:
FILEPATH=/file/path
Properties file:
something=?
file=?
The replacement produces:
Properties file:
something=?
file=/file/path?
Is there a way to replace the ? completely?
I'd use sed for that:
sed -i "s/file=?/file=$FILEPATH/g" your_file
If your $FILEPATH has / then use a different sed separator, something like:
sed -i "s,file=?,file=$FILEPATH,g"
Don't escape your question mark
[jaypal:~/Temp] cat temp
file=?
file=?
[jaypal:~/Temp] echo $filepath
/usr/bin
[jaypal:~/Temp] sed -e 's_file=?_file='$filepath'_g' temp
file=/usr/bin
file=/usr/bin
Also to make inline changes I would recommend to use the following -
[jaypal:~/Temp] sed -ibak 's_file=?_file='$filepath'_g' temp
[jaypal:~/Temp] ls temp*
temp tempbak
[jaypal:~/Temp] cat temp
file=/usr/bin
file=/usr/bin
[jaypal:~/Temp] cat tempbak
file=?
file=?
This will make a backup copy of your original file before making any changes. In case if anything goes wrong you will have your original copy protected.
If you are using Bash, you can simply use Bash builtins and substitutions instead of sed:
#!/bin/bash
FILEPATH="/file/path"
while read line; do
echo "${line/file=\?/${line/\?/}$FILEPATH}"
done < yourfile
I was doing some hands-on with the Unix sed command. I was trying out the substitution and append command, in a file. But the difficulty is, I have to create an intermediate file, and then do mv to rename it to the original file.
Is there any way to do it at one shot in the same file?
[root#dhcppc0 practice]# sed '1i\
> Today is Sunday
> ' file1 > file1
[root#dhcppc0 practice]# cat file1
[root#dhcppc0 practice]#
The file is deleted!
[root#dhcppc0 practice]# sed 's/director/painter/' file1 > file1
[root#dhcppc0 practice]# cat file1
The file is deleted!
Try this -
sed -i '' 's/originaltext/replacementtext/g' filename | cat filename
-i '' is meant for providing a backup file. If you are confident your replacement won't cause an issue you can put '' to pass no backup file
/g is for replacing globally. If you have more than one originaltext in one line then with /g option will replace all else it will only replace the first.
GNU sed knows an option -i which does in-place edit of the given files.
When doing an operation file1 > file1 what actually happens is, that the file is opened and truncated by the shell before the program (which gets it's name as argument) comes around reading anything from it.
Update:
sed's man page states the following on the -i option (thanks Delan for mentioning it):
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied)
sed -i.bak 's/director/painter/' file1
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied)