I'm using for loop, with arguments i. Each argument contains ".net" at the end and in directory they are in one line, divided by some space. Now I need to get rid of these ".net" using substitution of sed, but it's not working. I went through different options, the most recent one is
sed 's/\.(net)//g' $i;
which is obviously not correct, but I just can't find anything online about this.
To make it clear, lets say I have a directory with 5 files with names
file1.net
file2.net
file3.net
file4.net
file5.net
I would like my output to be
file1
file2
file3
file
file5
...Could somebody give me some advice?
You can use
for f in *.net; do mv "$f" "${f%.*}"; done
Details:
for f in *.net; - iterates over files with net extension
mv "$f" "${f%.*}" - renames the files with the file without net extension (${f%.*} removes all text - as few as possible - from the end of f till the first ., see Parameter expansion).
This is a work for perl's rename :
rename -n 's/\.net//' *.net
The -n is for test purpose. Remove it if the output looks good for you
This way:
sed -i.backup 's/\.net$//g' "$1";
It will create a backup for safeness
I want to replace any special character (not a number or letter) to one single '-'.
I tried the code below with some characters, but it doesn't work when the character is repeated more than 1 time because would still have more than one '-'.
#!/bin/bash
for f in *; do mv "$f" "${f// /-}"; done
for f in *; do mv "$f" "${f//_/-}"; done
for f in *; do mv "$f" "${f//-/-}"; done
what I want:
test---file -> test-file
test file -> test-file
test______file -> test-file
teeesst--ffile -> teeesst-ffile
test555----file__ -> test555-file
Please, explain your answer because I don't know much about bash, regexp...
There are a couple of different rename (or prename) commands available in various distributions of Linux what will handle regex substitutions.
But you can also use Bash's extended globbing to do some of that. The pattern ${var//+([-_ ])/-} says to replace any runs of one or more characters that are listed in the square brackets with one hyphen.
shopt -s extglob
# demonstration:
for file in test---file 'test file' test______file teeesst--ffile test555----file__
do
echo "${file//+([-_ ])/-}"
done
Output:
test-file
test-file
test-file
teeesst-ffile
test555-file-
The extended glob +() is similar to .+ in regex. Other Bash extended globs (from man bash):
?(pattern-list)
Matches zero or one occurrence of the given patterns
*(pattern-list)
Matches zero or more occurrences of the given patterns
+(pattern-list)
Matches one or more occurrences of the given patterns
#(pattern-list)
Matches one of the given patterns
!(pattern-list)
Matches anything except one of the given patterns
Note that the final hyphen is not removed here, but could be using an additional parameter expansion:
file=${file/%-/}
which says to remove a hyphen at the end of the string.
You can use tr (as shown above in the comment) or, actually, sed makes more sense in this case. For example, given your list of filenames:
$ cat fnames
test---file
test file
test______file
teeesst--ffile
test555----file__
You can use the sed expression:
sed -e 's/[[:punct:] ][[:punct:] ]*/-/' -e 's/[[:punct:] ]*$//'
Example Use/Output
$ sed -e 's/[[:punct:] ][[:punct:] ]*/-/' -e 's/[[:punct:] ]*$//' fnames
test-file
test-file
test-file
teeesst-ffile
test555-file
Depending on how your filenames are stored, you can either use command substitution individually, or you can use process substitution and feed the updated names into a while loop or something similar.
I want to find all the .pdf files recursively by using find
So I typed in find . -name *.pdf
And the output was weird ,it only contains all the pdf files in the current directory , other pdf fils in the sub directory is omitted
Then I tried this find . -name '*.pdf'
This time ,every thing is fine .And I got what I want, I mean all the pdf files including those located in the sub directory.
So here comes the deal: what is the difference between find . -name *.pdf and find . -name '*.pdf'
Yes as you've found that quoting makes all the difference there.
Without quoting *.pdf gets expanded by shell glob expansion rule even before find runs and thus find command shows all the pdf files from that list only.
In other words this find command:
find . -name *.pdf
is same as:
printf "%s\n" *.pdf
So right way to use find is:
find . -name '*.pdf'
In the Linux shell (bash, csh, sh, and probably many others I'm not as familiar with), different quotes mean different things.
Fundamentally, quotes do two things for you:
Limiting text substitution: You're experiencing the shell substituting references to all PDF files in your current directory for *.pdf. That's an example of text substitution. Text substitution can also occur with variable names--for example:
bash$ MYVAR=test
bash$ echo $MYVAR
test
bash$ echo '$MYVAR'
$MYVAR
will give you the output test, because $MYVAR is substituted with the value the variable is set to.
Overriding space as an argument separator: Let's pretend you have a directory with these files
bash$ ls -1
file1
file1 file2
file2
If you type ls file1, you'd get file1 as you expect. Similarly, ls file2 gives you file2. The following commands show the significance of quotes overriding space as an argument separator:
bash$ ls -1 file1 file2
file1
file2
bash$ ls -1 "file1 file2"
file1 file2
Notice how the first example displays two files file1 and file2, while the second example displays one file file1 file2. That's because the quotes stop " " (a single space) from being used as an argument separator.
One final note: your original question asks about the difference between something and 'something'. It's worth nothing that there is actually a difference between something, 'something', and "something". Consider this:
bash$ MYVAR=test
bash$ echo $MYVAR
test
bash$ echo '$MYVAR'
$MYVAR
bash$ echo "$MYVAR"
test
Note the difference between '$MYVAR' and "$MYVAR". The ' (single quote) is considered a "strong quote," meaning everything contained inside it is explicit. The " (double quote) is a "weak quote," which does not expand * or ?, but does expand variable names and command substitutions.
The Grymorie provides a tremendous amount of information about quotes. Have fun learning!
I have a text file which has a particular line something like
sometext sometext sometext TEXT_TO_BE_REPLACED sometext sometext sometext
I need to replace the whole line above with
This line is removed by the admin.
The search keyword is TEXT_TO_BE_REPLACED
I need to write a shell script for this. How can I achieve this using sed?
You can use the change command to replace the entire line, and the -i flag to make the changes in-place. For example, using GNU sed:
sed -i '/TEXT_TO_BE_REPLACED/c\This line is removed by the admin.' /tmp/foo
You need to use wildcards (.*) before and after to replace the whole line:
sed 's/.*TEXT_TO_BE_REPLACED.*/This line is removed by the admin./'
The Answer above:
sed -i '/TEXT_TO_BE_REPLACED/c\This line is removed by the admin.' /tmp/foo
Works fine if the replacement string/line is not a variable.
The issue is that on Redhat 5 the \ after the c escapes the $. A double \\ did not work either (at least on Redhat 5).
Through hit and trial, I discovered that the \ after the c is redundant if your replacement string/line is only a single line. So I did not use \ after the c, used a variable as a single replacement line and it was joy.
The code would look something like:
sed -i "/TEXT_TO_BE_REPLACED/c $REPLACEMENT_TEXT_STRING" /tmp/foo
Note the use of double quotes instead of single quotes.
The accepted answer did not work for me for several reasons:
my version of sed does not like -i with a zero length extension
the syntax of the c\ command is weird and I couldn't get it to work
I didn't realize some of my issues are coming from unescaped slashes
So here is the solution I came up with which I think should work for most cases:
function escape_slashes {
sed 's/\//\\\//g'
}
function change_line {
local OLD_LINE_PATTERN=$1; shift
local NEW_LINE=$1; shift
local FILE=$1
local NEW=$(echo "${NEW_LINE}" | escape_slashes)
# FIX: No space after the option i.
sed -i.bak '/'"${OLD_LINE_PATTERN}"'/s/.*/'"${NEW}"'/' "${FILE}"
mv "${FILE}.bak" /tmp/
}
So the sample usage to fix the problem posed:
change_line "TEXT_TO_BE_REPLACED" "This line is removed by the admin." yourFile
All of the answers provided so far assume that you know something about the text to be replaced which makes sense, since that's what the OP asked. I'm providing an answer that assumes you know nothing about the text to be replaced and that there may be a separate line in the file with the same or similar content that you do not want to be replaced. Furthermore, I'm assuming you know the line number of the line to be replaced.
The following examples demonstrate the removing or changing of text by specific line numbers:
# replace line 17 with some replacement text and make changes in file (-i switch)
# the "-i" switch indicates that we want to change the file. Leave it out if you'd
# just like to see the potential changes output to the terminal window.
# "17s" indicates that we're searching line 17
# ".*" indicates that we want to change the text of the entire line
# "REPLACEMENT-TEXT" is the new text to put on that line
# "PATH-TO-FILE" tells us what file to operate on
sed -i '17s/.*/REPLACEMENT-TEXT/' PATH-TO-FILE
# replace specific text on line 3
sed -i '3s/TEXT-TO-REPLACE/REPLACEMENT-TEXT/'
for manipulation of config files
i came up with this solution inspired by skensell answer
configLine [searchPattern] [replaceLine] [filePath]
it will:
create the file if not exists
replace the whole line (all lines) where searchPattern matched
add replaceLine on the end of the file if pattern was not found
Function:
function configLine {
local OLD_LINE_PATTERN=$1; shift
local NEW_LINE=$1; shift
local FILE=$1
local NEW=$(echo "${NEW_LINE}" | sed 's/\//\\\//g')
touch "${FILE}"
sed -i '/'"${OLD_LINE_PATTERN}"'/{s/.*/'"${NEW}"'/;h};${x;/./{x;q100};x}' "${FILE}"
if [[ $? -ne 100 ]] && [[ ${NEW_LINE} != '' ]]
then
echo "${NEW_LINE}" >> "${FILE}"
fi
}
the crazy exit status magic comes from https://stackoverflow.com/a/12145797/1262663
In my makefile I use this:
#sed -i '/.*Revision:.*/c\'"`svn info -R main.cpp | awk '/^Rev/'`"'' README.md
PS: DO NOT forget that the -i changes actually the text in the file... so if the pattern you defined as "Revision" will change, you will also change the pattern to replace.
Example output:
Abc-Project written by John Doe
Revision: 1190
So if you set the pattern "Revision: 1190" it's obviously not the same as you defined them as "Revision:" only...
bash-4.1$ new_db_host="DB_HOSTNAME=good replaced with 122.334.567.90"
bash-4.1$
bash-4.1$ sed -i "/DB_HOST/c $new_db_host" test4sed
vim test4sed
'
'
'
DB_HOSTNAME=good replaced with 122.334.567.90
'
it works fine
To do this without relying on any GNUisms such as -i without a parameter or c without a linebreak:
sed '/TEXT_TO_BE_REPLACED/c\
This line is removed by the admin.
' infile > tmpfile && mv tmpfile infile
In this (POSIX compliant) form of the command
c\
text
text can consist of one or multiple lines, and linebreaks that should become part of the replacement have to be escaped:
c\
line1\
line2
s/x/y/
where s/x/y/ is a new sed command after the pattern space has been replaced by the two lines
line1
line2
cat find_replace | while read pattern replacement ; do
sed -i "/${pattern}/c ${replacement}" file
done
find_replace file contains 2 columns, c1 with pattern to match, c2 with replacement, the sed loop replaces each line conatining one of the pattern of variable 1
To replace whole line containing a specified string with the content of that line
Text file:
Row: 0 last_time_contacted=0, display_name=Mozart, _id=100, phonebook_bucket_alt=2
Row: 1 last_time_contacted=0, display_name=Bach, _id=101, phonebook_bucket_alt=2
Single string:
$ sed 's/.* display_name=\([[:alpha:]]\+\).*/\1/'
output:
100
101
Multiple strings delimited by white-space:
$ sed 's/.* display_name=\([[:alpha:]]\+\).* _id=\([[:digit:]]\+\).*/\1 \2/'
output:
Mozart 100
Bach 101
Adjust regex to meet your needs
[:alpha] and [:digit:]
are Character Classes and Bracket Expressions
This worked for me:
sed -i <extension> 's/.*<Line to be replaced>.*/<New line to be added>/'
An example is:
sed -i .bak -e '7s/.*version.*/ version = "4.33.0"/'
-i: The extension for the backup file after the replacement. In this case, it is .bak.
-e: The sed script. In this case, it is '7s/.*version.*/ version = "4.33.0"/'. If you want to use a sed file use the -f flag
s: The line number in the file to be replaced. In this case, it is 7s which means line 7.
Note:
If you want to do a recursive find and replace with sed then you can grep to the beginning of the command:
grep -rl --exclude-dir=<directory-to-exclude> --include=\*<Files to include> "<Line to be replaced>" ./ | sed -i <extension> 's/.*<Line to be replaced>.*/<New line to be added>/'
The question asks for solutions using sed, but if that's not a hard requirement then there is another option which might be a wiser choice.
The accepted answer suggests sed -i and describes it as replacing the file in-place, but -i doesn't really do that and instead does the equivalent of sed pattern file > tmp; mv tmp file, preserving ownership and modes. This is not ideal in many circumstances. In general I do not recommend running sed -i non-interactively as part of an automatic process--it's like setting a bomb with a fuse of an unknown length. Sooner or later it will blow up on someone.
To actually edit a file "in place" and replace a line matching a pattern with some other content you would be well served to use an actual text editor. This is how it's done with ed, the standard text editor.
printf '%s\n' '/TEXT_TO_BE_REPLACED/' d i 'This line is removed by the admin' . w q | \
ed -s /tmp/foo > /dev/null
Note that this only replaces the first matching line, which is what the question implied was wanted. This is a material difference from most of the other answers.
That disadvantage aside, there are some advantages to using ed over sed:
You can replace the match with one or multiple lines without any extra effort.
The replacement text can be arbitrarily complex without needing any escaping to protect it.
Most importantly, the original file is opened, modified, and saved. A copy is not made.
How it works
How it works:
printf will use its first argument as a format string and print each of its other arguments using that format, effectively meaning that each argument to printf becomes a line of output, which is all sent to ed on stdin.
The first line is a regex pattern match which causes ed to move its notion of "the current line" forward to the first line that matches (if there is no match the current line is set to the last line of the file).
The next is the d command which instructs ed to delete the entire current line.
After that is the i command which puts ed into insert mode;
after that all subsequent lines entered are written to the current line (or additional lines if there are any embedded newlines). This means you can expand a variable (e.g. "$foo") containing multiple lines here and it will insert all of them.
Insert mode ends when ed sees a line consisting of .
The w command writes the content of the file to disk, and
the q command quits.
The ed command is given the -s switch, putting it into silent mode so it doesn't echo any information as it runs,
the file to be edited is given as an argument to ed,
and, finally, stdout is thrown away to prevent the line matching the regex from being printed.
Some Unix-like systems may (inappropriately) ship without an ed installed, but may still ship with an ex; if so you can simply use it instead. If have vim but no ex or ed you can use vim -e instead. If you have only standard vi but no ex or ed, complain to your sysadmin.
It is as similar to above one..
sed 's/[A-Za-z0-9]*TEXT_TO_BE_REPLACED.[A-Za-z0-9]*/This line is removed by the admin./'
Below command is working for me. Which is working with variables
sed -i "/\<$E\>/c $D" "$B"
I very often use regex to extract data from files I just used that to replace the literal quote \" with // nothing :-)
cat file.csv | egrep '^\"([0-9]{1,3}\.[0-9]{1,3}\.)' | sed s/\"//g | cut -d, -f1 > list.txt
I have long file with the following list:
/drivers/isdn/hardware/eicon/message.c//add_b1()
/drivers/media/video/saa7134/saa7134-dvb.c//dvb_init()
/sound/pci/ac97/ac97_codec.c//snd_ac97_mixer_build()
/drivers/s390/char/tape_34xx.c//tape_34xx_unit_check()
(PROBLEM)/drivers/video/sis/init301.c//SiS_GetCRT2Data301()
/drivers/scsi/sg.c//sg_ioctl()
/fs/ntfs/file.c//ntfs_prepare_pages_for_non_resident_write()
/drivers/net/tg3.c//tg3_reset_hw()
/arch/cris/arch-v32/drivers/cryptocop.c//cryptocop_setup_dma_list()
/drivers/media/video/pvrusb2/pvrusb2-v4l2.c//pvr2_v4l2_do_ioctl()
/drivers/video/aty/atyfb_base.c//aty_init()
/block/compat_ioctl.c//compat_blkdev_driver_ioctl()
....
It contains all the functions in the kernel code. The notation is file//function.
I want to copy some 100 files from the kernel directory to another directory, so I want to strip every line from the function name, leaving just the filename.
It's super-easy in python, any idea how to write a 1-liner in the bash prompt that does the trick?
Thanks,
Udi
cat "func_list" | sed "s#//.*##" > "file_list"
Didn't run it :)
You can use pure Bash:
while read -r line; do echo "${line%//*}"; done < funclist.txt
Edit:
The syntax of the echo command is doing the same thing as the sed command in Eugene's answer: deleting the "//" and everything that comes after.
Broken down:
"echo ${line}" is the same as "echo $line"
the "%" deletes the pattern that follows it if it matches the trailing portion of the parameter
"%" makes the shortest possible match, "%%" makes the longest possible
"//*" is the pattern to match, "*" is similar to sed's ".*"
See the Parameter Expansion section of the Bash man page for more information, including:
using ${parameter#word} for matching the beginning of a parameter
${parameter/pattern/string} to do sed-style replacements
${parameter:offset:length} to retrieve substrings
etc.
here's a one liner in (g)awk
awk -F"//" '{print $1}' file
Here's one using cut and rev
cat file | rev | cut -d'/' -f2-| rev