I am working on a script that involves paths with wildcards. I know the wildcards will match one file only, I just don't know what the file extension will be ahead of time so I am using a wildcard.
The goal here is to find the path to the appropriate file, and then add that path to line 16 of a script.
I have something like this:
path=/path/to/somewhere/fileName*
sed "16 a file=$path" myScript.sh
What I expect to get is this (on line 16):
file=/path/to/somewhere/fileName.extension
But what I get is:
file=/path/to/somewhere/fileName*
For some reason sed is not expanding the wildcard when it adds the contents of $path and I can't figure out how to make sed do such a thing. I'm looking for a solution that either a) has sed properly expand $path or b) a way to get $path to contain the fully expanded string before being passed to sed.
Your variable contains just a string and you then interpolate that string. sed can't know that's not what you mean. If you want the shell (not sed!) to expand the wildcard, probably use a loop.
for path in /path/to/somewhere/fileName*; do
if [ -e "$path" ]; then # handle wildcard possibly not matching
sed "16 a file=$path" myScript.sh
fi
done
It's unclear what should happen if the wildcard matches multiple files; perhaps you want to add a break before fi to only substitute the first one if that happens.
This might work for you (GNU sed, echo and bash):
export path='/path/toSomeWhere/filename*'
sed '16{s/$/\na file=$(echo $path)/;s/.*/echo "&"/e}' file
Export the variable path which has been set to /path/toSomeWhere/filename* (notice the single quotes which prevents interpolation).
On line 16 of file append a line a file=$(echo $path) and then surround both lines by double quotes and prepend the echo command and evaluate (the e flag on the second substitution command) the expression.
Alternative:
sed '17e echo "a file=$(echo /path/toSomeWhere/filename*)"' file
Related
Given an example that my file name is
A_BC_DEF_GH_IJ_LMNO_PQ_11111111_1111111111_111111_AB.dat.meta
i am trying to rename this with unix command but when i tried using this cmd
for f in *.meta; do mv "$f" "$(echo $f|sed s/[0-9]/?/g|sed 's/-/*/g')" ; done
my file is renamed to
A_BC_DEF_GH_IJ_LMNO_PQ_????????_????????????????????_???????_AB.dat.meta
it is expected to rename the file to
A_BC_DEF_GH_IJ_LMNO_PQ__????????_????????????????????_*_AB.dat.meta
Im quite new with unix cmd , any approach that i should try ?
Since [0-9] and ? are undergoing filename expansion, you should quote them to avoid nasty error messages. With this in mind, I did a
echo A_BC_DEF_GH_IJ_LMNO_PQ_11111111_1111111111_111111_AB.dat.meta | sed 's/[0-9]/?/g'|sed 's/-/*/g'
and got as output A_BC_DEF_GH_IJ_LMNO_PQ????????????????????????AB.dat.meta, which makes sense to me. Why would you expect an asterisk in the resulting filename? In your second sed command, you are turning the hyphens into asterisks, but there is no hyphen in the input.
Of course it is pretty unsane to use question marks and asterisks in a file name, as this is just begging for trouble, but there is no law that you must not do this.
A_BC_DEF_GH_IJ_LMNO_PQ_11111111_1111111111_111111_AB.dat.meta
Match it with a regex. Remember which characters need to be escaped in sed. Remember about proper quoting - if you write $ it should be inside ". Note that if there are no files named *.meta it will just iterate over a string *.meta unless nullglob is set.
$ touch A_BC_DEF_GH_IJ_LMNO_PQ_11111111_1111111111_111111_AB.dat.meta
$ for f in *.meta; do mv "$f" "$(echo "$f" | sed 's/[0-9]/?/g; s/_\(?*\)_\(?*\)_\(?*\)_\([^_]*\)$/__\1_\2_*_\4/')" ; done
I'm new to shell scripting. My requirement is to retrieve lines between two pattern, its working fine if I run it from the terminal without using variables inside sed cmd. But the problem arises when I put all those below cmd in a file and tried to execute it.
#!/bin/sh
word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
fileC=`cat test.log`
output=`echo $fileC | sed -e "n/\$word/$upto/p"`
printf '%s\n' "$output"
If I use the below cmd in the terminal it works fine
sed -n '/ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34/,/2017-01-03 23:00/ p' test.log
Please suggest a workaround.
If we put aside for a moment the fact you shouldn't cat a file to a variable and then echo it for sed filtering, the reason why your command is not working is because you're not quoting the file content variable, fileC when echoing. This will munge together multiple whitespace characters and turn them into a single space. So, you're losing newlines from the file, as well as multiple spaces, tabs, etc.
To fix it, you can write:
fileC=$(cat test.log)
output=$(echo "$fileC" | sed -n "/$word/,/$upto/p")
Note the double-quotes around fileC (and a fixed sed expression, similar to your second example). Without the quotes (try echo $fileC), your fileC is expanded (with the default IFS) into a series of words, each being one argument to echo, and echo will just print those words separated with a single space. Additionally, if the file contains some of the globbing characters (like *), those patterns are also expanded. This is a common bash pitfall.
Much better would be to write it like this:
output=$(sed -n "/$word/,/$upto/p" test.log)
And if your patterns include some of the sed metacharacters, you should really escape them before using with sed, like this:
escape() {
sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$1";
}
output=$(sed -n "/$(escape "$word")/,/$(escape "$upto")/ p" test.log)
The correct approach will be something like:
word="ajp-qdcls2228.us.qdx.com%2F156.30.35.204-8009-34"
upto="2017-01-03 23:00"
awk -v beg="$word" -v end="$upto" '$0==beg{f=1} f{print; if ($0==end) exit}' file
but until we see your sample input and output we can't know for sure what it is you need to match on (full lines, partial lines, all text on one line, etc.) or what you want to print (include delimiters, exclude one, exclude both, etc.).
I have a text file which has a particular line something like
sometext sometext sometext TEXT_TO_BE_REPLACED sometext sometext sometext
I need to replace the whole line above with
This line is removed by the admin.
The search keyword is TEXT_TO_BE_REPLACED
I need to write a shell script for this. How can I achieve this using sed?
You can use the change command to replace the entire line, and the -i flag to make the changes in-place. For example, using GNU sed:
sed -i '/TEXT_TO_BE_REPLACED/c\This line is removed by the admin.' /tmp/foo
You need to use wildcards (.*) before and after to replace the whole line:
sed 's/.*TEXT_TO_BE_REPLACED.*/This line is removed by the admin./'
The Answer above:
sed -i '/TEXT_TO_BE_REPLACED/c\This line is removed by the admin.' /tmp/foo
Works fine if the replacement string/line is not a variable.
The issue is that on Redhat 5 the \ after the c escapes the $. A double \\ did not work either (at least on Redhat 5).
Through hit and trial, I discovered that the \ after the c is redundant if your replacement string/line is only a single line. So I did not use \ after the c, used a variable as a single replacement line and it was joy.
The code would look something like:
sed -i "/TEXT_TO_BE_REPLACED/c $REPLACEMENT_TEXT_STRING" /tmp/foo
Note the use of double quotes instead of single quotes.
The accepted answer did not work for me for several reasons:
my version of sed does not like -i with a zero length extension
the syntax of the c\ command is weird and I couldn't get it to work
I didn't realize some of my issues are coming from unescaped slashes
So here is the solution I came up with which I think should work for most cases:
function escape_slashes {
sed 's/\//\\\//g'
}
function change_line {
local OLD_LINE_PATTERN=$1; shift
local NEW_LINE=$1; shift
local FILE=$1
local NEW=$(echo "${NEW_LINE}" | escape_slashes)
# FIX: No space after the option i.
sed -i.bak '/'"${OLD_LINE_PATTERN}"'/s/.*/'"${NEW}"'/' "${FILE}"
mv "${FILE}.bak" /tmp/
}
So the sample usage to fix the problem posed:
change_line "TEXT_TO_BE_REPLACED" "This line is removed by the admin." yourFile
All of the answers provided so far assume that you know something about the text to be replaced which makes sense, since that's what the OP asked. I'm providing an answer that assumes you know nothing about the text to be replaced and that there may be a separate line in the file with the same or similar content that you do not want to be replaced. Furthermore, I'm assuming you know the line number of the line to be replaced.
The following examples demonstrate the removing or changing of text by specific line numbers:
# replace line 17 with some replacement text and make changes in file (-i switch)
# the "-i" switch indicates that we want to change the file. Leave it out if you'd
# just like to see the potential changes output to the terminal window.
# "17s" indicates that we're searching line 17
# ".*" indicates that we want to change the text of the entire line
# "REPLACEMENT-TEXT" is the new text to put on that line
# "PATH-TO-FILE" tells us what file to operate on
sed -i '17s/.*/REPLACEMENT-TEXT/' PATH-TO-FILE
# replace specific text on line 3
sed -i '3s/TEXT-TO-REPLACE/REPLACEMENT-TEXT/'
for manipulation of config files
i came up with this solution inspired by skensell answer
configLine [searchPattern] [replaceLine] [filePath]
it will:
create the file if not exists
replace the whole line (all lines) where searchPattern matched
add replaceLine on the end of the file if pattern was not found
Function:
function configLine {
local OLD_LINE_PATTERN=$1; shift
local NEW_LINE=$1; shift
local FILE=$1
local NEW=$(echo "${NEW_LINE}" | sed 's/\//\\\//g')
touch "${FILE}"
sed -i '/'"${OLD_LINE_PATTERN}"'/{s/.*/'"${NEW}"'/;h};${x;/./{x;q100};x}' "${FILE}"
if [[ $? -ne 100 ]] && [[ ${NEW_LINE} != '' ]]
then
echo "${NEW_LINE}" >> "${FILE}"
fi
}
the crazy exit status magic comes from https://stackoverflow.com/a/12145797/1262663
In my makefile I use this:
#sed -i '/.*Revision:.*/c\'"`svn info -R main.cpp | awk '/^Rev/'`"'' README.md
PS: DO NOT forget that the -i changes actually the text in the file... so if the pattern you defined as "Revision" will change, you will also change the pattern to replace.
Example output:
Abc-Project written by John Doe
Revision: 1190
So if you set the pattern "Revision: 1190" it's obviously not the same as you defined them as "Revision:" only...
bash-4.1$ new_db_host="DB_HOSTNAME=good replaced with 122.334.567.90"
bash-4.1$
bash-4.1$ sed -i "/DB_HOST/c $new_db_host" test4sed
vim test4sed
'
'
'
DB_HOSTNAME=good replaced with 122.334.567.90
'
it works fine
To do this without relying on any GNUisms such as -i without a parameter or c without a linebreak:
sed '/TEXT_TO_BE_REPLACED/c\
This line is removed by the admin.
' infile > tmpfile && mv tmpfile infile
In this (POSIX compliant) form of the command
c\
text
text can consist of one or multiple lines, and linebreaks that should become part of the replacement have to be escaped:
c\
line1\
line2
s/x/y/
where s/x/y/ is a new sed command after the pattern space has been replaced by the two lines
line1
line2
cat find_replace | while read pattern replacement ; do
sed -i "/${pattern}/c ${replacement}" file
done
find_replace file contains 2 columns, c1 with pattern to match, c2 with replacement, the sed loop replaces each line conatining one of the pattern of variable 1
To replace whole line containing a specified string with the content of that line
Text file:
Row: 0 last_time_contacted=0, display_name=Mozart, _id=100, phonebook_bucket_alt=2
Row: 1 last_time_contacted=0, display_name=Bach, _id=101, phonebook_bucket_alt=2
Single string:
$ sed 's/.* display_name=\([[:alpha:]]\+\).*/\1/'
output:
100
101
Multiple strings delimited by white-space:
$ sed 's/.* display_name=\([[:alpha:]]\+\).* _id=\([[:digit:]]\+\).*/\1 \2/'
output:
Mozart 100
Bach 101
Adjust regex to meet your needs
[:alpha] and [:digit:]
are Character Classes and Bracket Expressions
This worked for me:
sed -i <extension> 's/.*<Line to be replaced>.*/<New line to be added>/'
An example is:
sed -i .bak -e '7s/.*version.*/ version = "4.33.0"/'
-i: The extension for the backup file after the replacement. In this case, it is .bak.
-e: The sed script. In this case, it is '7s/.*version.*/ version = "4.33.0"/'. If you want to use a sed file use the -f flag
s: The line number in the file to be replaced. In this case, it is 7s which means line 7.
Note:
If you want to do a recursive find and replace with sed then you can grep to the beginning of the command:
grep -rl --exclude-dir=<directory-to-exclude> --include=\*<Files to include> "<Line to be replaced>" ./ | sed -i <extension> 's/.*<Line to be replaced>.*/<New line to be added>/'
The question asks for solutions using sed, but if that's not a hard requirement then there is another option which might be a wiser choice.
The accepted answer suggests sed -i and describes it as replacing the file in-place, but -i doesn't really do that and instead does the equivalent of sed pattern file > tmp; mv tmp file, preserving ownership and modes. This is not ideal in many circumstances. In general I do not recommend running sed -i non-interactively as part of an automatic process--it's like setting a bomb with a fuse of an unknown length. Sooner or later it will blow up on someone.
To actually edit a file "in place" and replace a line matching a pattern with some other content you would be well served to use an actual text editor. This is how it's done with ed, the standard text editor.
printf '%s\n' '/TEXT_TO_BE_REPLACED/' d i 'This line is removed by the admin' . w q | \
ed -s /tmp/foo > /dev/null
Note that this only replaces the first matching line, which is what the question implied was wanted. This is a material difference from most of the other answers.
That disadvantage aside, there are some advantages to using ed over sed:
You can replace the match with one or multiple lines without any extra effort.
The replacement text can be arbitrarily complex without needing any escaping to protect it.
Most importantly, the original file is opened, modified, and saved. A copy is not made.
How it works
How it works:
printf will use its first argument as a format string and print each of its other arguments using that format, effectively meaning that each argument to printf becomes a line of output, which is all sent to ed on stdin.
The first line is a regex pattern match which causes ed to move its notion of "the current line" forward to the first line that matches (if there is no match the current line is set to the last line of the file).
The next is the d command which instructs ed to delete the entire current line.
After that is the i command which puts ed into insert mode;
after that all subsequent lines entered are written to the current line (or additional lines if there are any embedded newlines). This means you can expand a variable (e.g. "$foo") containing multiple lines here and it will insert all of them.
Insert mode ends when ed sees a line consisting of .
The w command writes the content of the file to disk, and
the q command quits.
The ed command is given the -s switch, putting it into silent mode so it doesn't echo any information as it runs,
the file to be edited is given as an argument to ed,
and, finally, stdout is thrown away to prevent the line matching the regex from being printed.
Some Unix-like systems may (inappropriately) ship without an ed installed, but may still ship with an ex; if so you can simply use it instead. If have vim but no ex or ed you can use vim -e instead. If you have only standard vi but no ex or ed, complain to your sysadmin.
It is as similar to above one..
sed 's/[A-Za-z0-9]*TEXT_TO_BE_REPLACED.[A-Za-z0-9]*/This line is removed by the admin./'
Below command is working for me. Which is working with variables
sed -i "/\<$E\>/c $D" "$B"
I very often use regex to extract data from files I just used that to replace the literal quote \" with // nothing :-)
cat file.csv | egrep '^\"([0-9]{1,3}\.[0-9]{1,3}\.)' | sed s/\"//g | cut -d, -f1 > list.txt
Assuming an ini-style file like this,
[Group]
Icon=xxx.ico
Title=An Image Editor
Description=Manipulates .ico, .png and .jpeg images
I want to replace/delete ".ico" ONLY in the line that starts with (or matches) "Icon="
I was trying this:
oldline="`cat "$file" | grep "Icon="`"
newline="`echo "$oldline" | tr ".ico" ".png"`"
cat "$oldfile" | tr "$oldline" "$newline" > $file
Then i realized that tr works completely different than i thought. Its NOT a tradicional "replace this for that" function. So i guess the correct way is using sed. But:
Ive never used sedbefore. No idea how it works. Is it overkill?
If the most indicated way is really using sed, given it is so powerful, is there any elegant way to accomplish this rather than this "fetch line -> modify line -> replace oldline for newline in file" approach?
Notes:
I cant replace ".ico" globally, i know that would be a LOT easier, i must restrict the replace to the Icon line, otherwise the Description line would be changed too.
Im new to shell scripting in Linux, so im looking not only to the solution itself, but also for the "proper" way to do it. Elegant, easy to read, conventional, etc
Thanks in advance!
Edit:
Thank you guys! Here is the final script, as a reference:
#! /bin/bash
# Fix the following WARNING in ~/.xsession-errors
# gnome-session[2035]: EggSMClient-WARNING: Desktop file '/home/xxx/.config/autostart/skype.desktop' has malformed Icon key 'skype.png'(should not include extension)
file="$HOME/.config/autostart/skype.desktop"
if [ -f "$file" ] ; then
if `cat "$file" | grep "Icon=" | grep -q ".png"` ; then
sed -i.bak '/^Icon=/s/\.png$//' "$file"
cp "$file" "$PWD"
cp "${file}.bak" "$PWD"
else
echo "Nothing to fix! (maybe fixed already?)"
fi
else
echo "Skype not installed (yet...)"
fi
MUCH sleeker than my original! The only thing i regret is that sed backup does not preserve original file timestamp. But i can live with that.
And, for the record, yes, ive created this script to fix an actual "bug" in Skype packaging.
Something like the following in sed should do what you need. First we check if the line starts with Icon= and if it does then we run the s command (i.e. substitute).
sed -i '/^Icon=/s/\.ico$/.png/' file
Edit: The sed script above can also be written like this:
/^Icon=/ { # Only run the following block when this matches
s/\.ico$/.png/ # Substitute '.ico' at the end of the line with '.png'
}
See this page for more details on how to restrict when commands are run.
sed is pretty easy to deal with. Here's one way:
sed 's/^\(Icon=.*\)\.ico$/\1.png/'
By default, sed works on every line in the file one at a time. The 's/.../.../' will do a regular expression match on the first argument and replace it with the second argument. The \1 stands for everything that matched the first group, which is demarcated by the parenthesis. You have to escape the parens with \.
The above works as part of a pipeline, but you can add an '-i' flag, like this
sed -i 's/^\(Icon=.*\)\.ico$/\1.png/' input.txt
to have it replace the file input.txt in place. Don't add that until you have tested your sed script a little.
I'm trying to write a shell script that does a search and replace inside a configuration file upon start-up.
The string we're trying to replace is:
include /etc/nginx/https.include;
and we want to replace it with a commented version:
#include /etc/nginx/https.include;
The file that contains the string that we want to replace is:
/etc/nginx/app-servers.include
I'm not a Linux guru and can't seem to find the command to do this.
perl -p -i -e 's%^(include /etc/nginx/https.include;)$%#$1%' /etc/nginx/ap-servers.include
If the line might not end in the ;, use instead:
perl -p -i -e 's%^(include /etc/nginx/https.include;.*)$%#$1%' /etc/nginx/ap-servers.include
If you want to preserve the original file, add a backup extension after -i:
perl -p -i.bak -e 's%^(include /etc/nginx/https.include;)$%#$1%' /etc/nginx/ap-servers.include
Now, explaining. The -p flag means replace in-place. All lines of the file will be fed to the expression, and the result will be used as replacement. The -i flag indicates the extension of the backup file. By using it without anything, you prevent generation of backups. The -e tells Perl to get the following parameter as an expression to be executed.
Now, the expression is s%something%other%. I use % instead of the more traditional / to avoid having to escape the slashes of the path. I use parenthesis in the expression and $1 in the substituted expression for safety -- if you change one, the other will follow. Thus, %#$1% is actually the second % of s, followed by the desired #, $1 indicating the pattern inside parenthesis, and the last % of s.
HTH. HAND.
sed -i 's/foo/bar/g' config.txt
This replaces all instances of foo (case insensitive) with bar in the file config.txt
Check out sed:
sed -i -r 's|^(include /etc/nginx/https.include;)$|#\1|' /etc/nginx/app-servers.include
-i means do the substitution in-place and -r means to use extended regular expressions.
cd pathname
for y in `ls *`;
do sed "s/ABCD/DCBA/g" $y > temp; mv temp $y;
done
This script shold replace string ABCD to DCBA in all the files in pathname