How to replace a specific line in a file with a string variable using the line number bash script? - linux

Here is the contents of a target.txt file:
line1
line2
Environment=xLink=https://11111/route
line4
line5
I am trying to write a bash script that will find the number of the line containing 'https' and then replace this entire line with a new string variable obtained within the bash script, here is the bash script without the replacement line:
#!/bin/bash
x="12345"
route="/route"
x_route="${x}${route}"
x_init="Environment=xLink=https://"
new_line="${x_init}${x_route}"
echo "${new_line}"
to_replace_line_number=$(find target.txt -type f | xargs grep -n 'https' | cut -c1-2)
echo "${to_replace_line_number}"
targetfile=target.txt
echo "${targetfile}"
Invoking this script outputs the following as expected:
Environment=xLink=https://12345/route
3:
target.txt
Now, without the bash script, if I invoked:
sudo sed -i '3 c\Environment=xLink=https://12345/route' target.txt
The target.txt changes as desired to:
line1
line2
Environment=xLink=https://12345/route
line4
line5
But the goal is to automate, so I am trying to use sed command to do the job inside the bash script. So far I tried two methods, none of them worked.
Method 1:
I added the following line to the bash script:
sudo sed -i "${to_replace_line_number}s/.*/${new_line}/" ${targetfile}
When I ran the script, it didn't work and I got this error:
sed: -e expression #1, char 2: : doesn't want any addresses
Method 2:
I added the following command to the bash script:
sudo sed -i "${to_replace_line_number} c\${new_line}" ${targetfile}
When I ran the script, it didn't work and I got this error:
sed: -e expression #1, char 2: : doesn't want any addresses
What is that I am missing exactly? Any help is very much appreciated.

This is because you are taking out two characters when reading the line number.As a result, an extra ':' is popping up in the variable. Instead, take out only the one field and it should work fine.
Replace
to_replace_line_number=$(find target.txt -type f | xargs grep -n 'https' | cut -c1-2)
with
to_replace_line_number=$(find target.txt -type f | xargs grep -n 'https' | cut -c1)

\ is a special character, so when you use it in double quotes you have to escape it:
# Set example values and create a test file:
to_replace_line_number="3"
new_line="Environment=xLink=https://12345/route"
targetfile="test.txt"
printf 'line%d\n' {1..5} > "$targetfile"
# The actual command
sudo sed -i "${to_replace_line_number} c\\${new_line}" "${targetfile}"
This would make it equivalent to your manual invocation.
If you have wondered why the documentation for c appears to be weirdly formatted compared to r or y, it's because the linefeed after the \ is intentional. This is the POSIX way of doing it:
sudo sed -i "${to_replace_line_number} c\\
${new_line}" "${targetfile}"

There are a couple issues with the current code:
to_replace_line_number == 3: - NOTICE the colon (:); this is fed into the sed command like such: sed -i "3: c\.... and is generating the error message stating an issue with the 2nd character, ie, the :
as #thatotherguy has pointed out in his answer, the c\ option requires an escape character and an embedded carriage return ... or ...
Minimal changes to OPs current code:
# parse the `grep -n` by having `cut` pull everthing before the (first) `:`
$ to_replace_line_number=$(find target.txt -type f | xargs grep -n 'https' | cut -d":" -f1)
$ echo "${to_replace_line_number}"
3
# modification to #thatotherguy's `sed/c` suggestion to allow all code to go on a single line:
$ sed -i -e "${to_replace_line_number} c\\" -e "${new_line}" ${targetfile}
$ cat "${targetfile}"
line1
line2
Environment=xLink=https://12345/route
line4
line5
Instead of spawning the sub-process calls to get the line number, there are several ways sed can be used to find and replace the desired line.
One sed idea:
sed -i "s|^.*https.*$|${new_line}|" ${targetfile}
Where:
| - use pipe as sed delimiter since ${new_line} contains forward slashes
^.*https.*$ - match any line that contains the string https
${new_line} - replace the line with the contents of ${new_line}
After running the above:
$ cat target.txt
line1
line2
Environment=xLink=https://12345/route
line4
line5

Related

sed no output on no pattern match

I want sed to give me a single line output irrespective of whether the matched pattern is found and substituted, or even if there is no pattern match, with same command options.
1. echo "700K" | sed -n 's/[A-Z]//gp' // gives one output
2. echo "700" | sed -n 's/[A-Z]//gp' // no output
Is there any way in sed i can get a single output for second case without removing the "-n" option, forcing it to print the input irrespective of substitution made or not?
It is not clear for me why you need to keep the -n option but if you really do need to keep it you can use the following sed command:
echo "700" | sed -n 's/[A-Z]//g;p'
this will first make the substitution if possible then print the line.
output:
You don't need to mess with all these sed options. Use sed in it's simpliest format which will make a substitution if pattern is found:
$ echo "700K" | sed 's/[A-Z]//g'
700
$ echo "700" | sed 's/[A-Z]//g'
700
$ sed --version
sed (GNU sed) 4.4
$ sed 's/[A-Z]//g' <<<$'700\n700K\n500\n3500A'
700
700
500
3500

using sed to set a variable works on command line, but not bash script

I have looked quite a bit for answers but I am not finding any suggestions that have worked so far.
on command line, this works:
$ myvar=$( cat -n /usr/share/dict/cracklib-small | grep $myrand | sed -e "s/$myrand//" )
$ echo $myvar
$ commonness
however, inside a bash script the same exact lines just echoes out a blank line
notes - $myrand is a number, like 10340 generated with $RANDOM
cat prints out a dictionary with line numbers
grep grabs the line with $myrand in it ; e.g. 10340 commonness
sed is intended to remove the $myrand part of the line and replace it with nothing. here is my sample script
#!/bin/bash
# prints out a random word
myrand=$RANDOM
export myrand
myword=$( cat -n /path/to/dict/cracklib-small | grep myrand | sed -e "s/$myrand//g" <<<"$myword" )
echo $myword
Your command line code is running:
grep $myrand
Your script is running:
grep myrand
These are not the same thing; the latter is looking for a word that contains "myrand" within it, not a random number.
By the way -- I'd suggest a different way to get a random line. If you have GNU coreutils, the shuf tool is built-to-purpose:
myword=$(shuf -n 1 /path/to/dict/cracklib-small)
#!/bin/bash
# prints out a random word
myrand=$RANDOM
export myrand
myword=$( cat -n /path/to/dict/cracklib-small | grep myrand | sed -e "s/$myrand//g" <<<"$myword" )
echo $myword
where is the $ sign in grep myrand ?
you must put in some work before posting it here.

How to to delete a line given with a variable in sed?

I am attempting to use sed to delete a line, read from user input, from a file whose name is stored in a variable. Right now all sed does is print the line and nothing else.
This is a code snippet of the command I am using:
FILE="/home/devosion/scripts/files/todo.db"
read DELETELINE
sed -e "$DELETELINE"'d' "$FILE"
Is there something I am missing here?
Edit: Switching out the -e option with -i fixed my woes!
You need to delimit the search.
#!/bin/bash
read -r Line
sed "/$Line/d" file
Will delete any line containing the typed input.
Bear in mind that sed matches on regex though and any special characters will be seen as such.
For example searching for 1* will actually delete lines containing any number of 1's not an actual 1 and a star.
Also bear in mind that when the variable expands, it cannot contain the delimiters or the command will break or have unexpexted results.
For example if "$Line" contained "/hello" then the sed command will fail with
sed: -e expression #1, char 4: extra characters after command.
You can either escape the / in this case or use different delimiters.
Personally i would use awk for this
awk -vLine="$Line" '!index($0,Line)' file
Which searches for an exact string and has none of the drawbacks of the sed command.
You might have success with grep instead of sed
read -p "Enter a regex to remove lines: " filter
grep -v "$filter" "$file"
Storing in-place is a little more work:
tmp=$(mktemp)
grep -v "$filter" "$file" > "$tmp" && mv "$tmp" "$file"
or, with sponge (apt install moreutils)
grep -v "$filter" "$file" | sponge "$file"
Note: try to get out of the habit of using ALLCAPSVARS: one day you'll accidentally use PATH=... and then wonder why your script is broken.
I found this, it allows for a range deletion with variables:
#!/bin/bash
lastline=$(whatever you need to do to find the last line)` //or any variation
lines="1,$lastline"
sed -i "$lines"'d' yourfile
keeps it all one util.
Please try this :
sed -i "${DELETELINE}d" $FILE

Text formating - sed, awk, shell

I need some assistance trying to build up a variable using a list of exclusions in a file.
So I have a exclude file I am using for rsync that looks like this:
*.log
*.out
*.csv
logs
shared
tracing
jdk*
8.6_Code
rpsupport
dbarchive
inarchive
comms
PR116PICL
**/lost+found*/
dlxwhsr*
regression
tmp
working
investigation
Investigation
dcsserver_weblogic_
dcswebrdtEAR_weblogic_
I need to build up a string to be used as a variable to feed into egrep -v, so that I can use the same exclusion list for rsync as I do when egrep -v from a find -ls.
So I have created this so far to remove all "*" and "/" - and then when it sees certain special characters it escapes them:
cat exclude-list.supt | while read line
do
echo $line | sed 's/\*//g' | sed 's/\///g' | 's/\([.-+_]\)/\\\1/g'
What I need the ouput too look like is this and then export that as a variable:
SEXCLUDE_supt="\.log|\.out|\.csv|logs|shared|PR116PICL|tracing|lost\+found|jdk|8\.6\_Code|rpsupport|dbarchive|inarchive|comms|dlxwhsr|regression|tmp|working|investigation|Investigation|dcsserver\_weblogic\_|dcswebrdtEAR\_weblogic\_"
Can anyone help?
A few issues with the following:
cat exclude-list.supt | while read line
do
echo $line | sed 's/\*//g' | sed 's/\///g' | 's/\([.-+_]\)/\\\1/g'
Sed reads files line by line so cat | while read line;do echo $line | sed is completely redundant also sed can do multiple substitutions by either passing them as a comma separated list or using the -e option so piping to sed three times is two too many. A problem with '[.-+_]' is the - is between . and + so it's interpreted as a range .-+ when using - inside a character class put it at the end beginning or end to lose this meaning like [._+-].
A much better way:
$ sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' file
\.log
\.out
\.csv
logs
shared
tracing
jdk
8\.6\_Code
rpsupport
dbarchive
inarchive
comms
PR116PICL
lost\+found
dlxwhsr
regression
tmp
working
investigation
Investigation
dcsserver\_weblogic\_
dcswebrdtEAR\_weblogic\_
Now we can pipe through tr '\n' '|' to replace the newlines with pipes for the alternation ready for egrep:
$ sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' file | tr "\n" "|"
\.log|\.out|\.csv|logs|shared|tracing|jdk|8\.6\_Code|rpsupport|dbarchive|...
$ EXCLUDE=$(sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' file | tr "\n" "|")
$ echo $EXCLUDE
\.log|\.out|\.csv|logs|shared|tracing|jdk|8\.6\_Code|rpsupport|dbarchive|...
Note: If your file ends with a newline character you will want to remove the final trailing |, try sed 's/\(.*\)|/\1/'.
This might work for you (GNU sed):
SEXCLUDE_supt=$(sed '1h;1!H;$!d;g;s/[*\/]//g;s/\([.-+_]\)/\\\1/g;s/\n/|/g' file)
This should work but I guess there are better solutions. First store everything in a bash array:
SEXCLUDE_supt=$( sed -e 's/\*//g' -e 's/\///g' -e 's/\([.-+_]\)/\\\1/g' exclude-list.supt)
and then process it again to substitute white space:
SEXCLUDE_supt=$(echo $SEXCLUDE_supt |sed 's/\s/|/g')

Remove empty lines in a text file via grep

FILE:
hello
world
foo
bar
How can I remove all the empty new lines in this FILE?
Output of command:
FILE:
hello
world
foo
bar
grep . FILE
(And if you really want to do it in sed, then: sed -e /^$/d FILE)
(And if you really want to do it in awk, then: awk /./ FILE)
Try the following:
grep -v -e '^$'
with awk, just check for number of fields. no need regex
$ more file
hello
world
foo
bar
$ awk 'NF' file
hello
world
foo
bar
Here is a solution that removes all lines that are either blank or contain only space characters:
grep -v '^[[:space:]]*$' foo.txt
If removing empty lines means lines including any spaces, use:
grep '\S' FILE
For example:
$ printf "line1\n\nline2\n \nline3\n\t\nline4\n" > FILE
$ cat -v FILE
line1
line2
line3
line4
$ grep '\S' FILE
line1
line2
line3
line4
$ grep . FILE
line1
line2
line3
line4
See also:
How to remove empty/blank lines (including spaces) in a file in Unix?
How to remove blank lines from a file in shell?
With sed: Delete empty lines using sed
With awk: Remove blank lines using awk
Simplest Answer -----------------------------------------
[root#node1 ~]# cat /etc/sudoers | grep -v -e ^# -e ^$
Defaults !visiblepw
Defaults always_set_home
Defaults match_group_by_gid
Defaults always_query_group_plugin
Defaults env_reset
Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL
[root#node1 ~]#
Try this: sed -i '/^[ \t]*$/d' file-name
It will delete all blank lines having any no. of white spaces (spaces or tabs) i.e. (0 or more) in the file.
Note: there is a 'space' followed by '\t' inside the square bracket.
The modifier -i will force to write the updated contents back in the file. Without this flag you can see the empty lines got deleted on the screen but the actual file will not be affected.
grep '^..' my_file
example
THIS
IS
THE
FILE
EOF_MYFILE
it gives as output only lines with at least 2 characters.
THIS
IS
THE
FILE
EOF_MYFILE
See also the results with grep '^' my_file outputs
THIS
IS
THE
FILE
EOF_MYFILE
and also with grep '^.' my_file outputs
THIS
IS
THE
FILE
EOF_MYFILE
Try ex-way:
ex -s +'v/\S/d' -cwq test.txt
For multiple files (edit in-place):
ex -s +'bufdo!v/\S/d' -cxa *.txt
Without modifying the file (just print on the standard output):
cat test.txt | ex -s +'v/\S/d' +%p +q! /dev/stdin
Perl might be overkill, but it works just as well.
Removes all lines which are completely blank:
perl -ne 'print if /./' file
Removes all lines which are completely blank, or only contain whitespace:
perl -ne 'print if ! /^\s*$/' file
Variation which edits the original and makes a .bak file:
perl -i.bak -ne 'print if ! /^\s*$/' file
If you want to know what the total lines of code is in your Xcode project and you are not interested in listing the count for each swift file then this will give you the answer. It removes lines with no code at all and removes lines that are prefixed with the comment //
Run it at the root level of your Xcode project.
find . \( -iname \*.swift \) -exec grep -v '^[[:space:]]*$' \+ | grep -v -e '//' | wc -l
If you have comment blocks in your code beginning with /* and ending with */ such as:
/*
This is an comment block
*/
then these will get included in the count. (Too hard).

Resources