Linux SED command in Bash script strips ' from string - linux

I have been putting together a bash script and within it I need to update a file with some neccasary information. The command I am using is below:
sudo sed '
/end/ a\
First line to update\
param 1 'var1'\
param 2 'var2'\
param 3 'var3'\
param 4 'var4'\
end\
' TestFile >TestFileNew
Now this should update the file with the data and should look like this:
end
First line to update
param 1 'var1'
param 2 'var2'
param 3 'var3'
param 4 'var4'
end
The file does get created and the data is in it however it seems to strip the ' symbols from the text and I don't want this to happen, can anyone please help?? An example of what is actually getting produced is below:
end
First line to update
param 1 var1
param 2 var2
param 3 var3
param 4 var4
end

Use "" as the outer quotes:
sudo sed "
/end/ a\\
First line to update\\
param 1 'var1'\\
param 2 'var2'\\
param 3 'var3'\\
param 4 'var4'\\
end\\
" TestFile >TestFileNew

Looks like Igor has already provided an answer that may work for the single quotes.
Regarding the file creation, remember that sudo affects the program (sed) you're running as its option. It does not affect redirection, which is handled by your shell. So if you don't have permission as a user to write TestFileNew, sudo won't help you, the way you're using it above.
You might be better off creating your output somewhere else, then using sudo to move it into place.
sudo sed "/end/ ..." TestFile > /tmp/TestFileNew
sudo mv /tmp/TestFileNew ./TestFileNew
Alternately, this whole script could be run with sudo... I.e. /path/to/myscript is:
#!/bin/bash
sed "/end/ ..." TestFile > TestFileNew
then:
$ sudo /path/to/myscript
Then sudo is running bash instead of sed, and the privileged bash instance is responsible for handling redirection within the script.

In this case, it would probably be simpler to use sed's r command:
$ cat > /tmp/repl << EOF
First line to update
param 1 'var1'
param 2 'var2'
param 3 'var3'
param 4 'var4'
end
EOF
$ sed '/end/r/tmp/repl' TestFile
$ rm /tmp/repl
(I've ignored the sudo/redirect issue, but you probably want to so sudo sh -c 'sed /end/r/tmp/repl/ TestFile > TestFileNew'

Sometimes you have to deal with two types of quotes (html and javascript, for example). Then it is very convenient to put sed commands in a script:
cat >FILE
end
cat >script.sed
/end/ a\
First line to update\
param 1 'var1'\
param 2 'var2'\
param 3 'var3'\
param 4 'var4'\
end
sed -f script.sed FILE
end
First line to update
param 1 'var1'
param 2 'var2'
param 3 'var3'
param 4 'var4'
end

Related

How to loop names with spaces in bash script?

I'm having a python list output:
names=[Raj Sonakshi, Getendar, Raghu Varan (Mr)]
I run python script using the below bash command
arr=$(python names.py)
Output I got is:
Raj Sonakshi Getendar Raghu Varan (Mr)
When I run for loop each word is printing instead of full name:
for i in $arr;
do
echo $i
done
Output:
Raj
Sonakshi
.....
.....
Expected Output is :
Raj Sonakshi
Put the names in the array in quotes
names=["Raj Sonakshi", "Getendar", "Raghu Varan (Mr)"]
Not sure what your python script's output looks like exactly; the "output" in your question seems to be after bash has turned it into one long string. And you probably don't want it to be a string (as you are doing with arr=$()), but rather you probably want to use a bash array:
declare -a arr
readarray -t arr <<< "$(python names.py)"
for i in "${arr[#]}"; do
echo "$i"
done

How do I add a new column with a specific word to a file in linux?

I have a file with one column containing 2059 ID numbers.
I want to add a second column with the word 'pop1' for all the 2059 ID numbers.
The second column will just mean that the ID number belongs to population 1.
How can I do this is linux using awk or sed?
The file currently has one column which looks like this
45958
480585
308494
I want it to look like:
45958 pop1
480585 pop1
308494 pop1
Maybe not the most elegant solution, and it doesn't use sed or awk, but I would do that:
while read -r line; do echo ""$line" pop1" >> newfile; done < test
This command will append stuff in the file 'newfile', so be sure that it's empty or it doesn't exist before executing the command.
Here is the resource I used, on reading a file line by line : https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/
A Perl solution.
$ perl -lpi -e '$_ .= " pop1"' your-file-name
Command line options:
-l : remove newline from input and replace it on output
-p : put each line of input into $_ and print $_ at the end of each iteration
-i : in-place editing (overwrite the input file)
-e : run this code for each line of the input
The code ($_ .= " pop1") just appends your string to the input record.

passing command line arguments to awk in shell script

Background information:
I am trying to write a small shell script, which searches a pattern (string) in a .fas-file and prints the line and position, where the pattern was found. The following code snippet works, when I call the shell script:
Script (search.sh):
#!/bin/bash
awk 's=index($0, "CAATCTCC"){print "line=" NR, "start position=" s}' 100nt_upstream_of_mTSS.fas
Command line call:
$ ./search.sh
First problem:
When I change the script to:
awk 's=index($0, "CAATCTCC"){print "line=" NR, "start position=" s}'
and do the following command line call in my bash:
$ ./search.sh 100nt_upstream_of_mTSS.fas
"nothing" happens (something is running, but it takes way too long and no results come up, so terminate the process).
Worth to know:
I am in the directory, where search.sh is located
the file 100nt_upstream_of_mTSS.fas is located there, too
search.sh is executable
I might be "screen blind", but I can't find the reason, why I am unable to pass a command line argument to my script.
Solution - see comments
Note: Only the first occurence of the pattern in a line is found this way.
Second problem:
Furthermore, I would like to make the motif (the string) I search for variable. I tried this:
Script:
#!/bin/bash
FILE=$1
MOTIF=$2
awk 's=index($0, "$MOTIF"){print "line=" NR, "start position=" s}' "$FILE"
Command line call:
$ ./search.sh 100nt_upstream_of_mTSS.fas CAATCTCC
Idea: First command-line argument worked and was substitued correctly. Why is the second one not substituted correctly?
Solution so far:
Script:
#!/bin/bash
file=$1
awk -v s="$2" 'i=index($0, s){print "line: " NR, "pos: " i}' "$file"
Testing:
Testfile (test.txt):
1 GAGAGAGAGA
2 CTCTCTCTCT
3 TATATATATA
4 CGCGCGCGCG
5 CCCCCCCCCC
6 GGGGGGGGGG
7 AAAAAAAAAA
8 TTTTTTTTTT
9 TGATTTTTTT
10 CCCCCCCCGA
$ ./search.sh test.txt GA
will print:
line: 1 pos: 1
line: 4 pos: 2
line: 6 pos: 1
line: 9 pos: 2
line: 10 pos: 9
This script will print line and first match position in the line of only the first character of my pattern. How do I manage to have all results printed and the full pattern being used?
As far as I understood you want to pass the Input_file(file which you want to process by script) as an argument, if this is the case then following may help you in same.
cat search.sh
#!/bin/bash
variable=$1
awk 's=index($0, "CAATCTCC"){print "line=" NR, "start position=" s}' "$variable"
./search.sh 100nt_upstream_of_mTSS.fas

How do I insert text to the 1st line of a file using sed?

Hi I'm trying to add text to the 1st line of a file using sed
so far iv'e tried
#!/bin/bash
touch test
sed -i -e '1i/etc/example/live/example.com/fullchain.pem;\' test
And this dosn't work
also tried
#!/bin/bash
touch test
sed -i "1i ssl_certificate /etc/example/live/example.com/fullchain.pem;" test
this dosn't seem to work either
oddly when I try
#!/bin/bash
touch test
echo "ssl_certificate /etc/example/live/example.com/fullchain.pem;" > test
I get the 1st line of text to appear when i use cat test
but as soon as i type sed -i "2i ssl_certificate_key /etc/example/live/example.com/privkey.pem;"
I can't see the information that i sould do on line 2 this being ssl_certificate_key /etc/example/live/example.com/privkey.pem;
so my question to summerise
Can text be inserted into the 1st line of a newly created file using sed?
If yes whats the best way of inserting text after the 1st line of text?
Suppose you have a file like this:
one
two
Then to append to the first line:
$ sed '1 s_$_/etc/example/live/example.com/fullchain.pem;_' file
one/etc/example/live/example.com/fullchain.pem;
two
To insert before the first line:
$ sed '1 i /etc/example/live/example.com/fullchain.pem;' file
/etc/example/live/example.com/fullchain.pem;
one
two
Or, to append after the first line:
$ sed '1 a /etc/example/live/example.com/fullchain.pem;' file
one
/etc/example/live/example.com/fullchain.pem;
two
Note the number 1 in those sed expressions - that's called the address in sed terminology. It tells you on which line the command that follows is to operate.
If your file doesn't contain the line you're addressing, the sed command won't get executed. That's why you can't insert/append on line 1, if your file is empty.
Instead of using stream editor, to append (to empty files), just use a shell redirection >>:
echo "content" >> file
Your problem stems from the fact that sed cannot locate the line you're telling it to write at, for example:
touch test
sed -i -e '1i/etc/example/live/example.com/fullchain.pem;\' test
attempts to write to insert at the line 1 of test, but that line doesn't exist at this point. If you've created your file as:
echo -en "\n" > test
sed -i '1i/etc/example/live/example.com/fullchain.pem;\' test
it would not complain, but you'd be having an extra line. Similarly, when you call:
sed -i "2i ssl_certificate_key /etc/example/live/example.com/privkey.pem;"
you're telling sed to insert the following data at the line 2 which doesn't exist at that point so sed doesn't get to edit the file.
So, for the initial line or the last line in the file, you should not use sed because simple > and >> stream redirects are more than enough.
Your command will work if you make sure the input file has at least one line:
[ "$(wc -l < test)" -gt 0 ] || printf '\n' >> test
sed -i -e '1 i/etc/example/live/example.com/fullchain.pem;\' test
To insert text to the first line and put the rest on a new line using sed on macOS this worked for me
sed -i '' '1 i \
Insert
' ~/Downloads/File-path.txt
First and Last
I would assume that anyone who searched for how to insert/append text to the beginning/end of a file probably also needs to know how to do the other also.
cal | \
gsed -E \
-e '1i\{' \
-e '1i\ "lines": [' \
-e 's/(.*)/ "\1",/' \
-e '$s/,$//' \
-e '$a\ ]' \
-e '$a\}'
Explanation
This is cal output piped to gnu-sed (called gsed on macOS installed via brew.sh) with extended RegEx (-E) and 6 "scripts" applied (-e) and line breaks escaped with \ for readability. Scripts 1 & 2 use 1i\ to "at line 1, insert". Scripts 5 & 6 use $a\ to "at line <last>, append". I vertically aligned the text outputs to make the code represent what is expected in the result. Scripts 3 & 4 do substitutions (the latter applying only to "line <last>"). The result is converting command output to valid JSON.
output
{
"lines": [
" October 2019 ",
"Su Mo Tu We Th Fr Sa ",
" 1 2 3 4 5 ",
" 6 7 8 9 10 11 12 ",
"13 14 15 16 17 18 19 ",
"20 21 22 23 24 25 26 ",
"27 28 29 30 31 ",
" "
]
}
For help getting this to work with the macos/BSD version of sed, see my answer here.

How to change characters in specific location using shell variable

I have a file which I wish to programmatically alter to set different parameters:
$ cat ex.dat
line 1
here's another line
this is the number I want to change --> 1e-11
more lines
I want to change 1e-11 to the values 5., 10, and 20.. Something like this:
mkd ()
{
mkdir $1;
cd $1
}
for j in 1 2 3 4 5 ; do
mkd WS$j
cp ../ex.dat .
sed -ie '3s/1e-11/${j}./' ex.dat
cd ..
done
My approach isn't working. How can I replace 1e-11 with $j's value?
$ cat WS3/ex.dat
line 1
here's another line
this is the number I want to change --> ${j}.
more lines
Try:
sed -ie '3s/1e-11/'"${j}"'./' ex.dat
Inside single quotes the shell doesn't replace variables by their values.
You can use double quotes to expand ${j} in your sed, and $j without curly braces should work just fine before a dot.
sed -ie "3s/1e-11/$j./" ex.dat

Resources