I want to add a text to the end of the first line of a file using a bash script.
The file is /etc/cmdline.txt which does not allow line breaks and needs new commands seperated by a blank, so text i want to add realy needs to be in first line.
What i got so far is:
line=' bcm2708.w1_gpio_pin=20'
file=/boot/cmdline.txt
if ! grep -q -x -F -e "$line" <"$file"; then
printf '%s' "$line\n" >>"$file"
fi
But that appends the text after the line break of the first line, so the result is wrong.
I either need to trim the file contend, add my text and a line feed or somehow just add it to first line of file not touching the rest somehow, but my knowledge of bash scripts is not good enough to find a solution here, and all the examples i find online add beginning/end of every line in a file, not just the first line.
This sed command will add 123 to end of first line of your file.
sed ' 1 s/.*/&123/' yourfile.txt
also
sed '1 s/$/ 123/' yourfile.txt
For appending result to the same file you have to use -i switch :
sed -i ' 1 s/.*/&123/' yourfile.txt
This is a solution to add "ok" at the first line on /etc/passwd, I think you can use this in your script with a little bit of 'tuning' :
$ awk 'NR==1{printf "%s %s\n", $0, "ok"}' /etc/passwd
root:x:0:0:root:/root:/bin/bash ok
To edit a file, you can use ed, the standard editor:
line=' bcm2708.w1_gpio_pin=20'
file=/boot/cmdline.txt
if ! grep -q -x -F -e "$line" <"$file"; then
ed -s "$file" < <(printf '%s\n' 1 a "$line" . 1,2j w q)
fi
ed's commands:
1: go to line 1
a: append (this will insert after the current line)
We're in insert mode and we're inserting the expansion of $line
.: stop insert mode
1,2j join lines 1 and 2
w: write
q: quit
This can be used to append a variable to the first line of input:
awk -v suffix="$suffix" '{print NR==1 ? $0 suffix : $0}'
This will work even if the variable could potentially contain regex formatting characters.
Example:
suffix=' [first line]'
cat input.txt | awk -v suffix="$suffix" '{print NR==1 ? $0 suffix : $0}' > output.txt
input.txt:
Line 1
Line 2
Line 3
output.txt:
Line 1 [first line]
Line 2
Line 3
Related
I'm making a script (in vim) that goes through a file and looks for a specific line. If the line matches, I want it to be moved to the top of the file.
The file looks something like this:
6596628c9cbab49b80d6a07d0304377768f5114e7f8b21edffa820aab1c508be, ./favicon.ico
150a04dd76f733e5ef05ece49de115d05f71efa8e73025e015dd4e0fb3217553, ./about.php
28acfc4b0d378c22a2c0e4913cae3d15aef9b21938de81be92f74aef85b0cc0e, ./info.php
67976bbbd1b62a00d454da3a9f95e72d97d0fd156d4c65a12707f2602cbea582, ./missing.php
6ed318718f4cc617c82121db7cde54188eac6f89c355f0bfe9d198218de7fffc, ./browse.php
abd277cc3453be980bb48cbffe9d1f7422ca1ef4bc0b7d035fda87cea4d55cbc, ./composer.phar
73ac79eccac12120dc601cd6cce1282a1d8a920d440d3d1141d257db1ed4b0f0, ./search.php
f412aabd74f4c99bd32c5e534132c565f52c2bd32fbf7f629eb5a4495ac46351, ./index.php
c2d49a4873088fbe635d8653494f7f1425b6ad9f55d63ee4de52170d8a8d01b8, ./content/style.css
18e7d61367d80bc125b309ac002bb3946c5e7ba419ef59537afc939eff799dfd, ./content/logo.png
d8da15f62d55641320f7e7c21d9be86db6d81f7667bbd35c738b4c917cad3ce9, ./robots.txt
How would I be able to move the content on line 8 (index.php) to the top so it looks like this:
f412aabd74f4c99bd32c5e534132c565f52c2bd32fbf7f629eb5a4495ac46351, ./index.php
6596628c9cbab49b80d6a07d0304377768f5114e7f8b21edffa820aab1c508be, ./favicon.ico
150a04dd76f733e5ef05ece49de115d05f71efa8e73025e015dd4e0fb3217553, ./about.php
28acfc4b0d378c22a2c0e4913cae3d15aef9b21938de81be92f74aef85b0cc0e, ./info.php
67976bbbd1b62a00d454da3a9f95e72d97d0fd156d4c65a12707f2602cbea582, ./missing.php
6ed318718f4cc617c82121db7cde54188eac6f89c355f0bfe9d198218de7fffc, ./browse.php
abd277cc3453be980bb48cbffe9d1f7422ca1ef4bc0b7d035fda87cea4d55cbc, ./composer.phar
73ac79eccac12120dc601cd6cce1282a1d8a920d440d3d1141d257db1ed4b0f0, ./search.php
c2d49a4873088fbe635d8653494f7f1425b6ad9f55d63ee4de52170d8a8d01b8, ./content/style.css
18e7d61367d80bc125b309ac002bb3946c5e7ba419ef59537afc939eff799dfd, ./content/logo.png
d8da15f62d55641320f7e7c21d9be86db6d81f7667bbd35c738b4c917cad3ce9, ./robots.txt
(The file contains about 9000 lines)
How can this be done most efficiently?
I think can use sed to do the operations needed:
This command will be used to get the content in the file at specific line
sed -n "$LINE_NUMBER"p "$FILENAME"
This command will be used to delete the content of the file at specified line
sed -i.bak -e "$LINE_NUMBER"'d' "$FILENAME"
And finally, this command will be used to append a string to the top of a file
sed -i -e "1i${LINE_CONTENT}" "$FILENAME"
Combine the 3 commands together, you can perform the operation mentioned in your question (test.log is the file that contain your sample input)
FILENAME="test.log"
LINE_NUMBER=8
LINE_CONTENT=$(sed -n "$LINE_NUMBER"p "$FILENAME")
sed -i.bak -e "$LINE_NUMBER"'d' "$FILENAME"
sed -i -e "1i${LINE_CONTENT}" "$FILENAME"
With ed, this will do in-place editing as well:
printf '8m0\nwq\n' | ed -s ip.txt -
Or, if you don't know line number:
printf '/index\.php/m0\nwq\n' | ed -s ip.txt -
Supposing your file name is test.txt, you can try this:
(Please backup your file first)
grep "./index.php" test.txt > out && grep -v "./index.php" test.txt >> out && mv out test.txt
To do this job reading the input file just once without reading the whole file into memory, and using any awk in any shell on every Unix box all you need is either of these:
awk 'f{print; next} NR==8{print $0 buf; f=1} {buf=buf ORS $0}' file
awk 'f{print; next} $2=="./index.php"{print $0 buf; f=1} {buf=buf ORS $0}' file
If you want to move the 8th line to the top:
$ awk 'f{print; next} NR==8{print $0 buf; f=1} {buf=buf ORS $0}' file
f412aabd74f4c99bd32c5e534132c565f52c2bd32fbf7f629eb5a4495ac46351, ./index.php
6596628c9cbab49b80d6a07d0304377768f5114e7f8b21edffa820aab1c508be, ./favicon.ico
150a04dd76f733e5ef05ece49de115d05f71efa8e73025e015dd4e0fb3217553, ./about.php
28acfc4b0d378c22a2c0e4913cae3d15aef9b21938de81be92f74aef85b0cc0e, ./info.php
67976bbbd1b62a00d454da3a9f95e72d97d0fd156d4c65a12707f2602cbea582, ./missing.php
6ed318718f4cc617c82121db7cde54188eac6f89c355f0bfe9d198218de7fffc, ./browse.php
abd277cc3453be980bb48cbffe9d1f7422ca1ef4bc0b7d035fda87cea4d55cbc, ./composer.phar
73ac79eccac12120dc601cd6cce1282a1d8a920d440d3d1141d257db1ed4b0f0, ./search.php
c2d49a4873088fbe635d8653494f7f1425b6ad9f55d63ee4de52170d8a8d01b8, ./content/style.css
18e7d61367d80bc125b309ac002bb3946c5e7ba419ef59537afc939eff799dfd, ./content/logo.png
d8da15f62d55641320f7e7c21d9be86db6d81f7667bbd35c738b4c917cad3ce9, ./robots.txt
If you want to move the line containing "index" to the top:
$ awk 'f{print; next} $2=="./index.php"{print $0 buf; f=1} {buf=buf ORS $0}' file
f412aabd74f4c99bd32c5e534132c565f52c2bd32fbf7f629eb5a4495ac46351, ./index.php
6596628c9cbab49b80d6a07d0304377768f5114e7f8b21edffa820aab1c508be, ./favicon.ico
150a04dd76f733e5ef05ece49de115d05f71efa8e73025e015dd4e0fb3217553, ./about.php
28acfc4b0d378c22a2c0e4913cae3d15aef9b21938de81be92f74aef85b0cc0e, ./info.php
67976bbbd1b62a00d454da3a9f95e72d97d0fd156d4c65a12707f2602cbea582, ./missing.php
6ed318718f4cc617c82121db7cde54188eac6f89c355f0bfe9d198218de7fffc, ./browse.php
abd277cc3453be980bb48cbffe9d1f7422ca1ef4bc0b7d035fda87cea4d55cbc, ./composer.phar
73ac79eccac12120dc601cd6cce1282a1d8a920d440d3d1141d257db1ed4b0f0, ./search.php
c2d49a4873088fbe635d8653494f7f1425b6ad9f55d63ee4de52170d8a8d01b8, ./content/style.css
18e7d61367d80bc125b309ac002bb3946c5e7ba419ef59537afc939eff799dfd, ./content/logo.png
d8da15f62d55641320f7e7c21d9be86db6d81f7667bbd35c738b4c917cad3ce9, ./robots.txt
To modify the original file if you have GNU awk:
awk -i inplace '...script...' file
or with any awk:
awk '...script...' file > tmp && mv tmp file
This might work for you (GNU sed):
sed -E '1,8{H;8!d;p;x;s/^(.)(.*)\1.*/\2/}' file
Create a range of lines in the hold space from the first to the line number to be inserted at the start of the file (in the example above line 8).
When the current line is the desired line, print it and then swap to the hold space, remove the newline introduced at the start of the range and the last line added (the desired line) and print the remainder too.
All other lines will be printed as normal.
Alternative:
sed -E '1{:a;N;8!ba;s/(.*)(\n)(.*)/\3\2\1/}' file
Yet another quirky solution:
sed '1,7{H;d};8{p;x;D}' file
I would like to match all lines from a file containing a word, and take all lines under until coming two two newline characters in a row.
I have the following sed code to cut and paste specific lines, but not subsequent lines:
sed 's|.*|/\\<&\\>/{w results\nd}|' teststring | sed -file.bak -f - testfile
How could I modify this to take all subsequent lines?
For example, say I wanted to match lines with 'dog', the following should take the first 3 lines of the 5:
The best kind of an animal is a dog, for sure
-man's best friend
-related to wolves
Racoons are not cute
Is there a way to do this?
This should do:
awk '/dog/ {f=1} /^$/ {f=0} f {print > "new"} !f {print > "tmp"}' file && mv tmp file
It will set f to true if word dog is found, then if a blank line is found set f to false.
If f is true, print to new file.
If f is false, print to tmp file.
Copy tmp file to original file
Edit: Can be shorten some:
awk '/dog/ {f=1} /^$/ {f=0} {print > (f?"new":"tmp")}' file && mv tmp file
Edit2: as requested add space for every section in the new file:
awk '/dog/ {f=1;print ""> "new"} /^$/ {f=0} {print > (f?"new":"tmp")}' file && mv tmp file
If the original files does contains tabs or spaces instead of just a blank line after each dog section, change from /^$/ to /^[ \t]*$/
This might work for you (GNU sed):
sed 's|.*|/\\<&\\>/ba|' stringFile |
sed -f - -e 'b;:a;w resultFile' -e 'n;/^$/!ba' file
Build a set of regexps from the stringFile and send matches to :a. Then write the matched line and any further lines until an empty line (or end of file) to the resultFile.
N.B. The results could be sent directly to resultFile,using:
sed 's#.*#/\\<&\\>/ba#' stringFile |
sed -nf - -e 'b;:a;p;n;/^$/!ba' file > resultFile
To cut the matches from the original file use:
sed 's|.*|/\\<&\\>/ba|' stringFile |
sed -f - -e 'b;:a;N;/\n\s*$/!ba;w resultFile' -e 's/.*//p;d' file
Is this what you're trying to do?
$ awk -v RS= '/dog/' file
The best kind of an animal is a dog, for sure
-man's best friend
-related to wolves
Could you please try following.
awk '/dog/{count="";found=1} found && ++count<4' Input_file > temp && mv temp Input_file
i have a file created in windows using notepad:
26453215432460
23543265235421
38654365876325
12354152435243
I have a script which will read every line, and create a command like below in other file for every line and will not consider blank lines:
CRE:EQU,264532154324600,432460,1;
Now if I save my input file after hitting enter after the last line of number 12354152435243, then the output file consists the command above corresponding to all numbers(including the last 12354152435243:
CRE:EQU,264532154324600,432460,1;
CRE:EQU,235432652354210,235421,1;
CRE:EQU,386543658763250,876325,1;
CRE:EQU,123541524352430,435243,1;
but if I save the file, without hitting enter after the last number is keyed in i.e after this 12354152435243, then after the script executes, I don't see the output file have the command for the last number:
CRE:EQU,264532154324600,432460,1;
CRE:EQU,235432652354210,235421,1;
CRE:EQU,386543658763250,876325,1;
Can somebody explain the error in the code:
while read LINE
do
[ -z "$LINE" ] && continue
IMEI=`echo $LINE | sed 's/ //g' | sed -e 's/[^ -~]//g'`
END_SERIAL=`echo $IMEI | cut -c9- | sed 's/ //g' | sed -e 's/[^ -~]//g'`
echo "CRE:EQU,${IMEI}0,${END_SERIAL},${list},,${TODAY};" >> /apps/ins/list.out
done < "${FILE_NAME}"
kindly help
Use
grep . "${FILE_NAME}" | while read LINE
or
while read LINE
do
....
done < <(grep . "${FILE_NAME}")
The grep is less sensible to line-ending, and you will get empty-line skip for a free... :)
Honestly, never tried windows, all above is OK for unix...
EDIT Explanation:
make the next file:
echo -n -e 'line\n\nanother\nno line ending here>' >file.txt
the file contains 4 lines (although the last "line" is not a "correct" one)
line
another
no line ending here>
Usual shell routines, as read or wc looking for line ending. Therefore,
$ wc -l file.txt
3 file.txt
When you grepping for '' (empty string) the grep returns every line where found the string, so
$ grep '' file.txt
prints
line
another
no line ending here>
When grep prints out the found lines - ensures than one `\n' exists at the end, so
$ grep '' file.txt | wc -l
returns
4
therefore, for these situations, is better to use grep with -c (count) and not wc.
$ grep -c '' file.txt
4
Now, the . dot. The dot mean any character. So, when you grepping for a ., you get all lines what contain at least one character. And therefore, it will skip all lines what doesn't contain any character = skips empty lines. So,
$ grep . file.txt
line
another
no line ending here>
again, with added line ending to the last line (and skipped the empty line). Remember, the (space) is character too, so when the line contains only one space it is NOT EMPTY. Counting non-empty lines
$ grep . file.txt | wc -l
3
or faster
$ grep -c . file.txt
3
If you do a help read it says for -d delim continue until the first character of DELIM is read, rather than newline.
So read will continue until it hits a \n or if you specify -d delim.
So you probably need to change the delim or you can try read -e
read will read untill a new line has been found and when it finds a new line, it will return you the line. But if the file ends without a new line, read treats this as an error. So, even if read has set the returning variable with the line read till now, the return code of read is set to indicate an error. Now the while read ... this loop body will only executes if the command executes with a success which is not a case here. Thus you miss the last line.
For overcoming this , you can change the condition to also check the returning variable is empty or not. Hence the condition succeeds even if read fails, as the variable is already set till the end of the file.
This is not related to line ending in different OS, i mean it's somehow related, but the exact root cause is always the read fails to find a new line at the end of the line/file, and the last line is missing the loop body.
Below is an example
[[bash_prompt$]]$ echo -ne 'hello\nthere' > log
[[bash_prompt$]]$ while read line; do echo $line; done < log
hello
[[bash_prompt$]]$ while read line || [ -n "$line" ]; do echo $line; done < log
hello
there
[[bash_prompt$]]$
read nees the end of line to read the input. Try
echo -n $'a\nb' | while read x ; do echo $x ; done
It only prints a.
To prevent script not reading last line of a file:
cat "somefile" | { cat ; echo ; } | while read line; do echo $line; done
Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/
I have a big file 150GB CSV file and I would like to remove the first 17 lines and the last 8 lines. I have tried the following but seems that's not working right
sed -i -n -e :a -e '1,8!{P;N;D;};N;ba'
and
sed -i '1,17d'
I wonder if someone can help with sed or awk, one liner will be great?
head and tail are better for the job than sed or awk.
tail -n+18 file | head -n-8 > newfile
awk -v nr="$(wc -l < file)" 'NR>17 && NR<(nr-8)' file
All awk:
awk 'NR>y+x{print A[NR%y]} {A[NR%y]=$0}' x=17 y=8 file
Try this :
sed '{[/]<n>|<string>|<regex>[/]}d' <fileName>
sed '{[/]<adr1>[,<adr2>][/]d' <fileName>
where
/.../=delimiters
n = line number
string = string found in in line
regex = regular expression corresponding to the searched pattern
addr = address of a line (number or pattern )
d = delete
Refer this link
LENGTH=`wc -l < file`
head -n $((LENGTH-8)) file | tail -n $((LENGTH-17)) > file
Edit: As mtk posted in comment this won't work. If you want to use wc and track file length you should use:
LENGTH=`wc -l < file`
head -n $((LENGTH-8)) file | tail -n $((LENGTH-8-17)) > file
or:
LENGTH=`wc -l < file`
head -n $((LENGTH-8)) file > file
LENGTH=`wc -l < file`
tail -n $((LENGTH-17)) file > file
What makes this solution less elegant than that posted by choroba :)
I learnt this today for the shell.
{
ghead -17 > /dev/null
sed -n -e :a -e '1,8!{P;N;D;};N;ba'
} < my-bigfile > subset-of
One has to use a non consuming head, hence the use of ghead from the GNU coreutils.
Similar to Thor's answer, but a bit shorter:
sed -i '' -e $'1,17d;:a\nN;19,25ba\nP;D' file.txt
The -i '' tells sed to edit the file in place. (The syntax may be a bit different on your system. Check the man page.)
If you want to delete front lines from the front and tail from the end, you'd have to use the following numbers:
1,{front}d;:a\nN;{front+2},{front+tail}ba\nP;D
(I put them in curly braces here, but that's just pseudocode. You'll have to replace them by the actual numbers. Also, it should work with {front+1}, but it doesn't on my machine (macOS 10.12.4). I think that's a bug.)
I'll try to explain how the command works. Here's a human-readable version:
1,17d # delete lines 1 ... 17, goto start
:a # define label a
N # add next line from file to buffer, quit if at end of file
19,25ba # if line number is 19 ... 25, goto start (label a)
P # print first line in buffer
D # delete first line from buffer, go back to start
First we skip 17 lines. That's easy. The rest is tricky, but basically we keep a buffer of eight lines. We only start printing lines when the buffer is full, but we stop printing when we reach the end of the file, so at the end, there are still eight lines left in the buffer that we didn't print - in other words, we deleted them.
Suppose I have a configuration file that can be in one of two format below (short example, but basically the first format is a line that is too long that you have to use a line continuation character, while the second format is just simply a long line without the line continuation)
data1=x data2=y data3=z \
datakey
second format
data=1 data2=y data3=z datakey
I want to match the exact line data1=x data2=y data3=x datakey for both situation. Is there simple way of doing that?
read interprets \ as the line continuation character:
while read line ; do
if [[ $line == 'data=1 data2=y data3=z datakey' ]] ; then
echo "$line"
fi
done
I would use sed to create an output without the ending \:
sed -e ':begin;/\\$/{N;bbegin};s/\\\n//g' your_file
Then you could grep it:
sed -e ':begin;/\\$/{N;bbegin};s/\\\n//g' your_file | grep your_pattern
You can even do this all in sed:
sed -n -e ':begin;/\\$/{N;bbegin};s/\\\n//g;/your_pattern/p' your_file
UPDATE:
To explain above:
:begin sets a tag to which I can branch (goto) with the b command.
/\\$/{N;bbegin} if the current line ends with a \ (/\\$/), append the next line to the buffer (N) and goto begin (bbegin).
Then, when the lines does not end with a \, remove all the \ and the line break (thes/\\n//g`).
Then -n option tells sed no to print the line at the end of the script.
/your_pattern/p prints the line if it matches your_pattern.
UPDATE2:
We could even do better and show the original lines of your file:
sed -n -e ':begin;/\\$/{N;bbegin};h;s/\\\n//g;/your_pattern/{g;p}' your_file
What this does is before removing the \ and the line break, it saves the data in the hold space (h) and if the line matches, it prints the data that was saved (g copies the hold space to the pattern space that is printed).
Maybe grep is not a best tool for problems like that.
You could join all lines ends with \ and then grep that output as usual:
Suppose you have a file:
$> cat text
1
2
fasdfasdf
data1=x data2=y data3=z \
datakey
fasfd
sdf
So you can join all lines ends with \:
$> awk '{line = $0}; /.*\\/ {split($0,tmp,"\\"); line = tmp[1]; getline; line = line $0}; { print line }' text
1
2
fasdfasdf
data1=x data2=y data3=z datakey
fasfd
sdf
awk -v i="data1=x data2=y data3=z datakey" '{x=x" "$0}END{y=match(x," "i);if(y) print "yes its a match"}' temp
tested below:
> cat temp
data1=x data2=y data3=z
datakey
> awk -v i="data1=x data2=y data3=z datakey" '{x=x" "$0}END{y=match(x," "i);if(y) print "yes its a match"}' temp
yes its a match
>
I like the sed example above, and didn't notice the awk ones already
submitted. Here is awk (nawk) version that keeps the formatting and deals
with multiple continuation lines and end-of-file.
nawk -v re="search-string" \
'{ls=ls""$0;lp=lp""$0}
/\\$/{ls=substr(ls,1,length(ls)-1);lp=lp"\n";next}
ls~re{print lp}
{lp=ls=""}
END{if (ls ~ re)print substr(lp,1,length(lp)-1)}' input-file
Explanation:
re=search-string sets the regular expression to look for.
ls=ls""$0 Concatenate new line to ls (line-search), also store
original line to lp (line-print) as per sed example above.
/\\$/ checks to see if input has continuation character and
if it does substr() removes the extra character from ls, lp has a
newline appended to keep original formatting. Lastly next
causes awk to read the next line and start from the first rule.
ls~i searches line-search for re and if it matches prints lp,
could add a switch to print ls if preferred.
Lastly (not penultimately) reset ls & lp when no continuation line.
END is special rule, used here to detect that ls still has a
value assigned, meaning the continuation line is at the EOF. Searched for the regex, when it
matches it cuts off the additional newline added above.
Now, no one would write '\' at the end of a line would they?