I am trying to use sed to reverse print a file (I know it can be done in myriads of other ways, like tac, for example). This is what I got so far:
sed -n -r '/.+/{x;H;$p}' nums
The file nums contains a number on each line, from 1 to 25. The logic is this, each number is first copied to the holdspace, while the contents of the holdspace are copied to patspace through x;. So now holdspace contains the current number, while patspace contains the numbers till now in reverse order. Then H; appends the list of numbers gone through until now to the holdspace, so that the latest numbr always stays on top. Finally, when last line is reached, it is done for one last time, and printed out through $p.
However, the command outputs only the numbers 24 to 1 in reverse order. The first line should be 25, but somehow that line is being skipped.
Can anyone tell me what is the problem with this code?
You're missing one last x. You swap the contents of the last line into the hold space, but you never pull it back out before printing.
sed -n -r -e '/.+/{x;H}' -e '${x;p}'
You also get an extra blank line because you really don't want to do the 'H' on the first line, since that preserves the initial blank contents of the hold space. I would do this:
sed -n -r -e '/.+/x' -e '2,$H' -e '${x;p}' nums
This sed should do:
sed '1!G;h;$!d' file
ot this:
sed -n '1!G;h;$p' file
Related
I am looking for a script to search a line if it contains two strings and then find if the first charater of that line contains certain character and remove it.
Eg. if line contains two strings as "abc" and "xyz" it should look for first character of the line and if it contains # , it should remove it and vice-versa.
It tried to run below command in crontab and got the result
crontab -l | grep az2-er32-cxv-iz| grep aze
Output
#5,10 * * * * /opt/apps/scripts/dsm-rync -q -del -s az2-er32-cxv-iz /opt/apps/sdl/scripts/aze-dsm-rync.app.config
since , its difficult to update the crontab entry directly , i copied it to tmpfile.
crontab -l > tmpfile and tried to run sed 's/^#//' tmpfile but it is removing all # instead of the line matching with two strings
You may use gnu awk to do this easily:
awk -i inplace '/az2-er32-cxv-iz/ && /aze/{sub(/^#/, "")} 1' crontab
This will remove # from first position if line has az2-er32-cxv-iz and aze in it.
As mentioned by Shloim, I won't give you the code itself, but I'm giving you a piece of pseudo-code to start:
In order to know if a line contains at least two words, you might search for a space within that line. (grep " " <filename>)
In order to know if a line starts with a certain character, you might search for the character, followed by that one certain character. (grep "<beginning_of_line>#")
Replacing a character in a line can be done, using the sed command.
I have a UTF-8/no BOM file (converted from ISO-8859-1) that has 31214 lines. I have already run dos2unix on the file. When I open it in notepad++, I see a blank line underneath. When I remove this blank line, the line count reduces by one. I save it under a different name and when I tail the file, the prompt displays on the same line. From bash, how do I delete the blank line in the 1st file to produce the result displayed below in the 2nd file?
The goal is to do this from bash w/o manually deleting the line in notepad++
1st file:
[user#server]$ cat file1.txt | wc -l
31214
[user#server]$ tail file1.txt
T 31212 Data 20170517
[user#server]$
2nd file (edited with notepad++)
[user#server]$ cat file2.txt | wc -l
31213
[user#server]$ tail file2.txt
T 31212 Data 20170517[user#server]$
That's the trailing newline of the last line. Some editors allow you to go to the nonexisting "empty" line at the end, some don't show it. Again, some programs may allow you to remove the final newline, but note that e.g. POSIX in effect requires it to be there, and some standard utilities act oddly if it isn't present.
E.g. wc -l counts the number of newlines in the input file (printf "foo\nbar" | wc -l shows 1) so removing the final newline does decrease the line count.
Also, Bash prints the prompt wherever it was that the cursor was left on the screen, so if you print something that doesn't have the trailing newline, the prompt will be placed where the final incomplete line ended, as you saw.
There's no need to remove that final newline, just leave it there.
To remove the final newline character it is possible, as explained here, to use
sed -i '$ s/.$//' your.file
which will substitute nothing for the last character in the last line of the file (if you want to delete smth else from the end of the file you can replace the regex .$ with smth-else$). -i means ‘substitute in-place’ (in FreeBSD/MacOS you need to add an empty string as an argument: sed -i "" '$ s/.$//' your.file)
The file2.txt is missing a trailing newline.
Yes, a text file should end on a newline character.
Given that you do know that a trailing newline is missing, this command should be enough to correct the problem:
$ echo >> file2.txt
I want to last few lines when a pattern is matched in a file using sed.
if file has following entries:
This is the first line.
This is the second line.
This is the third line.
This is the forth line.
This is the Last line.
so, search for pattern, "Last" and print last few lines ..
Find 'Last' using sed and pipe it to tail command which print last n lines of the file -n specifies the no. of lines to be read from end of the file, here I am reading last 2 lines of the file.
sed '/Last/ p' yourfile.txt|tail -n 2
For more info on tail use man tail.
Also, | symbol here is known as a pipe(unnamed pipe), which helps in inter-process communication. So, in simple words sed feeds data to tail command using pipe.
I assume you mean "find the pattern and also print the previous few lines". grep is your friend: to print the previous 3 lines:
$ grep -B 3 "Last" file
This is the second line.
This is the third line.
This is the forth line.
This is the Last line.
-B n for "before". There's also -A n ("after"), and -C n ("context", both before and after).
This might work for you (GNU sed):
sed ':a;$!{N;s/\n/&/2;Ta};/Last/P;D' file
This will print the line containing Last and the two previous lines.
N.B. This will only print the lines before the match once. Also more lines can by shown by changing the 2 to however many lines you want.
Suppose I have setA.txt:
a|b|0.1
c|d|0.2
b|a|0.3
and I also have setB.txt:
c|d|200
a|b|100
Now I want to delete from setA.txt lines that have the same first 2 fields with setB.txt, so the output should be:
b|a|0.3
I tried:
comm -23 <(sort setA.txt) <(sort setB.txt)
But the equality is defined for whole line, so it won't work. How can I do this?
$ awk -F\| 'FNR==NR{seen[$1,$2]=1;next;} !seen[$1,$2]' setB.txt setA.txt
b|a|0.3
This reads through setB.txt just once, extracts the needed information from it, and then reads through setA.txt while deciding which lines to print.
How it works
-F\|
This sets the field separator to a vertical bar, |.
FNR==NR{seen[$1,$2]=1;next;}
FNR is the number of lines read so far from the current file and NR is the total number of lines read. Thus, when FNR==NR, we are reading the first file, setB.txt. If so, set the value of associative array seen to true, 1, for the key consisting of fields one and two. Lastly, skip the rest of the commands and start over on the next line.
!seen[$1,$2]
If we get to this command, we are working on the second file, setA.txt. Since ! means negation, the condition is true if seen[$1,$2] is false which means that this combination of fields one and two was not in setB.txt. If so, then the default action is performed which is to print the line.
This should work:
sed -n 's#\(^[^|]*|[^|]*\)|.*#/^\1/d#p' setB.txt |sed -f- setA.txt
How this works:
sed -n 's#\(^[^|]*|[^|]*\)|.*#/^\1/d#p'
generates an output:
/^c|d/d
/^a|b/d
which is then used as a sed script for the next sed after the pipe and outputs:
b|a|0.3
(IFS=$'|'; cat setA.txt | while read x y z; do grep -q -P "\Q$x|$y|\E" setB.txt || echo "$x|$y|$z"; done; )
explanation: grep -q means only test if grep can find the regexp, but do not output, -P means use Perl syntax, so that the | is matched as is because the \Q..\E struct.
IFS=$'|' will make bash to use | instead of the spaces (SPC, TAB, etc.) as token separator.
Say I have some arbitrary multi-line text file:
sometext
moretext
lastline
How can I remove only the last character (the e, not the newline or null) of the file without making the text file invalid?
A simpler approach (outputs to stdout, doesn't update the input file):
sed '$ s/.$//' somefile
$ is a Sed address that matches the last input line only, thus causing the following function call (s/.$//) to be executed on the last line only.
s/.$// replaces the last character on the (in this case last) line with an empty string; i.e., effectively removes the last char. (before the newline) on the line.
. matches any character on the line, and following it with $ anchors the match to the end of the line; note how the use of $ in this regular expression is conceptually related, but technically distinct from the previous use of $ as a Sed address.
Example with stdin input (assumes Bash, Ksh, or Zsh):
$ sed '$ s/.$//' <<< $'line one\nline two'
line one
line tw
To update the input file too (do not use if the input file is a symlink):
sed -i '$ s/.$//' somefile
Note:
On macOS, you'd have to use -i '' instead of just -i; for an overview of the pitfalls associated with -i, see the bottom half of this answer.
If you need to process very large input files and/or performance / disk usage are a concern and you're using GNU utilities (Linux), see ImHere's helpful answer.
truncate
truncate -s-1 file
Removes one (-1) character from the end of the same file. Exactly as a >> will append to the same file.
The problem with this approach is that it doesn't retain a trailing newline if it existed.
The solution is:
if [ -n "$(tail -c1 file)" ] # if the file has not a trailing new line.
then
truncate -s-1 file # remove one char as the question request.
else
truncate -s-2 file # remove the last two characters
echo "" >> file # add the trailing new line back
fi
This works because tail takes the last byte (not char).
It takes almost no time even with big files.
Why not sed
The problem with a sed solution like sed '$ s/.$//' file is that it reads the whole file first (taking a long time with large files), then you need a temporary file (of the same size as the original):
sed '$ s/.$//' file > tempfile
rm file; mv tempfile file
And then move the tempfile to replace the file.
Here's another using ex, which I find not as cryptic as the sed solution:
printf '%s\n' '$' 's/.$//' wq | ex somefile
The $ goes to the last line, the s deletes the last character, and wq is the well known (to vi users) write+quit.
After a whole bunch of playing around with different strategies (and avoiding sed -i or perl), the best way i found to do this was with:
sed '$! { P; D; }; s/.$//' somefile
If the goal is to remove the last character in the last line, this awk should do:
awk '{a[NR]=$0} END {for (i=1;i<NR;i++) print a[i];sub(/.$/,"",a[NR]);print a[NR]}' file
sometext
moretext
lastlin
It store all data into an array, then print it out and change last line.
Just a remark: sed will temporarily remove the file.
So if you are tailing the file, you'll get a "No such file or directory" warning until you reissue the tail command.
EDITED ANSWER
I created a script and put your text inside on my Desktop. this test file is saved as "old_file.txt"
sometext
moretext
lastline
Afterwards I wrote a small script to take the old file and eliminate the last character in the last line
#!/bin/bash
no_of_new_line_characters=`wc '/root/Desktop/old_file.txt'|cut -d ' ' -f2`
let "no_of_lines=no_of_new_line_characters+1"
sed -n 1,"$no_of_new_line_characters"p '/root/Desktop/old_file.txt' > '/root/Desktop/my_new_file'
sed -n "$no_of_lines","$no_of_lines"p '/root/Desktop/old_file.txt'|sed 's/.$//g' >> '/root/Desktop/my_new_file'
opening the new_file I created, showed the output as follows:
sometext
moretext
lastlin
I apologize for my previous answer (wasn't reading carefully)
sed 's/.$//' filename | tee newFilename
This should do your job.
A couple perl solutions, for comparison/reference:
(echo 1a; echo 2b) | perl -e '$_=join("",<>); s/.$//; print'
(echo 1a; echo 2b) | perl -e 'while(<>){ if(eof) {s/.$//}; print }'
I find the first read-whole-file-into-memory approach can be generally quite useful (less so for this particular problem). You can now do regex's which span multiple lines, for example to combine every 3 lines of a certain format into 1 summary line.
For this problem, truncate would be faster and the sed version is shorter to type. Note that truncate requires a file to operate on, not a stream. Normally I find sed to lack the power of perl and I much prefer the extended-regex / perl-regex syntax. But this problem has a nice sed solution.