Extract K-th Line from Chunks Using Sed/AWK/Perl - linux

I have some data that looks like this. It comes in chunk of four lines. Each chunk starts with a # character.
#SRR037212.1 FC30L5TAA_102708:7:1:741:1355 length=27
AAAAAAAAAAAAAAAAAAAAAAAAAAA
+SRR037212.1 FC30L5TAA_102708:7:1:741:1355 length=27
::::::::::::::::::::::::;;8
#SRR037212.2 FC30L5TAA_102708:7:1:1045:1765 length=27
TATAACCAGAAAGTTACAAGTAAACAC
+SRR037212.2 FC30L5TAA_102708:7:1:1045:1765 length=27
88888888888888888888888888
What I want to do is to extract last line of each chunk. Yielding:
::::::::::::::::::::::::;;8
888888888888888888888888888
Note that the last line of the chunk may contain any standard ASCII character
including #.
Is there an effective one-liner to do it?

The following sed command will print the 3rd line after the pattern:
sed -n '/^#/{n;n;n;p}' file.txt

If there are no blank lines:
perl -ne 'print if $. % 4 == 0' file

$ awk 'BEGIN{RS="#";FS="\n"}{print $4 } ' file
::::::::::::::::::::::::;;8
88888888888888888888888888
If you always have those 4 lines in a chunk, some other ways
$ ruby -ne 'print if $.%4==0' file
::::::::::::::::::::::::;;8
88888888888888888888888888
$ awk 'NR%4==0' file
::::::::::::::::::::::::;;8
88888888888888888888888888
It also seems like your line is always after the line that start with "+", so
$ awk '/^\+/{getline;print}' file
::::::::::::::::::::::::;;8
88888888888888888888888888
$ ruby -ne 'gets && print if /^\+/' file
::::::::::::::::::::::::;;8
88888888888888888888888888

This prints the lines before lines that starts with #, and also the last line. It can work with non uniform sized chunks, but assumes that only a chunk leading line starts with #.
sed -ne '1d;$p;/^#/!{x;d};/^#/{x;p}' file
Some explanation is in order:
First you don't need the first line so delete it 1d
Next you always need the last line, so print it $p
If you don't have a match swap it into the hold buffer and delete it x;d
If you do have match swap it out of the hold buffer, and print it x;p

This works similarly to dogbane's answer
awk '/^#/ {mark = NR} NR == mark + 3 {print}' inputfile
And, like that answer, will work regardless of the number of lines in each chunk (as long as there are at least 4).
The direct analog to that answer, however, would be:
awk '/^#/ {next; next; next; print}' inputfile

this can be done using grep easily
grep -A 1 '^#' ./infile

This might work for you (GNU sed):
sed '/^#/,+2d' file

Related

Remove line feeds except the first line

I would like to create a file where the line feeds are removed, except in the first line.
Input:
EHH_2020_A1
CCAAGATATTTTATAT
CCATATACC
ATTAT
GTA
Desired output:
EHH_2020_A1
CCAAGATATTTTATATCCATATACCATTATGTA
Thanks a lot in advance!
Best,
Perl to the rescue!
perl -pe 'chomp unless 1 == $.' file
-p reads the input line by line and runs the code for each line,
$. stores the current line number,
chomp removes the final newline (if present)
If you want to keep the final newline, change the condition to
unless 1 == $. || eof
eof returns true when at the end of the file.
You can do it trivially in awk as well, e,g, with your input in the file genes, you would have:
$ awk 'FNR==1 {print; next} {printf "%s", $0} END {print ""}' genes
EHH_2020_A1
CCAAGATATTTTATATCCATATACCATTATGTA
Where the command takes the first record (line) where FNR==1 and simply prints it unchanged. The second rule prints all other lines without a '\n' effectively concatenating them together, and the END rule outputs the final newline.

awk, sed, grep specific strings from a file in Linux

Here is part of the complete file that I am trying to filter:
Hashmode: 13761 - VeraCrypt PBKDF2-HMAC-SHA256 + XTS 512 bit + boot-mode (Iterations: 200000)
Speed.#2.........: 2038 H/s (56.41ms) # Accel:128 Loops:32 Thr:256 Vec:1
Speed.#3.........: 2149 H/s (53.51ms) # Accel:128 Loops:32 Thr:256 Vec:1
Speed.#*.........: 4187 H/s
The aim is to print the following:
13761 VeraCrypt PBKDF2-HMAC-SHA256 4187 H/s
Here is what I tried.
The complete file is called complete.txt
cat complete.txt | grep Hashmode | awk '{print $2,$4,$5}' > mode.txt
Output:
13761 VeraCrypt PBKDF2-HMAC-SHA256
Then:
cat complete.txt | grep Speed.# | awk '{print $2,$3}' > speed.txt
Output:
4187 H/s
Then:
paste mode.txt speed.txt
The issue is that the lines do not match. There are approx 200 types of modes to filter within the file 'complete.txt'
I also have a feeling that this can be done using a much simpler command with sed or awk.
I am guessing you are looking for something like the following.
awk '/Hashmode:/ { if(label) print label, speed; label = $2 " " $4 " " $5 }
/Speed\.#/ { speed = $2 " " $ 3 }
END { if (label) print label, speed }' complete.txt
We match up the Hashmode line with the last Speed.# line which follows, then print when we see a new Hashmode, or reach end of file. (Failing to print the last one is a common beginner bug.)
This might work for you (GNU sed):
sed -E '/Hashmode:/{:a;x;s/^[^:]*: (\S+) -( \S+ \S+ ).*\nSpeed.*:\s*(\S+ \S+).*/\1\2\3/p;x;h;d};H;$!d;ba' file
If a line contains Hashmode, swap to the hold space and using pattern matching, manipulate its contents to the desired format and print, swap back to the pattern space, copy the current line to the hold space and delete the current line.
Otherwise, append the current line to the hold space and delete the current line, unless the current line is the last line in the file, in which case process the line as if it contained Hashmode.
N.B. The first time Hashmode is encountered, nothing is output. Subsequent matches and the end-of-file condition will be the only times printing occurs.

Use sed or awk to replace line after match

I'm trying to create a little script that basically uses dig +short to find the IP of a website, and then pipe that to sed/awk/grep to replace a line. This is what the current file looks like:
#Server
123.455.1.456
246.523.56.235
So, basically, I want to search for the '#Server' line in a text file, and then replace the two lines underneath it with an IP address acquired from dig.
I understand some of the syntax of sed, but I'm really having trouble figuring out how to replace two lines underneath a match. Any help is much appreciated.
Based on the OP, it's not 100% clear exactly what needs to replaced where, but here's a a one-liner for the general case, using GNU sed and bash. Replace the two lines after "3" with standard input:
echo Hoot Gibson | sed -e '/3/{r /dev/stdin' -e ';p;N;N;d;}' <(seq 7)
Outputs:
1
2
3
Hoot Gibson
6
7
Note: sed's r command is opaquely documented (in Linux anyway). For more about r, see:
"5.9. The 'r' command isn't inserting the file into the text" in this sed FAQ.
here's how in awk:
newip=12.34.56.78
awk -v newip=$newip '{
if($1 == "#Server"){
l = NR;
print $0
}
else if(l>0 && NR == l+1){
print newip
}
else if(l==0 || NR != l+2){
print $0
}
}' file > file.tmp
mv -f file.tmp file
explanation:
pass $newip to awk
if the first field of the current line is #Server, let l = current line number.
else if the current line is one past #Server, print the new ip.
else if the current row is not two past #Server, print the line.
overwrite original file with modified version.

replace string in a file with a string from within the same file

I have a file like this (tens of variables) :
PLAY="play"
APPS="/opt/play/apps"
LD_FILER="/data/mysql"
DATA_LOG="/data/log"
I need a script that will output the variables into another file like this (with space between them):
PLAY=${PLAY} APPS=${APPS} LD_FILER=${LD_FILER}
Is it possible ?
I would say:
$ awk -F= '{printf "%s=${%s} ", $1,$1} END {print ""}' file
PLAY=${PLAY} APPS=${APPS} LD_FILER=${LD_FILER} DATA_LOG=${DATA_LOG}
This loops through the file and prints the content before = in a format var=${var} together with a space. At the end, it prints a new line.
Note this leaves a trailing space at the end of the line. If this matters, we can check how to improve it.
< input sed -e 's/\(.*\)=.*/\1=${\1}/' | tr \\n \ ; echo
sed 's/"\([^"]*"\)"/={\1}/;H;$!d
x;y/\n/ /;s/.//' YourFile
your sample exclude last line so if this is important
sed '/DATA_LOG=/ d
s/"\([^"]*"\)"/={\1}/;H;$!d
x;y/\n/ /;s/.//' YourFile

print specific line if it is matches with the line after it

I have a log file containing the following info:
<msisdn>37495989804</msisdn>
<address>10.14.14.26</address>
<msisdn>37495371855</msisdn>
<address>10.14.0.172</address>
<msisdn>37495989832</msisdn>
<address>10.14.14.29</address>
<msisdn>37495479810</msisdn>
<address>10.14.1.11</address>
<msisdn>37495429157</msisdn>
<address>10.14.0.213</address>
<msisdn>37495275824</msisdn>
<msisdn>37495739176</msisdn>
<address>10.14.2.86</address>
<msisdn>37495479840</msisdn>
<address>10.14.1.12</address>
<msisdn>37495706059</msisdn>
<msisdn>37495619889</msisdn>
<address>10.14.1.198</address>
<msisdn>37495574341</msisdn>
<address>10.14.1.148</address>
<msisdn>37495391624</msisdn>
<address>10.14.0.188</address>
<msisdn>37495989796</msisdn>
<address>10.14.14.24</address>
<msisdn>37495835940</msisdn>
<address>10.14.2.164</address>
<msisdn>37495743249</msisdn>
<address>10.14.2.94</address>
<msisdn>37495674117</msisdn>
<address>10.14.1.236</address>
<msisdn>37495754536</msisdn>
<address>10.14.2.120</address>
<msisdn>37495576434</msisdn>
<msisdn>37495823889</msisdn>
<address>10.14.2.159</address>
There are some lines where the 'msisdn' line is not followed by an 'address' line, like this:
<msisdn>37495576434</msisdn>
<msisdn>37495823889</msisdn>
I would like to write a script which will output only the lines ('msisdn' lines), that aren't followed by 'address'. Expected output:
<msisdn>37495275824</msisdn>
<msisdn>37495706059</msisdn>
<msisdn>37495576434</msisdn>
If it will be smth with awk/sed, it will be perfect.
Thanks.
One way with awk:
awk '/address/{p=0}p{print a;p=0}/msisdn/{a=$0;p=1}' log
you can use pcregrep to match next line is not adddress and use awk show it
pcregrep -M '(.*</msisdn>)\n.*<msi' | awk 'NR % 2 == 1'
This might work for you (GNU sed):
sed -r '$!N;/(<msisdn>).*\n.*\1/P;D' file
This reads 2 lines into the pattern space and trys to match the pattern <msisdn> in both the 2 lines. If the pattern matchs it prints out the first line. The first line is then deleted and the process begins again, however since the pattern space contains the second line (now the first), the automatic reading of a line is forgone and process begins as of $!N.
Perl has its own way to do this:
perl -lne 'if($prev && $_!~/\./){print $prev}unless(/\./){$prev=$_}else{undef $prev}' your_file
Tested Below:
> perl -lne 'if($prev && $_!~/\./){print $prev}unless(/\./){$prev=$_}else{undef $prev}' temp
<msisdn>37495275824</msisdn>
<msisdn>37495706059</msisdn>
<msisdn>37495576434</msisdn>
>

Resources