Use awk command to get information below a pattern - linux

I have a file with a wide range of information and I want to extract some data from here. I only will post here the interesting part. I want to extract IQ and JQ values as well as the J_ij[meV] value which is two lines above. I read this question How to print 5 consecutive lines after a pattern in file using awk where a pattern is used to extract information bellow and I was thinking doing something similar. My initial idea was:
awk '/IQ =/ { print $6,$12 } /IQ =/ {for(i=2; i<=2; i++){ getline; print $11 }}' input.text > output.txt
Loop appears not to working
IT IQ JT JQ N1 N2 N3 DRX DRY DRZ DR J_ij [mRy] J_ij [meV]
IT = 1 IQ = **1** JT = 1 JQ = **1**
->Q = ( -0.250, 0.722, 0.203) ->Q = ( -0.250, 0.722, 0.203)
1 1 1 1 0 0 0 0.000 0.000 0.000 0.000 0.000000000 **0.000000000**
IT = 1 IQ = **1** JT = 6 JQ = **6**
->Q = ( -0.250, 0.722, 0.203) ->Q = ( 0.000, 1.443, 0.609)
1 1 6 6 -1 0 -1 -0.250 -0.144 -0.406 0.498 0.135692822 **1.846194885**
IT = 1 IQ = **1** JT = 8 JQ = **8**
->Q = ( -0.250, 0.722, 0.203) ->Q = ( 0.000, 0.577, 0.609)
1 1 8 8 0 0 -1 0.250 -0.144 -0.406 0.498 0.017676555 **0.240501782**
My expected output is:
IQ JQ J_ij [meV]
1 1 0.000000000
1 6 1.846194885
1 8 0.240501782
It comes from the bold words (** **), first line is only indicative.

Could you please try following. Written and tested with shown examples.
awk '
BEGIN{
print "IQ JQ J_ij [meV]"
}
FNR>1 && /IQ =/{
value=$6 OFS $12
found=1
next
}
found && NF && !/ ->Q/{
if(value){
print value OFS $NF
}
value=found=""
}' Input_file
Output will be as follows.
IQ JQ J_ij [meV]
1 1 0.000000000
1 6 1.846194885
1 8 0.240501782

Related

How to make this awk script simple and use in gnuscript in loop form

I am attempting to plot a multicolum file using gnuplot script.
I am doing it like
plot "100.dat" u ($1-CONS):($2*$3) w l lt 4 ,
"200.dat" u ($1-CONS):($2*$3) w l lt 2 ,
"300.dat" u ($1-CONS):($2*$3) w l lt 1
where CONS is my variable defined at the top of file.
My set xrange is [-0.2:0.2] while data in the scale is beyond this scale.
What I want to capture is (in loop form for multiple files):
maximum value of above three plots in negative and positive both sides and corresponding value of column 1 in my xrange for both the maximum.
in a shell script I can do it easily but I am facing problem in defining in my gnuscript
my shell script is below
for i in 100.0000 200.0000 200.0000
do
grep $i data.dat > $i.dat
awk '{print ($1-CONS), ($2*$3)}' $i.dat | awk '{ if($1 <= 0.2 && $1 >= 0.0) { print }}' > $i.p2.dat ; awk 'BEGIN {min=1000000; max=0;}; { if($2<min && $2 != "") min = $2; if($2>max && $2 != "") max = $2; } END {print min, max}' $i.p2.dat | awk '{print $2}' > $i.p2Max.dat ; PMAX=$(cat $i.p2Max.dat) ; grep "$PMAX" $i.p2.dat | tail -n 1 >> MAX.dat
awk '{print ($1-CONS), ($2*$3)}' $i.dat | awk '{ if($1 <= 0.0 && $1 >= -0.2) { print }}' > $i.mi.dat ; awk 'BEGIN {min=1000000; max=0;}; { if($2<min && $2 != "") min = $2; if($2>max && $2 != "") max = $2; } END {print min, max}' $i.mi.dat | awk '{print $2}' > $i.mi_Max.dat ; N_MAX=$(cat $i.mi_Max.dat) ; grep "$N_MAX" $i.mi.dat | tail -n 1 >> MAX.dat
done
I am looking for a simple script that can be used in the gnuplot script in loop form so that if I have multiple data file and I need to grep the maximum of a colum two (on both the sides of the zero) then it store the maximum value of column two wrt corresponding value of column 1 separately for negative and positive scale.
I would love to see if this can be done using a loop so that I do not need to write all the lines repetitively.
Your description is a bit confusing to me. My understanding is the following: loop through several files and extract the maxima in the xranges [-0.2:0] and [0:0.2],
respectively.
Test data:
100.dat
-0.17 0.447 0.287
-0.13 0.353 0.936
-0.09 0.476 0.309
-0.05 0.504 0.220
-0.01 0.340 0.564
0.03 0.096 0.947
0.07 0.564 0.885
0.11 0.312 0.957
0.15 0.058 0.347
0.19 0.016 0.923
0.23 0.835 0.461
200.dat
-0.17 0.608 0.875
-0.13 0.266 0.805
-0.09 0.948 0.696
-0.05 0.513 0.800
-0.01 0.736 0.392
0.03 0.318 0.312
0.07 0.708 0.534
0.11 0.246 0.975
0.15 0.198 0.914
0.19 0.174 0.318
0.23 0.727 0.341
300.dat
-0.17 0.527 0.658
-0.13 0.166 0.340
-0.09 0.695 0.031
-0.05 0.623 0.542
-0.01 0.996 0.674
0.03 0.816 0.365
0.07 0.286 0.433
0.11 0.069 0.381
0.15 0.719 0.621
0.19 0.516 0.701
0.23 0.248 0.659
Code:
### loop of files and extracting values
reset session
FILES = "100.dat 200.dat 300.dat"
Count = words(FILES)
CONS = 0.03
# get maxima
array NegMaxX[Count]
array NegMaxY[Count]
array PosMaxX[Count]
array PosMaxY[Count]
do for [i=1:Count] {
stats [-0.2:0] word(FILES,i) u ($1-CONS):($2*$3) nooutput
NegMaxX[i] = STATS_pos_max_y
NegMaxY[i] = STATS_max_y
stats [0:0.2] word(FILES,i) u ($1-CONS):($2*$3) nooutput
PosMaxX[i] = STATS_pos_max_y
PosMaxY[i] = STATS_max_y
}
set xrange[-0.2:0.2]
# set labels
do for [i=1:Count] {
set label i*2-1 at NegMaxX[i], NegMaxY[i] sprintf("%.3f/%.3f",NegMaxX[i],NegMaxY[i])
set label i*2 at PosMaxX[i], PosMaxY[i] sprintf("%.3f/%.3f",PosMaxX[i],PosMaxY[i])
}
plot for [i=1:Count] word(FILES,i) u ($1-CONS):($2*$3) w l lt i ti word(FILES,i), \
### end of code
Result:

Extract average time using fping

I want to extract the avg time using fping.
fping -q -b 12 -c 3 localhost 192.168.0.20 192.168.0.1 192.168.0.18 192.168.0.22
localhost : xmt/rcv/%loss = 3/3/0%, min/avg/max =
0.06/0.07/0.09
192.168.0.20 : xmt/rcv/%loss = 3/0/100%
192.168.0.1 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 2.00/2.57/3.11
192.168.0.18 : xmt/rcv/%loss = 3/0/100%
192.168.0.22 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 0.12/0.16/0.19
The average output should be of every device(-1 if device is unreachable), for example.
0.07
-1
2.57
-1
0.16
Thanks
Using awk:
fping -b 12 -c 3 localhost 192.168.0.20 192.168.0.1 192.168.0.18 192.168.0.22 |
awk -F'/' '{print ($8?$8:"-1")}'
0.07
-1
2.57
-1
0.16
Given the / as field delimiter, print the 8th field if it exists otherwise print the string -1
$ ... | awk -F/ '{print (/avg/?$(NF-1):-1)}'
search for "avg" keyword, if found print penultimate field, otherwise -1.

how to write a bash script for matching files [duplicate]

This question already has answers here:
Inner join on two text files
(5 answers)
Closed 6 years ago.
I would like to write a script to match two files. I have a file which is always change and a file acts as a database.
Input file1:
1
3
5
7
9
Database matched file1:
A B C D E F
1 0.27776079 0.302853938 1.52415756 2.751714059 1.363932416 2.286189771
2 0.332465 0.777918524 0.705056607 0.484138872 0.443787105 0.848742839
3 0.941768856 0.19125 0.573714912 0.5040488 0.526207725 1.554118026
4 1.717348092 0.19642752 0.315945 0.1331712 0.28427498 0.30113875
5 0.802253697 0.3768849 0.426688 0.27693 0.591697038 0.3832675
6 0.2752232 0.570078 0.3847095 0.659548575 0.327469824 0.3346875
7 0.153272 0.36594447 0.19125 0.526602427 0.44771265 0.31136
8 0.637448551 0.735756919 1.284158594 0.464060016 0.259459816 0.887975536
9 0.397221469 0.20808 0.268226 0.710250679 0.493069267 0.47672443
10 0.196928 0.492713856 0.22302 0.783853054 0.303534 1.736908487
11 0.510789888 0.14948712 0.26432 0.684485438 0.683017627 0.614033957
desired output file1:
A B C D E F
1 0.27776079 0.302853938 1.52415756 2.751714059 1.363932416 2.286189771
3 0.941768856 0.19125 0.573714912 0.5040488 0.526207725 1.554118026
5 0.802253697 0.3768849 0.426688 0.27693 0.591697038 0.3832675
7 0.153272 0.36594447 0.19125 0.526602427 0.44771265 0.31136
9 0.397221469 0.20808 0.268226 0.710250679 0.493069267 0.47672443
I would like to extract the matched lines from the database.
head -1 database1.txt > output1.txt
grep -wf inputfile1.txt database1.txt >> output1.txt
head -1 database1.txt > output2.txt
grep -wf inputfile2.txt database1.txt >> output2.txt
head -1 database2.txt > output3.txt
grep -wf inputfile3.txt database2.txt >> output3.txt
I try to use nano command but every time need to change the syntax.
You can use the join command to join the 2 files on 1st column:
$ cat file1
1
3
5
7
9
$ cat file2
A B C D E F
1 0.27776079 0.302853938 1.52415756 2.751714059 1.363932416 2.286189771
2 0.332465 0.777918524 0.705056607 0.484138872 0.443787105 0.848742839
3 0.941768856 0.19125 0.573714912 0.5040488 0.526207725 1.554118026
4 1.717348092 0.19642752 0.315945 0.1331712 0.28427498 0.30113875
5 0.802253697 0.3768849 0.426688 0.27693 0.591697038 0.3832675
6 0.2752232 0.570078 0.3847095 0.659548575 0.327469824 0.3346875
7 0.153272 0.36594447 0.19125 0.526602427 0.44771265 0.31136
8 0.637448551 0.735756919 1.284158594 0.464060016 0.259459816 0.887975536
9 0.397221469 0.20808 0.268226 0.710250679 0.493069267 0.47672443
10 0.196928 0.492713856 0.22302 0.783853054 0.303534 1.736908487
11 0.510789888 0.14948712 0.26432 0.684485438 0.683017627 0.614033957
$ sed -n '1p' file2 && join --nocheck-order file1 <(sed -n '1!p' file2)
A B C D E F
1 0.27776079 0.302853938 1.52415756 2.751714059 1.363932416 2.286189771
3 0.941768856 0.19125 0.573714912 0.5040488 0.526207725 1.554118026
5 0.802253697 0.3768849 0.426688 0.27693 0.591697038 0.3832675
7 0.153272 0.36594447 0.19125 0.526602427 0.44771265 0.31136
9 0.397221469 0.20808 0.268226 0.710250679 0.493069267 0.47672443
$

Perl Thread To Increment Variable

I have the following Perl code::
#!/usr/bin/perl
use threads;
use Thread::Queue;
use DateTime;
$| = 1; my $numthreads = 20;
$min = 1;
$max = 100;
my $fetch_q = Thread::Queue->new();
our $total = 0;
sub fetch {
while ( my $target = $fetch_q->dequeue() ) {
print $total++ . " ";
}
}
my #workers = map { threads->create( \&fetch ) } 1 .. $numthreads;
$fetch_q->enqueue( $min .. $max );
$fetch_q->end();
foreach my $thr (#workers) {$thr->join();}
The code creates 20 threads and then increments a variable $total.
The current output is something like:
0 0 0 0 0 1 0 0 1 1 2 0 0 1 0 2 0 3 0 1 0 2 1 0 2 1 0 0 3 0
But the desired output is:
1 2 3 4 5 6 7 8 9 10 .... 30
Is there a way to have Perl increment the variable? The order does not matter (i.e. its fine if it is 1 2 4 5 3).
use threads::shared;
my $total :shared = 0;
lock $total;
print ++$total . " ";

arrange file line into tabular form [closed]

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
This is a sample line from my file:
42001232 2011-07-01 51 100001 0 100002 0 2011-07-02 51 100003 0 100004 0
How do I arrange it to look like this
42001232 2011-07-01 51 100001 0
42001232 2011-07-01 51 100002 0
42001232 2011-07-02 51 100003 0
42001232 2011-07-02 51 100004 0
Apart from the first column, all the columns are repeating starting with a date.
I need to organize it in a tabular form. Also, the delimiter here is TAB.
Here's one way using awk. Run like:
awk -f script.awk file
Contents of script.awk:
BEGIN {
FS=OFS="\t"
}
{
for(i=2;i<=NF;i++) {
if ($i ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) {
for (j=i+2;j<=NF;j+=2) {
if ($j ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) {
break
}
else {
print $1, $i, $(i+1), $j, $(j+1)
}
}
}
}
}
Results:
42001232 2011-07-01 51 100001 0
42001232 2011-07-01 51 100002 0
42001232 2011-07-02 51 100003 0
42001232 2011-07-02 51 100004 0
Alternatively, here's the one-liner:
awk 'BEGIN { FS=OFS="\t" } { for(i=2;i<=NF;i++) if ($i ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) for (j=i+2;j<=NF;j+=2) if ($j ~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) break; else print $1, $i, $(i+1), $j, $(j+1) }' file
This works on the given data:
#!/usr/bin/env perl
use strict;
use warnings;
use English qw( -no_match_vars );
$OFS = qq"\t";
while (<>)
{
chomp;
my(#fields) = split /\s+/, $_;
my $col1 = shift #fields;
my $date = shift #fields;
my $col3 = shift #fields;
while (scalar(#fields) > 1)
{
if ($fields[0] =~ /^\d{4}-\d\d-\d\d$/)
{
$date = shift #fields;
$col3 = shift #fields;
next;
}
else
{
my $col4 = shift #fields;
my $col5 = shift #fields;
print $col1, $date, $col3, $col4, "$col5\n";
}
}
print STDERR "oops - debris $fields[0] left over\n" if (scalar(#fields) != 0);
}
The output I got is:
42001232 2011-07-01 51 100001 0
42001232 2011-07-01 51 100002 0
42001232 2011-07-02 51 100003 0
42001232 2011-07-02 51 100004 0
That's a perfectly horrid format to have to parse. I've had to make some assumptions about the way the repetitions are handled, so that the column after a date is fixed until the next date, for example.

Resources