How to make this awk script simple and use in gnuscript in loop form - gnuplot

I am attempting to plot a multicolum file using gnuplot script.
I am doing it like
plot "100.dat" u ($1-CONS):($2*$3) w l lt 4 ,
"200.dat" u ($1-CONS):($2*$3) w l lt 2 ,
"300.dat" u ($1-CONS):($2*$3) w l lt 1
where CONS is my variable defined at the top of file.
My set xrange is [-0.2:0.2] while data in the scale is beyond this scale.
What I want to capture is (in loop form for multiple files):
maximum value of above three plots in negative and positive both sides and corresponding value of column 1 in my xrange for both the maximum.
in a shell script I can do it easily but I am facing problem in defining in my gnuscript
my shell script is below
for i in 100.0000 200.0000 200.0000
do
grep $i data.dat > $i.dat
awk '{print ($1-CONS), ($2*$3)}' $i.dat | awk '{ if($1 <= 0.2 && $1 >= 0.0) { print }}' > $i.p2.dat ; awk 'BEGIN {min=1000000; max=0;}; { if($2<min && $2 != "") min = $2; if($2>max && $2 != "") max = $2; } END {print min, max}' $i.p2.dat | awk '{print $2}' > $i.p2Max.dat ; PMAX=$(cat $i.p2Max.dat) ; grep "$PMAX" $i.p2.dat | tail -n 1 >> MAX.dat
awk '{print ($1-CONS), ($2*$3)}' $i.dat | awk '{ if($1 <= 0.0 && $1 >= -0.2) { print }}' > $i.mi.dat ; awk 'BEGIN {min=1000000; max=0;}; { if($2<min && $2 != "") min = $2; if($2>max && $2 != "") max = $2; } END {print min, max}' $i.mi.dat | awk '{print $2}' > $i.mi_Max.dat ; N_MAX=$(cat $i.mi_Max.dat) ; grep "$N_MAX" $i.mi.dat | tail -n 1 >> MAX.dat
done
I am looking for a simple script that can be used in the gnuplot script in loop form so that if I have multiple data file and I need to grep the maximum of a colum two (on both the sides of the zero) then it store the maximum value of column two wrt corresponding value of column 1 separately for negative and positive scale.
I would love to see if this can be done using a loop so that I do not need to write all the lines repetitively.

Your description is a bit confusing to me. My understanding is the following: loop through several files and extract the maxima in the xranges [-0.2:0] and [0:0.2],
respectively.
Test data:
100.dat
-0.17 0.447 0.287
-0.13 0.353 0.936
-0.09 0.476 0.309
-0.05 0.504 0.220
-0.01 0.340 0.564
0.03 0.096 0.947
0.07 0.564 0.885
0.11 0.312 0.957
0.15 0.058 0.347
0.19 0.016 0.923
0.23 0.835 0.461
200.dat
-0.17 0.608 0.875
-0.13 0.266 0.805
-0.09 0.948 0.696
-0.05 0.513 0.800
-0.01 0.736 0.392
0.03 0.318 0.312
0.07 0.708 0.534
0.11 0.246 0.975
0.15 0.198 0.914
0.19 0.174 0.318
0.23 0.727 0.341
300.dat
-0.17 0.527 0.658
-0.13 0.166 0.340
-0.09 0.695 0.031
-0.05 0.623 0.542
-0.01 0.996 0.674
0.03 0.816 0.365
0.07 0.286 0.433
0.11 0.069 0.381
0.15 0.719 0.621
0.19 0.516 0.701
0.23 0.248 0.659
Code:
### loop of files and extracting values
reset session
FILES = "100.dat 200.dat 300.dat"
Count = words(FILES)
CONS = 0.03
# get maxima
array NegMaxX[Count]
array NegMaxY[Count]
array PosMaxX[Count]
array PosMaxY[Count]
do for [i=1:Count] {
stats [-0.2:0] word(FILES,i) u ($1-CONS):($2*$3) nooutput
NegMaxX[i] = STATS_pos_max_y
NegMaxY[i] = STATS_max_y
stats [0:0.2] word(FILES,i) u ($1-CONS):($2*$3) nooutput
PosMaxX[i] = STATS_pos_max_y
PosMaxY[i] = STATS_max_y
}
set xrange[-0.2:0.2]
# set labels
do for [i=1:Count] {
set label i*2-1 at NegMaxX[i], NegMaxY[i] sprintf("%.3f/%.3f",NegMaxX[i],NegMaxY[i])
set label i*2 at PosMaxX[i], PosMaxY[i] sprintf("%.3f/%.3f",PosMaxX[i],PosMaxY[i])
}
plot for [i=1:Count] word(FILES,i) u ($1-CONS):($2*$3) w l lt i ti word(FILES,i), \
### end of code
Result:

Related

awk transpose specific row to column and group them

I have the below data in a stdout:
09:13:32 19.2 cpu(1)
09:13:32 15.6 cpu(2)
09:13:32 16.7 cpu(3)
09:13:32 17.1 cpu(6)
09:13:32 17.1 cpu(7)
09:13:32 16.9 cpu(8)
09:13:32 16.7 cpu(9)
09:13:39 13.0 cpu(1)
09:13:39 9.2 cpu(2)
09:13:39 9.1 cpu(3)
09:13:39 7.1 cpu(6)
09:13:39 27.1 cpu(7)
09:13:39 46.9 cpu(8)
09:13:39 36.7 cpu(9)
Trying to convert this to something like below.
['Time', 'cpu(1)', 'cpu(2)', 'cpu(3)', 'cpu(6)', 'cpu(7)', 'cpu(8)', 'cpu(9)'],
['09:13:32', 19.2, 15.6, 16.7, 17.1, 17.1, 16.9, 16.7],
['09:13:39', 13.0, 9.2, 9.1, 7.1, 27.1, 46.9, 36.7]
In other words, I need the original data to be aligned with Google visualization line chart format as stated here: https://developers.google.com/chart/interactive/docs/gallery/linechart
I am trying to achieve this using awk and need some inputs.
awk '{ for(N=1; N<=NF; N+=2) print $N, $(N+1); }' | awk 'BEGIN{q="\047"; printf "["q"Time"q","q"CPU"q"],"}/master/{q="\047"; printf "["q$10q"," $3"],"}' | sed 's/,$//'"
Note: I can change the original data columns like below
Time CPU(%) CPU Number
OR
CPU(%) CPU Number Time
Using any awk for any number of times and any number of cpus, and will work even if you don't have data for some time+cpu combinations, as long as the input isn't so massive that it can't all fit in memory:
$ cat tst.awk
BEGIN {
OFS = ", "
}
!seenTimes[$1]++ {
times[++numTimes] = $1
}
!seenCpus[$3]++ {
cpus[++numCpus] = $3
}
{
vals[$1,$3] = $2
}
END {
printf "[\047%s\047%s", "Time", OFS
for ( cpuNr=1; cpuNr<=numCpus; cpuNr++ ) {
cpu = cpus[cpuNr]
printf "\047%s\047%s", cpu, (cpuNr<numCpus ? OFS : "]")
}
for ( timeNr=1; timeNr<=numTimes; timeNr++ ) {
time = times[timeNr]
printf ",%s[\047%s\047%s", ORS, time, OFS
for ( cpuNr=1; cpuNr<=numCpus; cpuNr++ ) {
cpu = cpus[cpuNr]
val = vals[time,cpu]
printf "%s%s", val, (cpuNr<numCpus ? OFS : "]")
}
}
print ""
}
$ awk -f tst.awk file
['Time', 'cpu(1)', 'cpu(2)', 'cpu(3)', 'cpu(6)', 'cpu(7)', 'cpu(8)', 'cpu(9)'],
['09:13:32', 19.2, 15.6, 16.7, 17.1, 17.1, 16.9, 16.7],
['09:13:39', 13.0, 9.2, 9.1, 7.1, 27.1, 46.9, 36.7]

Use awk command to get information below a pattern

I have a file with a wide range of information and I want to extract some data from here. I only will post here the interesting part. I want to extract IQ and JQ values as well as the J_ij[meV] value which is two lines above. I read this question How to print 5 consecutive lines after a pattern in file using awk where a pattern is used to extract information bellow and I was thinking doing something similar. My initial idea was:
awk '/IQ =/ { print $6,$12 } /IQ =/ {for(i=2; i<=2; i++){ getline; print $11 }}' input.text > output.txt
Loop appears not to working
IT IQ JT JQ N1 N2 N3 DRX DRY DRZ DR J_ij [mRy] J_ij [meV]
IT = 1 IQ = **1** JT = 1 JQ = **1**
->Q = ( -0.250, 0.722, 0.203) ->Q = ( -0.250, 0.722, 0.203)
1 1 1 1 0 0 0 0.000 0.000 0.000 0.000 0.000000000 **0.000000000**
IT = 1 IQ = **1** JT = 6 JQ = **6**
->Q = ( -0.250, 0.722, 0.203) ->Q = ( 0.000, 1.443, 0.609)
1 1 6 6 -1 0 -1 -0.250 -0.144 -0.406 0.498 0.135692822 **1.846194885**
IT = 1 IQ = **1** JT = 8 JQ = **8**
->Q = ( -0.250, 0.722, 0.203) ->Q = ( 0.000, 0.577, 0.609)
1 1 8 8 0 0 -1 0.250 -0.144 -0.406 0.498 0.017676555 **0.240501782**
My expected output is:
IQ JQ J_ij [meV]
1 1 0.000000000
1 6 1.846194885
1 8 0.240501782
It comes from the bold words (** **), first line is only indicative.
Could you please try following. Written and tested with shown examples.
awk '
BEGIN{
print "IQ JQ J_ij [meV]"
}
FNR>1 && /IQ =/{
value=$6 OFS $12
found=1
next
}
found && NF && !/ ->Q/{
if(value){
print value OFS $NF
}
value=found=""
}' Input_file
Output will be as follows.
IQ JQ J_ij [meV]
1 1 0.000000000
1 6 1.846194885
1 8 0.240501782

Extract average time using fping

I want to extract the avg time using fping.
fping -q -b 12 -c 3 localhost 192.168.0.20 192.168.0.1 192.168.0.18 192.168.0.22
localhost : xmt/rcv/%loss = 3/3/0%, min/avg/max =
0.06/0.07/0.09
192.168.0.20 : xmt/rcv/%loss = 3/0/100%
192.168.0.1 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 2.00/2.57/3.11
192.168.0.18 : xmt/rcv/%loss = 3/0/100%
192.168.0.22 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 0.12/0.16/0.19
The average output should be of every device(-1 if device is unreachable), for example.
0.07
-1
2.57
-1
0.16
Thanks
Using awk:
fping -b 12 -c 3 localhost 192.168.0.20 192.168.0.1 192.168.0.18 192.168.0.22 |
awk -F'/' '{print ($8?$8:"-1")}'
0.07
-1
2.57
-1
0.16
Given the / as field delimiter, print the 8th field if it exists otherwise print the string -1
$ ... | awk -F/ '{print (/avg/?$(NF-1):-1)}'
search for "avg" keyword, if found print penultimate field, otherwise -1.

Bash alias cpu usage

I've tried this command but I have a percentage error calculator:
alias cpu="mpstat | awk '\$12 ~ /[0-9.]+/ { print 100 - $12\"%\" }'"
Thank you for help
Change it to this:
alias cpu="mpstat | awk '\$12 ~ /[0-9.]+/ { print 100 - \$12\"%\" }'"
\ was missing after 100 -.
-> mpstat
Linux 3.2.0-69-virtual (myhost) 01/06/2017 _x86_64_ (8 CPU)
10:18:16 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
10:18:16 PM all 12.06 7.00 7.96 0.02 0.00 0.24 0.22 0.00 72.49
-> cpu
27.51%

Best way to divide in bash using pipes?

I'm just looking for an easy way to divide a number (or provide other math functions). Let's say I have the following command:
find . -name '*.mp4' | wc -l
How can I take the result of wc -l and divide it by 3?
The examples I've seen don't deal with re-directed out/in.
Using bc:
$ bc -l <<< "scale=2;$(find . -name '*.mp4' | wc -l)/3"
2.33
In contrast, the bash shell only performs integer arithmetic.
Awk is also very powerful:
$ find . -name '*.mp4' | wc -l | awk '{print $1/3}'
2.33333
You don't even need wc if using awk:
$ find . -name '*.mp4' | awk 'END {print NR/3}'
2.33333
Edit 2018-02-22: Adding shell connector
There is more than 1 way:
Depending on precision required and number of calcul to be done! See shell connector further!
Using bc (binary calculator)
find . -type f -name '*.mp4' -printf \\n | wc -l | xargs printf "%d/3\n" | bc -l
6243.33333333333333333333
or
echo $(find . -name '*.mp4' -printf \\n | wc -l)/3|bc -l
6243.33333333333333333333
or using bash, result in integer only:
echo $(($(find . -name '*.mp4' -printf \\n| wc -l)/3))
6243
Using bash interger builtin math processor
res=000$((($(find . -type f -name '*.mp4' -printf "1+")0)*1000/3))
printf -v res "%.2f" ${res:0:${#res}-3}.${res:${#res}-3}
echo $res
6243.33
Pure bash
With recent 64bits bash, you could even use #glennjackman's ideas of using globstar, but computing pseudo floating could be done by:
shopt -s globstar
files=(**/*.mp4)
shopt -u globstar
res=$[${#files[*]}000/3]
printf -v res "%.2f" ${res:0:${#res}-3}.${res:${#res}-3}
echo $res
6243.33
There is no fork and $res contain a two digit rounded floating value.
Nota: Care about symlinks when using globstar and **!
Introducing shell connector
If you plan to do a lot of calculs, require high precision and use bash, you could use long running bc sub process:
mkfifo /tmp/mybcfifo
exec 5> >(exec bc -l >/tmp/mybcfifo)
exec 6</tmp/mybcfifo
rm /tmp/mybcfifo
then now:
echo >&5 '12/34'
read -u 6 result
echo $result
.35294117647058823529
This subprocess stay open and useable:
ps --sid $(ps ho sid $$) fw
PID TTY STAT TIME COMMAND
18027 pts/9 Ss 0:00 bash
18258 pts/9 S 0:00 \_ bc -l
18789 pts/9 R+ 0:00 \_ ps --sid 18027 fw
Computing $PI:
echo >&5 '4*a(1)'
read -u 6 PI
echo $PI
3.14159265358979323844
To terminate sub process:
exec 6<&-
exec 5>&-
Little demo, about The best way to divide in bash using pipes!
Computing range {1..157} / 42 ( I will let you google for answer to the ultimate question of life, the universe, and everything ;)
... and print 13 result by lines in order to reduce output:
printf -v form "%s" "%5.3f "{,}{,}{,,};form+="%5.3f\n";
By regular way
testBc(){
for ((i=1; i<157; i++)) ;do
echo $(bc -l <<<"$i/42");
done
}
By using long running bc sub process:
testLongBc(){
mkfifo /tmp/mybcfifo;
exec 5> >(exec bc -l >/tmp/mybcfifo);
exec 6< /tmp/mybcfifo;
rm /tmp/mybcfifo;
for ((i=1; i<157; i++)) ;do
echo "$i/42" 1>&5;
read -u 6 result;
echo $result;
done;
exec 6>&-;
exec 5>&-
}
Let's see without:
time printf "$form" $(testBc)
0.024 0.048 0.071 0.095 0.119 0.143 0.167 0.190 0.214 0.238 0.262 0.286 0.310
0.333 0.357 0.381 0.405 0.429 0.452 0.476 0.500 0.524 0.548 0.571 0.595 0.619
0.643 0.667 0.690 0.714 0.738 0.762 0.786 0.810 0.833 0.857 0.881 0.905 0.929
0.952 0.976 1.000 1.024 1.048 1.071 1.095 1.119 1.143 1.167 1.190 1.214 1.238
1.262 1.286 1.310 1.333 1.357 1.381 1.405 1.429 1.452 1.476 1.500 1.524 1.548
1.571 1.595 1.619 1.643 1.667 1.690 1.714 1.738 1.762 1.786 1.810 1.833 1.857
1.881 1.905 1.929 1.952 1.976 2.000 2.024 2.048 2.071 2.095 2.119 2.143 2.167
2.190 2.214 2.238 2.262 2.286 2.310 2.333 2.357 2.381 2.405 2.429 2.452 2.476
2.500 2.524 2.548 2.571 2.595 2.619 2.643 2.667 2.690 2.714 2.738 2.762 2.786
2.810 2.833 2.857 2.881 2.905 2.929 2.952 2.976 3.000 3.024 3.048 3.071 3.095
3.119 3.143 3.167 3.190 3.214 3.238 3.262 3.286 3.310 3.333 3.357 3.381 3.405
3.429 3.452 3.476 3.500 3.524 3.548 3.571 3.595 3.619 3.643 3.667 3.690 3.714
real 0m10.113s
user 0m0.900s
sys 0m1.290s
Wow! Ten seconds on my raspberry-pi!!
Then with:
time printf "$form" $(testLongBc)
0.024 0.048 0.071 0.095 0.119 0.143 0.167 0.190 0.214 0.238 0.262 0.286 0.310
0.333 0.357 0.381 0.405 0.429 0.452 0.476 0.500 0.524 0.548 0.571 0.595 0.619
0.643 0.667 0.690 0.714 0.738 0.762 0.786 0.810 0.833 0.857 0.881 0.905 0.929
0.952 0.976 1.000 1.024 1.048 1.071 1.095 1.119 1.143 1.167 1.190 1.214 1.238
1.262 1.286 1.310 1.333 1.357 1.381 1.405 1.429 1.452 1.476 1.500 1.524 1.548
1.571 1.595 1.619 1.643 1.667 1.690 1.714 1.738 1.762 1.786 1.810 1.833 1.857
1.881 1.905 1.929 1.952 1.976 2.000 2.024 2.048 2.071 2.095 2.119 2.143 2.167
2.190 2.214 2.238 2.262 2.286 2.310 2.333 2.357 2.381 2.405 2.429 2.452 2.476
2.500 2.524 2.548 2.571 2.595 2.619 2.643 2.667 2.690 2.714 2.738 2.762 2.786
2.810 2.833 2.857 2.881 2.905 2.929 2.952 2.976 3.000 3.024 3.048 3.071 3.095
3.119 3.143 3.167 3.190 3.214 3.238 3.262 3.286 3.310 3.333 3.357 3.381 3.405
3.429 3.452 3.476 3.500 3.524 3.548 3.571 3.595 3.619 3.643 3.667 3.690 3.714
real 0m0.670s
user 0m0.190s
sys 0m0.070s
Less than one second!!
Hopefully, results are same, but execution time is very different!
My shell connector
I've published a connector function: Connector-bash on GitHub.com
and shell_connector.sh on my own site.
source shell_connector.sh
newConnector /usr/bin/bc -l 0 0
myBc 1764/42 result
echo $result
42.00000000000000000000
find . -name '*.mp4' | wc -l | xargs -I{} expr {} / 2
Best used if you have multiple outputs you'd like to pipe through xargs. Use{} as a placeholder for the expression term.
Depending on your bash version, you don't even need find for this simple task:
shopt -s nullglob globstar
files=( **/*.mp4 )
dc -e "3 k ${#files[#]} 3 / p"
This method will correctly handle the bizarre edgecase of filenames containing newlines.

Resources