In the directory l250 I have a couple of files. They each start off with "l2_" the part after it is the numerical value. I want to be able to plot what is in the file last.sh for about 300 files. I want to know what will be the way I can iterate over the file and have that be $1 and have the second value be $2.
I was thinking that because the files are in the current directory I can use for. But then I am reading about awk and csv and I don't know what to do. I would like insight on what will be the better way to iterate over a large amount of files rater then putting it in main.sh and running that.
main.sh
sh last.sh l2_4 1
sh last.sh l2_5 2
sh last.sh l2_6 3
sh last.sh l2_7 4
.
.
.
last.sh
gnuplot << EOF
set terminal png
set output "finished/$1.png"
set xlabel "Angs"
set ylabel "Angs"
set title "$1 | $2 fs "
set zrange [0: 0.00001]
set pm3d map
splot "$1"
EOF
I run the following in the command line:
$sh main.sh
First you generate a list containing all file names with
list = system('ls l2_*')
which you can then iterate over with do for ... (requires gnuplot 4.6):
set terminal pngcairo
set xlabel "Angs"
set ylabel "Angs"
set zrange [0: 0.00001]
set pm3d map
list = system('ls l2_*')
i = 1
do for [file in list] {
set output sprintf('finished/%s.png', file)
set title sprintf("%s | %d fs", file, i)
splot file
i = i + 1
}
Related
I'm running this script:
gnuplot -e "arg_xlabel='My X Label'" -e "arg_ylabel='My Y Label'" -e "arg_filename='my.csv'" -e "arg_columnindex=4" histogram.plt
All arguments work except for the arg_columnindex which yields:
#!/gnuplot
#
# Histogram of compression ratio distribution
#
set terminal postscript enhanced landscape
set output "histogram.ps"
set size ratio 0.5
set key top right
set xlabel arg_xlabel
set ylabel arg_ylabel
set style fill solid 1.0 border -1
set datafile separator ","
binwidth=0.02 # Adjust according to distribution of values in data file
set boxwidth binwidth * 20
bin(x,width)=width*floor(x/width) + binwidth/2
plot arg_filename using (bin($arg_columnindex,binwidth)):(1.0) smooth freq with boxes t ""
# EOF
gnuplot -e "arg_xlabel='My X Labnel'" -e "arg_ylabel='My Y Label'" -e "arg_filename='my.csv'" -e "arg_columnindex=4" histogram.plt (base)
plot arg_filename using (bin($arg_columnindex,binwidth)):(1.0) smooth freq with boxes t ""
^
"histogram.plt" line 16: Column number or datablock line expected
How can I pass the column index argument?
use column(arg_columnindex) instead of $arg_columnindex.
Besides Ethan's hint to $arg_columnindex, what is the reason that you are specifying the option -e multiple times?
The following works for me. Option -p for window persist (not necessary in your case with postscript terminal).
gnuplot -p -e "arg_xlabel='My X Label'; arg_ylabel='My Y Label'; arg_filename='SO70859921.dat'; arg_columnindex=4" SO70859921.gp
Data: SO70859921.dat
1 0 0 6
2 0 0 4
3 0 0 7
4 0 0 1
5 0 0 3
6 0 0 2
7 0 0 5
Code: SO70859921.gp
### calling the script from command line with arguments
set xlabel arg_xlabel
set ylabel arg_ylabel
plot arg_filename u 1:(column(arg_columnindex)) w lp pt 7
### end of code
Result:
How can I read a datafile as-is (or 1:1) into a datablock?
And how could I do this platform independently?
My attempt so far:
### load datafile "as is" into datablock for different platforms
FILE = 'Test.dat'
if (GPVAL_SYSNAME[:7] eq "Windows") { # "Windows_NT-6.1" is shown on a Win7 system
load "< echo $Data ^<^<EOD & type ".FILE
}
if (GPVAL_SYSNAME eq "Linux") { # that's shown on a Raspberry
load '< echo "\$Data << EOD" & cat '.FILE
}
if (GPVAL_SYSNAME eq "Darwin") { # this was shown on a MacOS Sierra 10.12.6
# how to load a datafile into datablock under MacOS?
}
print $Data
### end of code
What is the value of GPVAL_SYSNAME on a Win10, other Linux, and other MacOS systems?
How many if statements would I need to cover all common systems?
At least under Windows the console window is flashing. How could I possibly surpress this?
My thoughts behind reading data into a dataset are the following:
if you have data on a very(!) slow server path
if you have relatively large datafiles
if you fit and plot multiple curves from several files
For example something like:
FILE1 = '\\SlowServer\blah\BigDataFile.dat'
FILE2 = '\\SlowerServer\blah\BiggerDataFile.dat'
FILE3 = '\\SlowestServer\blah\BiggestDataFile.dat'
fit f(x) FILE1 u 1:2 via a,c,d,e
fit g(x) FILE2 u 2:3 via f,g,h,i
fit h(x) FILE3 u 2:3 via j,k,l,m
plot FILE1 u 1:2:3 w l, \
'' u (function($1)):(function($2)):3 with <whatever>, \
FILE2 u 4:5:6 w l, \
'' u 1:2:3 w l, \
FILE3 u 7:8:9 w l, \
'' u 1:2:3 w l , \
<and more...>
My questions:
Everytime you plot or fit FILE and '', will the content of FILE be loaded again and again or will it be kept in memory?
If you zoom in, e.g. in an interactive wxt terminal, it looks to me as if the files need to be loaded again. Is this true?
If the files are loaded again and again, wouldn't it be best practice to load files once into datablocks once at the beginning and then work with these datablocks?
Any explanations, limitations, pros & cons and comments are appreciated.
Addition:
(partial answer, but with new issue):
For the systems Windows,Linux and MacOS the following seems to work fine. Linux and MacOS are apparently identical.
if (GPVAL_SYSNAME[:7] eq "Windows") { load '< echo $Data ^<^<EOD & type "Test.dat"' }
if (GPVAL_SYSNAME eq "Linux" ) { load '< echo "\$Data << EOD" & cat "Test.dat"' }
if (GPVAL_SYSNAME eq "Darwin") { load '< echo "\$Data << EOD" & cat "Test.dat"' }
However, if I want to call this construct from an external gnuplot procedure "FileToDatablock.gpp" it reproduceably crashes gnuplot under Win7/64 (haven't had a chance to test Linux or MacOS).
"FileToDatablock.gpp"
### Load datafile "as is" 1:1 into datablock for different platforms
# ARG1 = input filename
# ARG2 = output datablock
# usage example: call "FileToDatablock.gpp" "Test.dat" "$Data"
if (ARGC<1) { ARG1 = "Test.dat" }
if (ARGC<2) { ARG2 = "$Data" }
if (GPVAL_SYSNAME[:7] eq "Windows") { load '< echo '.ARG2.' ^<^<EOD & type "'.ARG1.'"' }
if (GPVAL_SYSNAME eq "Linux" ) { load '< echo "\'.ARG2.' << EOD" & cat "'.ARG1.'"' }
if (GPVAL_SYSNAME eq "Darwin") { load '< echo "\'.ARG2.' << EOD" & cat "'.ARG1.'"' }
### end of code
And the file which calls this procedure:
### load datafile 1:1 into datablock
reset session
# this works fine under Win7/64
FILE = "Test.dat"
DATA = "$Data"
load '< echo '.DATA.' ^<^<EOD & type "'.FILE.'"'
print $Data
# this crashes gnuplot under Win7/64
call "tbFileToDatablock.gpp" "Test.dat" "$Data"
print $Data
### end of code
What's wrong with this? Can anybody explain why and how to solve this issue?
It is possible to read a file into a datablock, provided you know the input data format. For example, you have a file MyFile1 with numbers in 3 columns which you want to read into datablock MyBlock1, then plot in 3 ways:
set table $MyBlock1
plot "MyFile1" using 1:2:3 with table
unset table
plot $MyBlock1 using 1:2 with points
plot $MyBlock1 using 2:3 with points
plot $MyBlock1 using 1:3 with lines
This avoids reading the file several times, and should presumably work on any platform. Rather than doing this, I imagine it would be simpler to just copy your files from your slow filesystem to a local filesystem.
The idea is to get a datafile as is (1:1) into a datablock, including commented lines or empty lines, etc.
As far as I know, there seems to be no simple and direct platform-"independent" gnuplot command for this.
In some cases it might be advantageous to have data in datablocks (which are availabe since gnuplot 5.0), because you can simply address lines by index (only since gnuplot 5.2), e.g. $Data[7], or loop data forward and backwards, which you cannot do easily with data from a file.
Here is finally a solution which is acceptable for me and it seems to work on Windows and Linux (tested Windows 7 and 10 and Ubuntu 18.04.4). I couldn't test on MacOS, but I assume the command will be identical with Linux and it will work for MacOS as well. I don't know about other operating systems (feedback appreciated).
Code:
### load data file as is 1:1 into datablock
reset session
FileToDatablock(f,d) = GPVAL_SYSNAME[1:7] eq "Windows" ? \
sprintf('< echo %s ^<^<EOD & type "%s"',d,f) : \
sprintf('< echo "\%s <<EOD" & cat "%s"',d,f) # Linux/MacOS
FILE = 'Test.dat'
load FileToDatablock(FILE,'$Data')
print $Data
### end of code
Data file: (Test.dat)
# This is a test file
1.1 1.2
2.1 2.2
3.1 3.2
# another commented line
4.1 4.2
5.1 5.2
# some empty lines will follow
6.1 6.2 # some spaces at the beginning
7.1 7.3
# end of datafile
Result: (as expected $Data is 1:1 equal to Test.dat)
# This is a test file
1.1 1.2
2.1 2.2
3.1 3.2
# another commented line
4.1 4.2
5.1 5.2
# some empty lines will follow
6.1 6.2 # some spaces at the beginning
7.1 7.3
# end of datafile
I have a gnuplot script (template) which looks like this (a little bit shortened):
#!/bin/bash
F1=file_1
F2=file_2
pdffile=output.pdf
#
term="terminal pdfcairo enhanced fontscale .3 lw 3"
out="output '$pdffile'"
#
gnuplot -persist << EOF
#
set $term
set $out
#
call "statistic-file.txt"
#
# ... (some formating instructions removed)
#
plot "$F1" w l lt 1 lc rgb "red" t "Graph1" ,\
"$F2" w l lt 1 lc rgb "green" t "Graph2"
EOF
okular $pdffile
F1 F2 are variables for my data files. Via "call" command i try to include "statistic-file.txt" in which the variables F1 F2 are also used. This file looks like this (also shortened):
#
stats "$F1" u 2 nooutput name "Y1_"
stats "$F1" u 1 every ::Y1_index_min::Y1_index_min nooutput
X1_min = STATS_min
stats "$F1" u 1 every ::Y1_index_max::Y1_index_max nooutput
X1_max = STATS_max
#
# etc
#
set label \
front center point pt 6 lc rgb "red" ps 1.0 \
at first X1_max,Y1_max tc rgb "red" \
sprintf("%.0f kW / %.0f°", Y1_max, X1_max)
Executing the script, i get a error message:
"statistic-file.txt", line 2: Invalid substitution $F
Pasting the content of "statistic-file.txt" into the template file, then it works. It looks as if the variables in the second file have no relation to the variables in the template file. I prefer the 2 files solution, but how to solve it? Any help?
You are mixing bash variables and gnuplot variables. $F1 is a bash variable and will be substituted by bash only within your bash script. Within "statistic-file.txt", bash substitutes nothing, and gnuplot complains about the unknown $F1.
You can try to "convert" the bash variable $F1 into a gnuplot variable datafile1 like this:
#!/bin/bash
F1="file_1"
gnuplot -persist << EOF
datafile1="$F1"
call "statistic-file.txt"
plot datafile1 w lp
EOF
with the following statistic-file.txt:
stats datafile1
Then bash replaces $F1 with the respective filename and gnuplot uses its own variable datafile1, even within the subsequently called "statistic-file.txt".
Just for completeness: The error message "Invalid substitution $F" comes from gnuplot 4, gnuplot 5 complains about "no datablock named $F1".
One solution would be probably to just move the definition of those variables into the Gnuplot template which is being generated and use them directly, i.e.,
#!/bin/bash
pdffile=output.pdf
#
term="terminal pdfcairo enhanced fontscale .3 lw 3"
out="output '$pdffile'"
#
gnuplot -persist << EOF
#
F1="file_1"
F2="file_2"
set $term
set $out
#
call "statistic-file.txt"
#
# ... (some formating instructions removed)
#
plot F1 w l lt 1 lc rgb "red" t "Graph1" ,\
F2 w l lt 1 lc rgb "green" t "Graph2"
EOF
Similar changes would be necessary also in the file statistic-file.txt, e.g., stats F1 u 2 nooutput name "Y1_" instead of stats "$F1" u 2 nooutput name "Y1_".
Is there any way to iteratively retrieve data from multiple files and plot them on the same graph in gnuplot. Suppose I have files like data1.txt, data2.txt......data1000.txt; each having the same number of columns. Now I could write something like-
plot "data1.txt" using 1:2 title "Flow 1", \
"data2.txt" using 1:2 title "Flow 2", \
.
.
.
"data1000.txt" using 1:2 title "Flow 6"
But this would be really inconvenient. I was wondering whether there is a way to loop through the plot part in gnuplot.
There sure is (in gnuplot 4.4+):
plot for [i=1:1000] 'data'.i.'.txt' using 1:2 title 'Flow '.i
The variable i can be interpreted as a variable or a string, so you could do something like
plot for [i=1:1000] 'data'.i.'.txt' using 1:($2+i) title 'Flow '.i
if you want to have lines offset from each other.
Type help iteration at the gnuplot command line for more info.
Also be sure to see #DarioP's answer about the do for syntax; that gives you something closer to a traditional for loop.
Take a look also to the do { ... } command since gnuplot 4.6 as it is very powerful:
do for [t=0:50] {
outfile = sprintf('animation/bessel%03.0f.png',t)
set output outfile
splot u*sin(v),u*cos(v),bessel(u,t/50.0) w pm3d ls 1
}
http://www.gnuplotting.org/gnuplot-4-6-do/
I have the script all.p
set ...
...
list=system('ls -1B *.dat')
plot for [file in list] file w l u 1:2 t file
Here the two last rows are literal, not heuristic. Then i run
$ gnuplot -p all.p
Change *.dat to the file type you have, or add file types.
Next step: Add to ~/.bashrc this line
alias p='gnuplot -p ~/./all.p'
and put your file all.p int your home directory and voila. You can plot all files in any directory by typing p and enter.
EDIT I changed the command, because it didn't work. Previously it contained list(i)=word(system(ls -1B *.dat),i).
Use the following if you have discrete columns to plot in a graph
do for [indx in "2 3 7 8"] {
column = indx + 0
plot ifile using 1:column ;
}
I wanted to use wildcards to plot multiple files often placed in different directories, while working from any directory. The solution i found was to create the following function in ~/.bashrc
plo () {
local arg="w l"
local str="set term wxt size 900,500 title 'wild plotting'
set format y '%g'
set logs
plot"
while [ $# -gt 0 ]
do str="$str '$1' $arg,"
shift
done
echo "$str" | gnuplot -persist
}
and use it e.g. like plo *.dat ../../dir2/*.out, to plot all .dat files in the current directory and all .out files in a directory that happens to be a level up and is called dir2.
Here is the alternative command:
gnuplot -p -e 'plot for [file in system("find . -name \\*.txt -depth 1")] file using 1:2 title file with lines'
Depending on various factors i may not have 1 or more data files, referenced in pre-defined gnuplot plot instructions, that don't exist. When this is the case i get "warning: Skipping unreadable file" which cancels the rest of the instructions.
Is there any way i can ask gnuplot to skip any missing data files and plot all of the existing ones?
Here is a similar solution without a helper script
file_exists(file) = system("[ -f '".file."' ] && echo '1' || echo '0'") + 0
if ( file_exists("mydatafile") ) plot "mydatafile" u 1:2 ...
the + 0 part is to convert the result from string to integer, in this way you can also use the negation
if ( ! file_exists("mydatafile") ) print "mydatafile not found."
Unfortunately, I can't seem to figure out how to do this without a simple helper script. Here's my solution with the "helper":
#!/bin/bash
#script ismissing.sh. prints 1 if the file is missing, 0 if it exists.
test -e $1
echo $?
Now, make it executable:
chmod +x ismissing.sh
Now in your gnuplot script, you can create a simple function:
is_missing(x)=system("/path/to/ismissing.sh ".x)
and then you guard your plot commands as follows:
if (! is_missing("mydatafile") ) plot "mydatafile" u 1:2 ...
EDIT
It appears that gnuplot isn't choking because your file is missing -- The actual problem arises when gnuplot tries to set the range for the plot from the missing data (I assume you're autoscaling the axis ranges). Another solution is to explicitly set the axis ranges:
set xrange [-10:10]
set yrange [-1:1]
plot "does_not_exist" u 1:2
plot sin(x) #still plots