Related
How do I plot this time samples with a string (W) inside (column 3) ?
And How do I control xtics increment with time format ?
Few lines of Data :
France,FR,2020-W09,118,3318,67012883,4.95128675481698,3.55635925256178,TESSy
France,FR,2020-W10,996,11101,67012883,16.5654714482288,8.97216466984956,TESSy
France,FR,2020-W11,4297,29623,67012883,44.2049329529667,14.5056206326166,TESSy
France,FR,2020-W12,10595,73235,67012883,109.28495644636,14.4671263740015,TESSy
France,FR,2020-W13,24156,122870,67012883,183.35280396756,19.6598030438675,TESSy
France,FR,2020-W14,30304,127029,67012883,189.55907329043,23.8559698966378,TESSy
France,FR,2020-W15,24925,140316,67012883,209.386604065371,17.7634767239659,TESSy
My script :
#https://www.ecdc.europa.eu/en/publications-data/covid-19-testing
#Data (105,77K) here :
system("wget https://opendata.ecdc.europa.eu/covid19/testing/csv -P $PWD -O testing.csv")
reset
set term wxt font ',11' size 1200,800
set datafile separator ","
set grid
#set key at screen 0.9, 0.9
timefmt = "%Y-%s%W"
set xdata time
set xtics format timefmt timedate rotate by -45
SECPERWEEK = 3600.*24.*7.
Y_W(col) = timecolumn(col,timefmt) + SECPERWEEK * (strcol(col)[2:3] - 1)
plot '< grep France testing.csv' u (Y_W(3)):4 notitle w l
Thank you
Here is a suggestion how I would do it. It's maybe not obvious and looks maybe a bit complicated, but it is a gnuplot-only solution.
Since I do not run Linux, I do not have grep, that's why I define myFilter() in gnuplot itself which is platform independent.
Everytime this filter gives a hit, the counter t will be increased by one which has the advantage that the data can contain a interlaced mix of countries. I assume that's what grep would allow as well. The only assumption here is that the week numbers are in (ascending) order, they would not be sorted.
I guess here it is not necessary to have the x-axis as timeformat.
The situation would be different if there are missing calendar week(s) and you want to keep an according gap for them.
With myOffset=0 and myEvery=2 you set how many x-tic labels you want to have displayed.
There is certainly room for improvement and I'm sure there are other solutions... so, just as a starting point...
Code:
### plot filtered data with custom xtics
reset session
$Data <<EOD
France,FR,2020-W09,118,3318,67012883,4.95128675481698,3.55635925256178,TESSy
France,FR,2020-W10,996,11101,67012883,16.5654714482288,8.97216466984956,TESSy
France,FR,2020-W11,4297,29623,67012883,44.2049329529667,14.5056206326166,TESSy
Luxembourg,LU,2020-W11,11,222,33333333,44.4444444444444,55.5555555555555,fghij
Luxembourg,LU,2020-W12,11,222,33333333,44.4444444444444,55.5555555555555,fghij
France,FR,2020-W12,10595,73235,67012883,109.28495644636,14.4671263740015,TESSy
France,FR,2020-W13,24156,122870,67012883,183.35280396756,19.6598030438675,TESSy
Belgium,BE,2020-W13,1111,222222,33333333,444.44444444444,55.5555555555555,abcde
Belgium,BE,2020-W14,1111,222222,33333333,444.44444444444,55.5555555555555,abcde
France,FR,2020-W14,30304,127029,67012883,189.55907329043,23.8559698966378,TESSy
France,FR,2020-W15,24925,140316,67012883,209.386604065371,17.7634767239659,TESSy
EOD
set datafile separator comma
set datafile missing NaN
set xtics rotate by -45
myFilter(dcol,fcol,key) = strcol(fcol) eq key ? (t=t+1, column(dcol)) : NaN
myXtic(col) = sprintf("%s",(t+myOffset)% myEvery ? "" : strcol(col))
myKey = 'France'
myOffset = 0
myEvery = 2
plot t=1 $Data u (t):(myFilter(4,1,myKey)):xtic(myXtic(3)) w lp pt 7 title myKey
### end of code
Result:
The basic error is that the Y_W function is looking in the wrong columns for the week number. It should be substring 7 to 8 not 2 to 3.
Y_W(col) = timecolumn(col,"%Y") + SECPERWEEK * (strcol(col)[7:8])
As explained by theozh in this answer, gnuplot uses American week numbers by default, not ISO 8601, so I have not addressed that here.
I have a file of data where the first column is time in seconds. The start of time is 0, and onward from there.
I want to plot the data with an x-axis formatted as days:hours:minutes:seconds. T=0 should map to 00:00:00:00. I can't figure out how to get days to start at 00 instead of 01. I have tried the below. I also tried setting xrange to [-86400:173000], but that maps to day 365, not 0. Shouldn't it be common to plot some time-sampled data, that may span days, starting with T=0?
It seems that GnuPlot needs a different set of time format characters for zero-based time plotting, instead of date-based. Unless it already has it and I have missed it.
data
0 0
3600 10
7200 30
21600 50
160000 100
GnuPlot script
set xdata time
set format x "%02j:%H:%M:%S"
set timefmt "%s"
set xrange [0:173000]
plot "data" using 1:2 with lines
I can get you part way there. Gnuplot has a separate set of time formats for relative time. Zero-based and handles positive and negative intervals. It's hard to find in the documentation, but here is a section from "help time_specifiers".
Format Explanation
%tH +/- hours relative to time=0 (does not wrap at 24)
%tM +/- minutes relative to time=0
%tS +/- seconds associated with previous tH or tM field
Examples of time format:
The date format specifiers encode a time in seconds as a clock time
on a particular day. So hours run only from 0-23, minutes from 0-59,
and negative values correspond to dates prior to the epoch
(1-Jan-1970). In order to report a time value in seconds as some
number of hours/minutes/seconds relative to a time 0, use time
formats %tH %tM %tS. To report a value of -3672.50 seconds
set format x # default date format "12/31/69 \n 22:58"
set format x "%tH:%tM:%tS" # "-01:01:12"
set format x "%.2tH hours" # "-1.02 hours"
set format x "%tM:%.2tS" # "-61:12.50"
Using these relative time formats with your sample data I can get as far as:
$data << EOD
0 0
3600 10
7200 30
21600 50
160000 100
EOD
set xtics time format "%tH:%tM:%tS"
set title 'set xtics time format "%tH:%tM:%tS"'
set xrange [0:173000]
plot $data using 1:2 with lp
Now the problem is that there is no equivalent relative day format. Call that a bug or at least a missing feature. Let's take a stab at adding days to the format by hand.
secperday = 3600*24
days(t) = gprintf("%02g:", int(t)/secperday)
hours(t) = strftime("%02tH:%tM:%tS", int(t)%secperday)
# Create ten days worth of tic labels
# Every six hours with no label; once a day with full label
set xtics 6*3600 format ""
do for [i=0:10] {
T = day * secperday
set xtics add ( days(T).hours(T) T )
}
plot $data using 1:2 with lp
As mentioned in the comments above, one workaround would be using week days, which however would limit you to 7 days.
Since 0 seconds correspond to Thursday, 01.01.1970 00:00:00 you have to subtract 4 days = 24*3600*4 seconds to make it a Sunday (=0).
Another strange workaround would be to use multiplot and plot twice, just for the day labels. You have to set a bottom margin to exactly "overplot" the previous plot. There would be still room for fine tuning.
By the way: If the scale is several days then the question is if seconds in the label are actually relevant?
Code:
### timedate days starting from zero
reset session
$Data <<EOD
0 0
3600 10
7200 30
21600 50
160000 100
450000 222
500000 333
EOD
set multiplot layout 2,1
# first workaround, limited to 7 days
set format x "day %1w\n%H:%M:%S" timedate
plot $Data u ($1-24*3600*4):2 w lp pt 7 notitle
# second workaround, using multiplot
set format x "\n%H:%M:%S" timedate
set bmargin 3
plot $Data u 1:2 w lp pt 7 notitle
set multiplot previous
set format x "day %s"
set xrange[GPVAL_X_MIN/86400:GPVAL_X_MAX/86400]
plot $Data u ($1/86400):2 w p ps 0 notitle # point size zero, i.e. invisible
unset multiplot
### end of code
Result:
My Data Looks like this:
File_A
2015-08-01 00:00 424.9563018750976
2015-08-01 00:01 785.7380434263472
....
2015-08-01 23:59
File_B
2015-08-02 00:01 1.4625096463725384
2015-08-02 00:02 6.0047607076482485
....
2015-08-02 23:59
So the Date Format is: %Y-%m-%d %H:%M
I'd like to plot a series for the whole month but for each day only the timeperiod fom e.g. 09:00 - 15:00.
Here's what I tried:
clear
reset
cd 'C:\Users\SammerP\Desktop\Desktop Calc\AIT ALL\CatchAllDataAIT\data'
set title 'Data of Month' font 'bold verdana,08'
set xdata time
set timefmt '%Y-%m-%d %H:%M'
set format x '%Y-%m-%d %H:%M'
set ylabel 'Value'
set yrange [550:700]
set ytics (550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710,720,730,740,750,760,770,780,790,800)
set xtics rotate by 45 offset -1.5,-1.2
set xrange ['2014-08-01 09:00':'2014-08-31 15:00']
set xtics ('08-01' '2014-08-01 13:00','08-02' '2014-08-02 13:00','08-03' '2014-08-03 13:00','08-04' '2014-08-04 13:00','08-05' '2014-08-05 13:00',\
'08-06' '2014-08-06 13:00','08-07' '2014-08-07 13:00','08-08' '2014-08-08 13:00','08-09' '2014-08-09 13:00','08-10' '2014-08-10 13:00','08-11' '2014-08-11 13:00',\
'08-12' '2014-08-12 13:00','08-13' '2014-08-13 13:00','08-14' '2014-08-14 13:00','08-15' '2014-08-15 13:00','08-16' '2014-08-16 13:00',\
'08-17' '2014-08-17 13:00','08-18' '2014-08-18 13:00','08-19' '2014-08-19 13:00','08-20' '2014-08-20 13:00','08-21' '2014-08-21 13:00',\
'08-22' '2014-08-22 13:00','08-23' '2014-08-23 13:00','08-24' '2014-08-24 13:00','08-25' '2014-08-25 13:00','08-26' '2014-08-26 13:00',\
'08-27' '2014-08-27 13:00','08-28' '2014-08-28 13:00','08-29' '2014-08-29 13:00','08-30' '2014-08-30 13:00','08-31' '2014-08-31 13:00')
set grid
set nokey
plot'File_A.csv' using 1:2 axes x1y1 linewidth 1 lc rgb 'red' w l,\
'File_B' using 1:2 axes x1y1 linewidth 1 lc rgb 'red' w l,
..........
I tried it by plotting only the certain line from each file but than I end up with hole between the days.
I thought that maybe the problem was the xrange setting so I deleted that but the Problem was the same
Is there a way to set the xrange so that Gnuplot plots From Day 1 to 31, every Day from 09:00 from 15:00 and without the empty spaces between the days.
Here's an example of how it looks like now:
That cannot be done automatically with gnuplot. In my opinion, your best option is to not use a time-axis, but a normal numerical axis and provide your own mapping to label the xtics properly and to adjust the gap between the days like you want.
Here is a possible solution, but without having proper data for testing I couldn't verify if I got all steps correct:
gapBetweenDaysInSeconds = 1000
startHour = 9.0
endHour = 15.0
set xtics ("01-08" (13-startHour)*60*60)
set for [i=2:31] xtics add (sprintf("08-%02d", i) ((i-1)*(endHour-startHour) + 13 - startHour)*60*60 + (i-1)*gapBetweenDaysInSeconds)
# for gnuplot 5.0
# gettime(col) = timecolumn(col, '%Y-%m-%d %H:%M')
# for all gnuplot version.
gettime(col) = strptime('%Y-%m-%d %H:%M', strcol(col).' '.strcol(col+1))
files="File_A.csv File_B.csv File_C.csv" # ... to be continued
plot for [file in files] file using (t=gettime(1), tm_hour(t) < startHour || tm_hour(t) >= endHour ? 1/0 : (((tm_mday(t)-1)*(endHour-startHour) + (tm_hour(t)-startHour))*60 + tm_min(t))*60 + tm_sec(t) + (tm_mday(t)-1)*gapBetweenDaysInSeconds):2 lt 1
That assumes, that you have one file per day, and those files are listed, separated by white spaces, in the variable files.
The first call to set xtics sets only a single xtic and deletes all other. Then I use set xtics add to add all following tics in a loop.
For each row, the time stamp is parsed using the gettime user-defined function and assigned to the variable t. In a second step I extract only the day of the mont, hour, minutes and seconds using the tm_* functions and calculate a position on the xaxis given in seconds since the first of the respective month. The gap between two days can be given using the variable gapBetweenDaysInSeconds.
At the moment the month is hardcoded to 08.
That should be a reasonable starting point.
i have a csv file from 5 year malware data collected there are 2 columns the dates and the ips every date have 1 or more ips example
1/5/2013 12.234.123
1/5/2013 45.123.566
1/5/2013 100.546.12
1/6/2013 42.153.756
3/4/2014 75.356.258 etc... (every day for 5 years)
now i am trying to get the percentage difference between every month example:
November 2014 - 10%
December 2014 - 15%
i tried to put the percentage on y axis and in x2 axis but im getting some crazy results i am new to gnuplot and im still learning it here is the code i have right now:
set title 'Results Per Month'
set xlabel 'Date'
set ylabel 'Percentage'
set terminal png size 2800,900
set datafile sep ','
set xdata time
set timefmt '%Y/%m/%d'
set xrange['2009/3/22':'2014/12/02']
set xtics 30*24*60*60
set format x '%Y/%m'
set autoscale x2fix
set x2tics
set x2range[0:*]
set format x2 "%g %%"
set xtics nomirror rotate by -90
set grid ytics xtics
set ytics 10
set yrange [0:*]
set term png
set output 'file.png'
plot 'export.csv' using (timecolumn(1) - (tm_mday(timecolumn(1))-1)*24*60*60):(1) smooth frequency w lp pt 7 ps 2 notitle, \
'' using (($1-$2)/$1*100):x2ticlabels(2) axes x2y1 with points ps 2 lw 2
I would suggest you to use some external script for such kind of preprocessing (you can also do this on-the-fly). Yes, you can do this in gnuplot in two steps, but can become quite complicated and requires some more profound knowledge of gnuplot.
Here is a working script, but I won't go into detail about the many different aspects of the actual implementation:
set xdata time
set timefmt '%Y/%m/%d'
set datafile separator ','
set table 'temporaryfile.dat'
set format x '%Y/%m/%d'
plot 'export.csv' using (timecolumn(1) - (tm_mday(timecolumn(1))-1)*24*60*60):(1) smooth frequency
unset table
set y2tics
set ytics nomirror
set timefmt '"%Y/%m/%d"'
set format x '%b %Y'
set xtics rotate by 90 right
set datafile separator white
set yrange[0:*]
x0=x1=0
plot 'temporaryfile.dat' using 1:(strcol(3) eq "i" ? $2 : 1/0) w lp pt 7 ps 2 title 'IP count', \
'' using 1:(x1=x0, x0=$2, strcol(3) eq "i" ? ($0 == 0 || x0 == 0 ? 0 : (x0-x1)/x0 * 100.0) : 1/0) axes x1y2 w lp title 'percentual change'
Basically, first you plot the result data of smooth frequency into a second data file. Then you can plot this, and to the calculations for the percentages.
Please note, that I used a timeformat which corresponds to your test data (and the data of your previous question), which doesn't correspond with what you have in your script! Please pay attention to this.
Also note, that the timefmt before the actual plot must be extended by quote signs which are written around the dates in tmp.dat.
Finally, the strcol(3) eq 'i' is necessary to circumvent a gnuplot bug, which causes a last line to be written with invalid data.
I've been searching for a while now to find out how to remove days of the week from a financial plot with no success.
I need the plot to just include the days of the week and completely miss out the weekends such that there is no 2-day gap in the financial chart.
I have the data in CSV format Open/Low/Close/High and it has the weekend data missing, it plots fine but I can't find how to not show the weekends, any help would be really appreciated.
I'd like to see it say M/T/W/T/F/M/T/W/T/F on the X basically rather than M/T/W/T/F/S/S/M etc...
Cheers,
Chris.
As far as I know, this cannot be done with gnuplot itself - you need to bring the file into the desired shape before. If you are on Linux, this can be done with something like
awk '{if( index( $1, "S" ) == 0 ) print $0 >> "new.dat"}' old.dat
where old.dat is your original file and new.dat the new file without weekends. I have assumed here that your data file has the weekday as the first entry in each line.
This would work under Windows as well, but you would need to install Gawk for Windows first.
The data is not shown in the file, the file is just weekday based and misses the weekends. If you plot the data you get these 2day gaps at the weekend so I want to remove these gaps. It's more really to do with the x axis having weekends in to make it linear.
Here is an example of part of the file:
2006-03-23T16:59 1.7470 1.7324 1.7471 1.7344 0.0000 0.0000 0.0000 0.0000
2006-03-24T16:59 1.7346 1.7308 1.7441 1.7428 0.0000 0.0000 0.0000 0.0000
2006-03-27T17:59 1.7424 1.7415 1.7492 1.7459 0.0000 0.0000 0.0000 0.0000
2006-03-28T17:59 1.7462 1.7422 1.7537 1.7424 0.0000 0.0000 0.0000 0.0000
If you look at the dates, there are gaps in the file. There should be gaps because these days have no data. The graph however should run without gaps and that is what I am trying to achieve.
I just came across set xdtics today. I doubt you're still working on this, but maybe that will be helpful for someone else... (see help xdtics)
Assuming your data file isn't missing any weekdays, you can treat the date column as a string type. (If you're missing weekdays, your chart will skip over those dates without allocating any space for them, which is easy to miss, so beware.)
I have a date as the first column in my data file in YYYY-MM-DD format. The data I'm plotting is in the second column. Here are the relevant lines of my gnuplot configuration:
set format x '%s'
plot 'file' using 0:2:xtic(substr(strcol(1),6,10))
The set format line tells gnuplot how to print the x labels. The using config uses column 0 (the index) as the x parameter, column 2 (the data) as the y parameter, and provides special instructions for printing labels: only print characters 6–10. (This chops off the year portion, which helps the label fit without overlapping in my case.)
Also see this SO answer. I wouldn't want to replicate this "broken axis" solution for every weekend, but perhaps it could serve to inspire.
If you want to neglect the weekends on the time scale you can simply define a function which returns the day number after time(0) omitting the weekends. Note, that time(0) is 1970-01-01 00:00:00 for gnuplot 5.x and 2000-01-01 00:00:00 for gnuplot 4.x.
dw5(t) returns day number after time(0) omitting the weekends and NaN if t is a weekend day.
dw5tow7(n) returns the date from the "5-day-week" day number and NaN if input is NaN.
dw7Tic(n) returns the date label for xtic and empty string '' if input is NaN.
For example, the command:
do for [i=0:10] { print sprintf("%s % 4d",strftime("%Y-%m-%d",dw5tow7(i)),i) }
will return in gnuplot5.x:
1970-01-01 0
1970-01-02 1
1970-01-05 2
1970-01-06 3
1970-01-07 4
1970-01-08 5
1970-01-09 6
1970-01-12 7
1970-01-13 8
1970-01-14 9
1970-01-15 10
Script: (works for gnuplot>=5.0.0, Jan. 2015)
### remove weekends on time scale
reset
FILE = "SO9680677.dat"
# create some random test data
set print FILE
t0 = time(0)
y0 = 100
do for [i=0:60] {
t = t0 + i*3600*24
if (int(tm_wday(t)+1)%7>1) {
print sprintf("%s %g",strftime("%Y-%m-%d",t),y0=y0+rand(0)*1-0.5)
}
}
set print
SecPerDay = 3600*24
SecPerWeek = 7*SecPerDay
isWeekend(t) = int(tm_wday(t)+1)%7 < 2
myTimeFmt = "%Y\n%m-%d"
tOff = tm_year(0)==1970 ? 3 : 5 # offset gnuplot5.x: 3, gnuplot4.x: 5
dw5(t) = isWeekend(t) ? NaN : int(t/SecPerDay) - 2*int((t+tOff*SecPerDay)/SecPerWeek)
dw5tow7(n) = n==n ? n*SecPerDay + (int(n+tOff)/5)*2*SecPerDay : NaN
dw7Tic(n) = n==n ? strftime(myTimeFmt,dw5tow7(n)) : ''
set key top center out noautotitle
set grid x,y
set ytics 1
set multiplot layout 2,1
set xrange[:] noextend
set format x myTimeFmt timedate
plot FILE u (timecolumn(1,"%Y-%m-%d")):2 w lp pt 7 lc rgb "red" ti "with weekends"
set format x "%g\n" numeric
plot FILE u (dw5(timecolumn(1,"%Y-%m-%d"))):2 w lp pt 7 lc rgb "web-green" ti "without weekends", \
'' u (t0=dw5(timecolumn(1,"%Y-%m-%d"))):(NaN):xtic(dw7Tic(t0)) every 5
unset multiplot
### end of script
Replace the above multiplot section with the snippet below and the script will run with gnuplot>=4.6.0 (March 2012). Maybe with further tweaking it can be made work with gnuplot 4.4.0.
### version for gnuplot 4.6.0, March 2012
set multiplot layout 2,1
set timefmt "%Y-%m-%d"
set xdata time
set format x myTimeFmt
plot FILE u 1:2 w lp pt 7 lc rgb "red" ti "with weekeends"
set format x "%g\n"
plot FILE u (dw5(timecolumn(1))):2 w lp pt 7 lc rgb "web-green" ti "without weekends", \
'' u (t0=dw5(timecolumn(1))):(NaN):xtic(dw7Tic(t0)) every 5 w p
unset multiplot
Result:
Actually, after all, the larger your time range the less you will notice if there are weekends or not.
Using some external tool (I would wrote a bash or python script for that, I believe; it should not be difficult), you can insert lines for weekend days (one line for a day) into your data file, like this:
2006-03-26T00:00 NaN NaN NaN NaN NaN NaN NaN NaN
(or you can just append those NaNs for weekends at the end of data file and use unique keyword)
and then plot, let's say, the first data with using 1:($2) with linespoints, not using 1:2 ...
This should work for you.