Cumulative data and extrapolation with gnuplot - gnuplot

Question

Having a list of dates and events which is not necessarily sorted by date
e.g. like
# Date Event
04.12.2018 -4
23.06.2018 5
04.10.2018 3
11.11.2018 -9
08.03.2018 -4
08.03.2018 2
11.11.2018 -3
I would like to sum up the events and do a (e.g. linear) extrapolation, e.g. when the data will hit a certain threshold (e.g. zero).
It looks like smooth frequency and smooth cumulative seemed to be made for this.
But I am struggeling with the following:
a) how can I add a start value (offset), e.g. StartValue = 500
plot $Data u (strftime("%d.%m.%Y",timecolumn(1,"%d.%m.%Y"))):($2+StartValue) smooth cumulative w l t "Cumulated Events"
doesn't do it.
b) how can I get the cumulative data? Especially if the data is not sorted by date?
set table "DataCumulative.dat"
plot $Data u (strftime("%d.%m.%Y",timecolumn(1,"%d.%m.%Y"))):2 smooth cumulative with table
unset table
This look similar to this question (GNUPLOT: saving data from smooth cumulative) but I don't get the expected numbers. In my example below in the file "DataCumulative.dat", I expected unique dates and basically the data from the lower plot. How to get this?
The code:
### start code
reset session
set colorsequence classic
# function for creating a random date between two dates
t(date_str) = strptime("%d.%m.%Y", date_str)
date_random(d0,d1) = strftime("%d.%m.%Y",rand(0)*(t(d1)-t(d0)) + t(d0))
# create some random date data
date_start = "01.01.2018"
date_end = "30.06.2018"
set print $Data
do for [i=1:1000] {
print sprintf("%s\t%g", date_random(date_start,date_end), floor(rand(0)*10-6))
}
set print
set xdata time
set timefmt "%d.%m.%Y"
set xtics format "%b"
set xrange[date_start:"31.12.2018"]
set multiplot layout 2,1
plot $Data u (strftime("%d.%m.%Y",timecolumn(1,"%d.%m.%Y"))):2 smooth frequency with impulses t "Events"
plot $Data u (strftime("%d.%m.%Y",timecolumn(1,"%d.%m.%Y"))):2 smooth cumulative w l t "Cumulated Events"
unset multiplot
# attempt to get cumulative data into datablock
set table "DataCumulative.dat"
plot $Data u (strftime("%d.%m.%Y",timecolumn(1,"%d.%m.%Y"))):2 smooth cumulative with table
unset table
### end of code
The plots:

Answer 1

I guess, I finally got it now. However, there are a few learnings which I still don't understand completely.
1.
In order to get the cumulative data you should not set
set table $DataCumulative
plot $Data u (stringcolumn(1)):2 smooth cumulative with table
unset table
but instead:
set table $DataCumulative
plot $Data u (stringcolumn(1)):2 smooth cumulative
unset table
note the missing "with table" in the plot command.
The first version gives you the original data, the second one the desired cumulative data. But I don't yet understand why.
2.
the default datafile separator setting
which is
set datafile separator whitespace
it doesn't seem not to work. It will give an error message like line xxx: No data to fit
instead, you have to set
set datafile separator " \t" # space and TAB
But I don't understand why.
3.
fitting time date
f_lin(x) = m*x + c
won't give a good fit at all. Apparently, you have to subtract the start date and do the fitting.
f_lin(x) = m*(x-strptime("%d.%m.%Y", Date_Start)) + c
I remember reading this long time ago in the gnuplot documention but I can't find it anymore.
For the time being, I am happy now with the following.
The modified code:
### generate random date between two dates
reset session
# function for creating a random date between two dates
t(date_str) = strptime("%d.%m.%Y", date_str)
date_random(d0,d1) = strftime("%d.%m.%Y",rand(0)*(t(d1)-t(d0)) + t(d0))
# create some random date data
Date_Start = "01.01.2018"
Date_End = "30.06.2018"
set print $Data
do for [i=1:100] {
print sprintf("%s\t%g", date_random(Date_Start,Date_End), floor(rand(0)*10-6))
}
set print
set xdata time
set timefmt "%d.%m.%Y"
# get cumulative data into datablock
set xtics format "%d.%m.%Y"
set table $DataCumulative
plot $Data u (stringcolumn(1)):2 smooth cumulative
unset table
set xtics format "%b"
set datafile separator " \t" # space and TAB
# linear function and fitting
f_lin(x) = m*(x-strptime("%d.%m.%Y", Date_Start)) + c
set fit nolog quiet
fit f_lin(x) $DataCumulative u 1:2 via m,c
Level_Start = 500
Level_End = 0
x0 = (Level_End - Level_Start - c)/m + strptime("%d.%m.%Y", Date_Start)
set multiplot layout 3,1
# event plot & cumulative plot
set xrange[Date_Start:"31.12.2018"]
set xtics format ""
set lmargin 7
set bmargin 0
plot $Data u (timecolumn(1,"%d.%m.%Y")):2 smooth frequency with impulses lc rgb "red" t "Events 2018"
set xtics format "%b"
set bmargin
plot $Data u (timecolumn(1,"%d.%m.%Y")):2 smooth cumulative w l lc rgb "web-green" t "Cumulated Events 2018"
# fit & extrapolation plot
set label 1 at x0, graph 0.8 strftime("%d.%m.%Y",x0) center
set arrow 1 from x0, graph 0.7 to x0, Level_End
set key at graph 0.30, graph 0.55
set xrange[Date_Start:x0+3600*24*50] # end range = extrapolated date + 50 days
set xtics format "%m.%y"
set yrange [-90:]
plot $DataCumulative u (timecolumn(1,"%d.%m.%Y")):($2+Level_Start) w l lc rgb "blue" t "Cumulated Events",\
Level_End w l lc rgb "red" not,\
f_lin(x)+Level_Start w l ls 0 t "Fitting \\& Extrapolation"
unset multiplot
### end of code
will result in: