Let say I have got two time series and I would like to know if there is some kind of diminishing return property between those two time series. Namely, there is a decrease in the marginal output of the first one as you increase the second one. How would you proceed? I guess you should detrend the time series first, eventually removing the seasonality as well? Do you have example on internet/books of such study?
Related
So for an assignment I have to find the schedule that minimizes the sum of absolute differences
between the demanded and scheduled number of workers per time interval by solving an
integer linear optimization model.
So I modeled my schedule as a set cover problem and created a row with the demanded number of workers and a row with the actual number of workers.
I take the summation of the absolute differences between the rows as object and try to minimize that.
=SUM(ABS(C39:Z39-C33:Z33))
However I get the error "The linearity conditions required by LP solver are not satisfied" and I don't get why since the Linearity report says yes on everything.
*X_i is the number of times a shift is chosen.
ABS() is not a linear function. Who knows why excel doesn't call that out... it's internal solver does not have a great reputation.
You might try to just change your OBJ function to some penalty * uncovered jobs and see if you can get your mode up & running. Then maybe subtract the used workers from the available, sum that up and add in a penalty for unused workers....
As #AirSquid has already pointed out the absolute value is not a linear function. However in your context it is possible to linearize it. You can us that
minimizing abs(sum x_i)
is equivalent to
minimising sum a_i, where a_i are new variables with constraints a_i>=x_i, a_i>=-x_i.
I am evaluating VictoriaMetrics for an IoT application where we sometimes have gaps in a series due to hardware or communication issues. In some time series reporting situations it is helpful for us to interpolate values for the missing time intervals. I see that MetricsQL (which extends PromQL) has a keep_last_value() function that will fill gaps by holding the last observed value until a new one appears (which will be helpful to us) but in some situations a linear interpolation between the values before and after the gap is a more realistic estimate for the missing portion. Is there a function in PromQL or MetricsQL that will do linear interpolation of missing data in a series, or is it possible to construct a more complex query that will achieve this?
Clarifying the desired interpolation
What I would like is a simple interpolation between the points immediately before and after the gap; this is, I believe, what TimescaleDB's interpolate() function does. In other words, if my time series is:
(1:00, 2)
(2:00, 4)
(3:00, NaN)
(4:00, 5)
(5:00, 1)
I would like the interpolated 3:00 value to be 4.5, half way between the points immediately before and after. I don't want it to be 6 (which is what I would get by extrapolating from the points before the missing one, ignoring the points after) and I don't want whatever value I would get if I did linear regression on the whole series and interpolated at 3:00 (presumably 3, or something close to it).
Of course, this is a simple illustration and it's also possible that the gap could last more than one time step. But in that case I would still like the interpolation to be based solely off of the points immediately before and immediately after the gap, ignoring the rest of the series.
Final answer
Use the interpolate function, now available in VictoriaMetrics starting from v1.38.0.
Original suggestion
This does not achieve the exact interpolation requested in the revised question, but may be useful for others with slightly different requirements
Try combining predict_linear function with default operator from MetricsQL in the following way:
metric default predict_linear(metric[1h], 0)
Try modifying the value in square brackets in order to get the desired level of interpolation.
So I have two subsets of data that represent two situations. The one that look more consistent needs to be filtered out (they are noise) while the one looks random are kept (they are motions). The method I was using was to define a moving window = 10 and whenever the standard deviation of the data within the window was smaller than some threshold, I suppressed them. However, this method could not filter out all "consistent" noise while also hurting the inconsistent one (real motion). I was hoping to use some kinds of statistical models and not machine learning to accomplish this. Any suggestions would be appreciated!
noise
real motion
The Kolmogorov–Smirnov test is used to compare two samples to determine if they come from the same distribution. I realized that real world data would never be uniform. So instead of comparing my noise data against the uniform distribution, I used scipy.stats.ks_2samp function to compare any bursts against one real motion burst. I then muted the motion if the return p-value is significantly small, meaning I can reject the hypothesis that two samples are from the same distribution.
I had a question about solving a weighted interval scheduling problem given a fixed number of classrooms. So, initially, we are given a set of intervals, each with a starting time and finishing time, and each with a weight. So, the aim of the problem is to find a scheduling in two classrooms that maximizes the weight. Is there an efficient way to do this by dynamic programming?
My approach was trivial, since I built an algorithm that simply maximizes the intervals for each classroom. Is there a better way to do this?
My idea is not fully dynamic programming. But I think it will help.
Sort all classes by their starting time.
Now for a class i find next class j which start time is greater or equal then this end time. (Using binary search you can find this because we have an sorted array which is sorted by starting time)
Assume max_so_far is an array and max_so_far[z] contain the max_weight class from z to last
For all i find the max of summation of weight of class[i] and weight max_so_far[j]
Please find the code here
Time complexity of this code is O(nLog(n)).
I've been tasked with using fmin_tnc for an optimization problem. For the time being, I am only allowed to use fmin_tnc. I would like to display the number of iterations whether or not the results converge. Upon convergence (or line search error, which I often get), I receive a string of numbers, but they are not labeled so I'm not sure if one of them is an iteration number or function evaluation.
Additionally, I would like to store the values of my function output every time fmin_tnc iterates it.
So far, I can only find answers revolving around "fmin" but not "fmin_tnc". The code for the optimizer is as follows (unfortunately, I am not allowed to show more than this):
optimize.fmin_tnc(func1, x0, approx_grad=True, bounds=(bounds), epsilon=0.001, messages=15, stepmx=21, ftol=1e-06, xtol=1e-6)
Thank you very much!