Related
I am trying to fit a piecewise polynomial function
Code:
import numpy as np
import scipy
from scipy.interpolate import UnivariateSpline, splrep
from scipy.optimize import curve_fit
from matplotlib import pyplot as plt
def piecewise_func(x, X, Y):
"""
cond_l: condition list
func_l: function list
"""
spl = UnivariateSpline(X, Y, k=3, s=0.5)
tck = (spl._data[8], spl._data[9], 3) # tck = (knots, coefficients, degree)
p = scipy.interpolate.PPoly.from_spline(tck)
cond_l = []
func_l = []
for idx, i in enumerate(range(3, len(spl.get_knots()) + 3 - 1)):
cond_l.append([(x >= p.x[i] & x < p.x[i + 1])])
func_l.append([lambda x: p.c[3, i] + p.c[2, i] * x + p.c[1, i] * x ** 2 + p.c[0, i] * x ** 3])
return np.piecewise(x, cond_l, func_l)
if __name__ == '__main__':
xdata = [0.28190937, 0.63429607, 0.91620544, 1.68793236, 2.32350115, 2.95215219, 4.5,
4.78103382, 7.2, 7.53430054, 8.03627018, 9., 9.86212529, 11.25951191, 11.62658532, 11.65598578, 13.90295926]
ydata = [0.36273168, 0.81614628, 1.17887796, 1.4475374, 5.52692706, 2.17548169, 3.55313396, 3.80326533, 7.75556311, 8.30176616, 10.72117182, 11.2499386,
11.72296513, 11.02146624, 14.51260631, 20.59365525, 21.77847853]
spl = UnivariateSpline(xdata, ydata, k=3, s=1)
plt.plot(xdata, ydata, '*')
plt.plot(xdata, spl(xdata))
plt.show()
p, e = curve_fit(piecewise_func, xdata, ydata)
# x_plot = np.linspace(0., 0.15, len(x))
# plt.plot(x, y, "+")
# plt.plot(x, (piecewise_func(x_plot, *p)), 'C3-', lw=3)
I tried the UnivariateSpline function to interpolate, I see the following result
However, I don't want the polynomial curve to pass through all data points. I tried varying the smoothing factor but I am not able to obtain something like the one below.
Expected output:
I'm trying curve fitting (Use UnivariateSpline to fit data tightly) to get the expected output and I have the following issues.
piecewise_func in the code posted returns the piecewise polynomial.
Passing this to curve_fit(piecewise_func, xdata, ydata) returns an error
Error:
res = leastsq(func, p0, Dfun=jac, full_output=1, **kwargs)
ValueError: diff requires input that is at least one dimensional
I am not sure what is wrong.
Suggestions on how to get the expected fit will be
of great help.
I would recommend having a closer look at the parameter s in the UnivariateSpline documentation:
s : float or None, optional
Positive smoothing factor used to choose the number of knots. Number of knots will be increased until the smoothing condition is satisfied:
sum((w[i] * (y[i]-spl(x[i])))**2, axis=0) <= s
If s is None, s = len(w) which should be a good value if 1/w[i] is an estimate of the standard deviation of y[i]. If 0, spline will interpolate through all data points. Default is None.
Since you do not set w, this is just a complicated way of saying that s is the least squares error that you allow, i.e., squared errors summed over all the data points. Your value of 1 does not lead to interpolation but it is quite tight compared to what you want to achieve.
Taking
spl = UnivariateSpline(xdata, ydata, k=3, s=10)
you get the following:
Yet closer to your goal is s=100:
So my recommendation is to play around with s and if that proves insufficient, to ask a new question describing what you need more precisely. I haven't had a proper look at the problem with piecewise_func.
There are a few posts about this and normally the answer is to have a good initial guess and bounds. I've played around with it for a while and cannot find a configuration that produces any sort of curve.
import numpy as np
array1 = np.array(column1).astype(float)
array2 = np.array(column2).astype(float)
print(array1)
print(array2)
Output:
[18.7327 9.3784 6.6293 20.8361 11.2603 19.3706 5.4302 10.1293 13.7516
8.0567 16.8688 4.969 3.94 19.4793 11.7527 13.2811 13.338 0.5944
7.4406 11.2338 6.2283 3.4818 10.1056 16.2689 22.442 18.7345 5.2605
5.6405 12.7186 18.2497 5.4315 14.2651 16.7544 12.9192 13.5955 10.9256
5.7798 8.4485 8.5229 11.879 6.5271 10.3376 7.781 31.4558 8.0236
2.3527 10.8926 16.1995 11.1924 25.8071 13.9692 20.7791 10.3045 12.2833
7.4066 15.9807 11.4462 15.1504 5.9021 19.1184]
[83.85 52.45 41.2 92.59 62.65 86.77 30.63 53.78 73.34 48.55 82.53 28.3
23.87 90.99 62.95 68.82 71.06 20.74 45.25 60.65 39.07 21.93 53.35 79.61
93.27 85.88 28.95 32.73 65.89 83.51 30.74 75.22 79.8 67.43 71.12 58.41
35.83 49.61 50.72 63.49 40.67 55.75 46.49 96.22 47.62 21.8 56.23 76.97
59.07 94.67 74.9 92.52 55.61 63.51 41.34 76.8 62.81 75.99 36.34 85.96]
import pylab
from scipy.optimize import curve_fit
def sigmoid(x, a, b):
y = 1 / (1 + np.exp(-b*(x-a)))
return y
popt, pcov = curve_fit(sigmoid, array1, array2, p0 = [5,20], method='dogbox', bounds=([0, 20],[40, 100]))
print(popt)
x = np.linspace(0, 35, 50)
y = sigmoid(x, *popt)
pylab.plot(array1, array2, 'o', label='data')
pylab.plot(x,y, label='fit')
pylab.ylim(0, 100)
pylab.legend(loc='best')
pylab.show()
Output:
Graph
As you can see it just not doing anything at all. Would really appreciate any help on this to get a rough sigmoid curve. Doesn't need to be super accurate.
Many Thanks.
In your case, the problem wasn't a good initial guess, but an inappropriate model. Note how your sigmoid cannot be larger than 1, yet your data is in the range of ~10 - 100.
xs = np.linspace(0, 15)
as_ = np.linspace(0, 5, num=10)
bs_ = np.linspace(0, 5, num=10)
for a in as_:
for b in bs_:
plt.plot(xs, sigmoid(xs, a, b))
Therefore, you either have to modify your model to accept a scaling parameter, or scale down your data to a range your model can fit. Here's the two solutions:
Preamble
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
import pandas as pd
array1 = np.array([18.7327,9.3784,6.6293,20.8361,11.2603,19.3706,5.4302,10.1293,13.7516,8.0567,16.8688,4.969,3.94,19.4793,11.7527,13.2811,13.338,0.5944,7.4406,11.2338,6.2283,3.4818,10.1056,16.2689,22.442,18.7345,5.2605,5.6405,12.7186,18.2497,5.4315,14.2651,16.7544,12.9192,13.5955,10.9256,5.7798,8.4485,8.5229,11.879,6.5271,10.3376,7.781,31.4558,8.0236,2.3527,10.8926,16.1995,11.1924,25.8071,13.9692,20.7791,10.3045,12.2833,7.4066,15.9807,11.4462,15.1504,5.9021,19.1184])
array2 = np.array([83.85,52.45,41.2,92.59,62.65,86.77,30.63,53.78,73.34,48.55,82.53,28.3,23.87,90.99,62.95,68.82,71.06,20.74,45.25,60.65,39.07,21.93,53.35,79.61,93.27,85.88,28.95,32.73,65.89,83.51,30.74,75.22,79.8,67.43,71.12,58.41,35.83,49.61,50.72,63.49,40.67,55.75,46.49,96.22,47.62,21.8,56.23,76.97,59.07,94.67,74.9,92.52,55.61,63.51,41.34,76.8,62.81,75.99,36.34,85.96])
df = pd.DataFrame({'x':array1, 'y':array2})
df = df.sort_values('x')
Scaling data to match parameter
def sigmoid(x, a, b):
y = 1 / (1 + np.exp(-b*(x-a)))
return y
popt, pcov = curve_fit(sigmoid, df['x'], df['y'] / df['y'].max(), p0 = [5,20], method='dogbox', bounds=([0, 0],[40, 100]))
plt.plot(df['x'], df['y'] / df['y'].max(), label='data')
plt.plot(df['x'], sigmoid(df['x'], *popt))
popt is [8.56754823 0.20609918]
Adding new parameter to function
def sigmoid2(x, a, b, scale):
y = scale / (1 + np.exp(-b*(x-a)))
return y
popt, pcov = curve_fit(sigmoid2, df['x'], df['y'], p0 = [5,20, 100], method='dogbox', bounds=([0, 0, 0],[40, 100, 1E5]))
plt.plot(df['x'], df['y'], label='data')
plt.plot(df['x'], sigmoid2(df['x'], *popt))
popt is array([ 8.81708442, 0.19749557, 98.357044 ])
In the following curve, I would like to extend the measurements beyond x=1 in order to have a better estimate of the green curve compared to red line.
Note: I do not have the analytical form of the function but only x, y data sets in the range (0, 1).
Is there any package in python in order to extrapolate a curve beyond some value given that we have the interpolated form of the curve?
Here is my attempt assuming a linear drop:
from scipy.interpolate import interp1d
import numpy as np
CURVE_CUT_INDEX = #the index corresponding to x=1 in the x array
def extrapolator_function(x_vals, y_vals, x_list):
interpolator = interp1d(x_vals, y_vals, kind='cubic')
x_1 = x_vals[-1]
y_1 = interpolator(x_1)
y_grad, x_grad = (np.gradient(y_vals, np.arange(y_vals.size)),
np.gradient(x_vals, np.arange(x_vals.size)))
slope = np.divide(y_grad, x_grad, out=np.zeros_like(y_grad), where=x_grad != 0)[-1]
x_out = x_list[CURVE_CUT_INDEX + 1:]
y_pred = np.array([slope * (x-x_1) + y_1 for x in x_out])
return x_vals, y_vals, x_out, y_pred
def plotter(ax, x_list, y_list):
x_vals, y_vals = x_list[0:CURVE_CUT_INDEX + 1], y_list(x_list)[0:CURVE_CUT_INDEX + 1]
x_vals, y_vals, x_out, y_pred = extrapolator_function(x_vals, y_vals, x_list)
return ax.plot(x_vals, y_vals, 'g-', x_out, y_pred, 'r-', alpha=1, lw=2)
which will result in the following extrapolation scheme (which is not what I want).
The figure above is a great artwork showing the wind speed, wind direction and temperature simultaneously. detailedly:
The X axes represent the date
The Y axes shows the wind direction(Southern, western, etc)
The variant widths of the line were stand for the wind speed through timeseries
The variant colors of the line were stand for the atmospheric temperature
This simple figure visualized 3 different attribute without redundancy.
So, I really want to reproduce similar plot in matplotlib.
My attempt now
## Reference 1 http://stackoverflow.com/questions/19390895/matplotlib-plot-with-variable-line-width
## Reference 2 http://stackoverflow.com/questions/17240694/python-how-to-plot-one-line-in-different-colors
def plot_colourline(x,y,c):
c = plt.cm.jet((c-np.min(c))/(np.max(c)-np.min(c)))
lwidths=1+x[:-1]
ax = plt.gca()
for i in np.arange(len(x)-1):
ax.plot([x[i],x[i+1]], [y[i],y[i+1]], c=c[i],linewidth = lwidths[i])# = lwidths[i])
return
x=np.linspace(0,4*math.pi,100)
y=np.cos(x)
lwidths=1+x[:-1]
fig = plt.figure(1, figsize=(5,5))
ax = fig.add_subplot(111)
plot_colourline(x,y,prop)
ax.set_xlim(0,4*math.pi)
ax.set_ylim(-1.1,1.1)
Does someone has a more interested way to achieve this? Any advice would be appreciate!
Using as inspiration another question.
One option would be to use fill_between. But perhaps not in the way it was intended. Instead of using it to create your line, use it to mask everything that is not the line. Under it you can have a pcolormesh or contourf (for example) to map color any way you want.
Look, for instance, at this example:
import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import interp1d
def windline(x,y,deviation,color):
y1 = y-deviation/2
y2 = y+deviation/2
tol = (y2.max()-y1.min())*0.05
X, Y = np.meshgrid(np.linspace(x.min(), x.max(), 100), np.linspace(y1.min()-tol, y2.max()+tol, 100))
Z = X.copy()
for i in range(Z.shape[0]):
Z[i,:] = c
#plt.pcolormesh(X, Y, Z)
plt.contourf(X, Y, Z, cmap='seismic')
plt.fill_between(x, y2, y2=np.ones(x.shape)*(y2.max()+tol), color='w')
plt.fill_between(x, np.ones(x.shape) * (y1.min() - tol), y2=y1, color='w')
plt.xlim(x.min(), x.max())
plt.ylim(y1.min()-tol, y2.max()+tol)
plt.show()
x = np.arange(100)
yo = np.random.randint(20, 60, 21)
y = interp1d(np.arange(0, 101, 5), yo, kind='cubic')(x)
dv = np.random.randint(2, 10, 21)
d = interp1d(np.arange(0, 101, 5), dv, kind='cubic')(x)
co = np.random.randint(20, 60, 21)
c = interp1d(np.arange(0, 101, 5), co, kind='cubic')(x)
windline(x, y, d, c)
, which results in this:
The function windline accepts as arguments numpy arrays with x, y , a deviation (like a thickness value per x value), and color array for color mapping. I think it can be greatly improved by messing around with other details but the principle, although not perfect, should be solid.
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.collections import LineCollection
x = np.linspace(0,4*np.pi,10000) # x data
y = np.cos(x) # y data
r = np.piecewise(x, [x < 2*np.pi, x >= 2*np.pi], [lambda x: 1-x/(2*np.pi), 0]) # red
g = np.piecewise(x, [x < 2*np.pi, x >= 2*np.pi], [lambda x: x/(2*np.pi), lambda x: -x/(2*np.pi)+2]) # green
b = np.piecewise(x, [x < 2*np.pi, x >= 2*np.pi], [0, lambda x: x/(2*np.pi)-1]) # blue
a = np.ones(10000) # alpha
w = x # width
fig, ax = plt.subplots(2)
ax[0].plot(x, r, color='r')
ax[0].plot(x, g, color='g')
ax[0].plot(x, b, color='b')
# mysterious parts
points = np.array([x, y]).T.reshape(-1, 1, 2)
segments = np.concatenate([points[:-1], points[1:]], axis=1)
# mysterious parts
rgba = list(zip(r,g,b,a))
lc = LineCollection(segments, linewidths=w, colors=rgba)
ax[1].add_collection(lc)
ax[1].set_xlim(0,4*np.pi)
ax[1].set_ylim(-1.1,1.1)
fig.show()
I notice this is what I suffered.
I'm new to python (and programming in general) and want to make a polynomial fit using curve_fit, where the order of the polynomials (or the number of fit parameters) is variable.
I made this code which is working for a fixed number of 3 parameters a,b,c
# fit function
def fit_func(x, a,b,c):
p = np.polyval([a,b,c], x)
return p
# do the fitting
popt, pcov = curve_fit(fit_func, x_data, y_data)
But now I'd like to have my fit function to only depend on a number N of parameters instead of a,b,c,....
I'm guessing that's not a very hard thing to do, but because of my limited knowledge I can't get it work.
I've already looked at this question, but I wasn't able to apply it to my problem.
You can define the function to be fit to your data like this:
def fit_func(x, *coeffs):
y = np.polyval(coeffs, x)
return y
Then, when you call curve_fit, set the argument p0 to the initial guess of the polynomial coefficients. For example, this plot is generated by the script that follows.
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt
# Generate a sample input dataset for the demonstration.
x = np.arange(12)
y = np.cos(0.4*x)
def fit_func(x, *coeffs):
y = np.polyval(coeffs, x)
return y
fit_results = []
for n in range(2, 6):
# The initial guess of the parameters to be found by curve_fit.
# Warning: in general, an array of ones might not be a good enough
# guess for `curve_fit`, but in this example, it works.
p0 = np.ones(n)
popt, pcov = curve_fit(fit_func, x, y, p0=p0)
# XXX Should check pcov here, but in this example, curve_fit converges.
fit_results.append(popt)
plt.plot(x, y, 'k.', label='data')
xx = np.linspace(x.min(), x.max(), 100)
for p in fit_results:
yy = fit_func(xx, *p)
plt.plot(xx, yy, alpha=0.6, label='n = %d' % len(p))
plt.legend(framealpha=1, shadow=True)
plt.grid(True)
plt.xlabel('x')
plt.show()
The parameters of polyval specify p is an array of coefficients from the highest to lowest. With x being a number or array of numbers to evaluate the polynomial at. It says, the following.
If p is of length N, this function returns the value:
p[0]*x**(N-1) + p[1]*x**(N-2) + ... + p[N-2]*x + p[N-1]
def fit_func(p,x):
z = np.polyval(p,x)
return z
e.g.
t= np.array([3,4,5,3])
y = fit_func(t,5)
503
which is if you do the math here is right.