I reproduced the issue I have with my more complex code with the following example:
import numpy as np
from scipy.integrate import solve_ivp
def funct(t, y):
return np.sqrt(100-y)
def event(t, y):
return y-100
event.terminal = True
sol = solve_ivp(funct, [0, 50], np.array([0]), events=event, rtol=1e-9, atol=1e-12)
As you can see, the integration should stop when y = 100. For this reason, I created an event in order to detect that and stop the integration.
However, the function is being computed with values y > 100 because I get the warning:
RuntimeWarning: invalid value encountered in sqrt return np.sqrt(100-y)
I guess this happens because according to Scipy documentation
The solver will find an accurate value of t at which event(t, y(t)) = 0 using a root-finding algorithm.
Is there a way to avoid this from happening?
Related
In the documentation page of rv_continuous we can find a 'custom' gaussian being subclassed as follows.
from scipy.stats import rv_continuous
import numpy as np
class gaussian_gen(rv_continuous):
"Gaussian distribution"
def _pdf(self, x):
return np.exp(-x**2 / 2.) / np.sqrt(2.0 * np.pi)
gaussian = gaussian_gen(name='gaussian')
In turn, I attempted to create a class for an exponential distribution with base 2, to model some nuclear decay:
class time_dist(rv_continuous):
def _pdf(self, x):
return 2**(-x)
random_var = time_dist(name = 'decay')
This had the purpose of then calling random_var.rvs() in order to generate a randomly distributed sample of values according to the pdf I defined. However, when I run this, I receive an OverflowError, and I don't quite understand why. Initially I thought it had to do with the fact that the function was not normalized. However, I keep making changes to the _pdf definition to no avail. Is there anything wrong with the code or is this method just ill-advised for defining functions of this sort?
According to wikipedia, the pdf of an exponential distribution would be:
lambda * exp(-lambda*x) for x >= 0
0 for x < 0
So, probably the function should be changed as follows:
from scipy.stats import rv_continuous
import numpy as np
import matplotlib.pyplot as plt
class time_dist(rv_continuous):
def _pdf(self, x):
return np.log(2) * 2 ** (-x) if x >= 0 else 0
random_var = time_dist(name='decay')
plt.hist(random_var.rvs(size=500))
plt.show()
The gradients from tf.GradientTape seem not to match the correct minimum in the function I'm trying to minimise.
I'm trying to use tensorflowprobability's black-box variational inference (using tf2), with the tf.GradientTape, a keras optimizer, calling the apply_gradients function. The surrogate posterior is a simple 1d Normal. I'm trying to approximate a pair of normals, see pdist function. For simplicity I just try to optimise the scale parameter.
Current code:
from scipy.special import erf
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import tensorflow as tf
import tensorflow_probability as tfp
from tensorflow_probability import distributions as tfd
def pdist(x):
return (.5/np.sqrt(2*np.pi)) * np.exp((-(x+3)**2)/2) + (.5/np.sqrt(2*np.pi)) * np.exp((-(x-3)**2)/2)
def logpdist(x):
logp = np.log(1e-30+pdist(x))
assert np.all(np.isfinite(logp))
return logp
optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)
mu = tf.Variable(0.0,dtype=tf.float64)
scale = tf.Variable(1.0,dtype=tf.float64)
for it in range(100):
with tf.GradientTape() as tape:
surrogate_posterior = tfd.Normal(mu,scale)
elbo_loss = tfp.vi.monte_carlo_variational_loss(logpdist,surrogate_posterior,sample_size=10000)
gradients = tape.gradient(elbo_loss, [scale])
optimizer.apply_gradients(zip(gradients, [scale]))
if it%10==0: print(scale.numpy(),gradients[0].numpy(),elbo_loss.numpy())
Output (showing every 10th iteration):
SCALE GRAD ELBO_LOSS
1.100, -1.000, 2.697
2.059, -0.508, 1.183
2.903, -0.354, 0.859 <<< (right answer about here)
3.636, -0.280, 1.208
4.283, -0.237, 1.989
4.869, -0.208, 3.021
5.411, -0.187, 4.310
5.923, -0.170, 5.525
6.413, -0.157, 7.250
6.885, -0.146, 8.775
For some reason the gradient doesn't reflect the true gradient, which should be about zero around scale=2.74.
Why does the gradient not relate to the actual elbo_loss?
Hopefully someone can elaborate on why the previous implementation failed (and also why it doesn't except, but instead just has the wrong answer). Anyway, I found I could fix it by ensuring that key expressions used the tensorflow maths library and not numpy's. Specifically replacing the two methods above with;
def pdist(x):
return (.5/np.sqrt(2*np.pi)) * tf.exp((-(x+3)**2)/2) + (.5/np.sqrt(2*np.pi)) * tf.exp((-(x-3)**2)/2)
def logpdist(x):
return tf.math.log(pdist(x))
The stochastic optimisation now works.
Output:
2.020, -0.874, 1.177
2.399, -0.393, 0.916
2.662, -0.089, 0.857
2.761, 0.019, 0.850
2.765, 0.022, 0.843
2.745, -0.006, 0.851
2.741, 0.017, 0.845
2.752, 0.005, 0.852
2.744, 0.015, 0.852
2.747, 0.013, 0.862
I'm not going to accept my own answer as I'd be grateful if some answers could be given that give intuition about why this now works and why it failed previously (and why the failure mode wasn't an exception or similar but instead an incorrect gradient).
I try to build up my own loss function as follows
import numpy as np
from keras import backend as K
def MyLoss(self, x_input, x_reconstruct):
a = np.copy(x_reconstruct)
a = np.asarray(a, dtype='float16')
a = np.floor(4*a)/4
return K.mean(K.square(a - x_input), axis=-1)`
In compilation, it says
ValueError: setting an array element with a sequence
Both x_input and x_reconstruct are [m, n, 1] np arrays. The last line of code is actually copied directly from Keras' built-in MSE loss function.
Also, I suppose loss is calculated per sample. If dimensions of the input and reconstructed input are both [m, n, 1], the result of Keras' built-in loss will also be a matrix sized [m, n]. So why does it work properly?
I then tried to us np's functions directly by
def MyLoss(self, x_input, x_reconstruct):
a = np.copy(x_reconstruct)
a = np.asarray(a, dtype=self.precision)
a = np.floor(4*a)/4
Diff = a - x_input
xx = np.mean(np.square(Diff), axis=-1)
yy = np.sum(xx)
return yy
yet the error persists. What mistake did I make? How should write the code?
Having borrowed the suggestion from Make a Custom loss function in Keras in detail, I tried following
def MyLoss(self, x_input, x_reconstruct):
if self.precision == 'float16':
K.set_floatx('float16')
K.set_epsilon(1e-4)
a = K.cast_to_floatx(x_input)
a = K.round(a*4.-0.5)/4.0
return K.sum(K.mean(K.square(x_input-a), axis=-1))
But the same error happens
You can not use numpy arrays in your loss. You have to use TensorFlow or Keras backend operations. Try this maybe:
import tensorflow as tf
import keras.backend as K
def MyLoss(x_input, x_reconstruct):
a = tf.cast(x_input, dtype='tf.float16')
a = tf.floor(4*a)/4
return K.mean(K.square(a - x_input), axis=-1)
I found the answer myself, and let me share it here
If I write code like this
def MyLoss(self, y_true, y_pred):
if self.precision == 'float16':
K.set_floatx('float16')
K.set_epsilon(1e-4)
return K.mean(K.square(y_true-K.round(y_pred*4.-0.5)/4.0), axis=-1)
It works. The trick is, I think, that I cannot use 'K.cast_to_floatx(y_true)'. Instead, simply use y_true directly. I still do not understand why...
Recently I was working on some data for which I was able to obtain a curve using curve_fit after saving the plot and the values obtained I returned to the same code later only to find it does not work.
#! python 3.5.2
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
from scipy.optimize import curve_fit
data= np.array([
[24, 0.176644513],
[27, 0.146382841],
[30, 0.129891534],
[33, 0.105370908],
[38, 0.077820511],
[50, 0.047407538]])
x, y = np.array([]), np.array([])
for val in data:
x = np.append(x, val[0])
y = np.append(y, (val[1]/(1-val[1])))
def f(x, a, b):
return (np.exp(-a*x)**b)
# The original a and b values obtained
a = -0.2 # after rounding
b = -0.32 # after rounding
plt.scatter(x, y)
Xcurve = np.linspace(x[0], x[-1], 500)
plt.plot(Xcurve, f(Xcurve,a,b), ls='--', color='k', lw=1)
plt.show()
# the original code to get the values
a = b = 1
popt, pcov = curve_fit(f, x, y, (a, b))
Whereas, previously curve_fit returned the values a, b = -0.2, -0.32 now returns:
Warning (from warnings module):
File "C:/Users ... line 22
return (np.exp(-a*x)**b)
RuntimeWarning: overflow encountered in exp
The code as far as I am aware did not change. Thanks
Without knowing what changed in the code, it is hard to say what changed between your state of "working" and "not working". It may be that changes in the version of scipy you used give different results: there have changes to the underlying implementation in curve_fit() over the past few years.
But also: curve_fit() (and the underlying python and Fortran code it uses) requires reasonably good initial guesses for the parameters for many problems to work at all. With bad guesses for the parameters, many problems will fail.
Exponential decay problems seem to be especially challenging for the Levenberg-Marquardt algorithm (and the implementations used by curve_fit(), and do require reasonable starting points. It's also easy to get into a part of parameter space where the function evaluates to zero, and changes in the parameter values have no effect.
If possible, if your problem involves exponential decay, it is helpful to work in log space. That is, model log(f), not f itself. For your problem in particular, your model function is exp(-a*x)**b. Is that really what you mean? a and bwill be exactly correlated.
In addition, you may find lmfit helpful. It has a Model class for curve-fitting, using similar underlying code, but allows fixing or setting bounds on any of the parameters. An example for your problem would be (approximately):
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
from scipy.optimize import curve_fit
import lmfit
data= np.array([
[24, 0.176644513],
[27, 0.146382841],
[30, 0.129891534],
[33, 0.105370908],
[38, 0.077820511],
[50, 0.047407538]])
x, y = np.array([]), np.array([])
for val in data:
x = np.append(x, val[0])
y = np.append(y, (val[1]/(1-val[1])))
def f(x, a, b):
print("In f: a, b = " , a, b)
return (np.exp(-a*x)**b)
fmod = lmfit.Model(f)
params = fmod.make_params(a=-0.2, b=-0.4)
# set bounds on parameters
params['a'].min = -2
params['a'].max = 0
params['b'].vary = False
out = fmod.fit(y, params, x=x)
print(out.fit_report())
plt.plot(x, y)
plt.plot(x, out.best_fit, '--')
plt.show()
I am trying to learn about solving ODEs in python and scipy seemed like a good starting point. I've become comfortable with using odeint and am now trying to learn how to use ode. I tried running the sample code in the scipy docs, but it is returning an error. I've copied the code below alongside the error.
CODE
from scipy.integrate import ode
y0, t0 = [1.0j, 2.0], 0
def f(t, y, arg1):
return [1j*arg1*y[0] + y[1], -arg1*y[1]**2]
def jac(t, y, arg1):
return [[1j*arg1, 1], [0, -arg1*2*y[1]]]
r = ode(f, jac).set_integrator('zvode', method='bdf', with_jacobian=True)
r.set_initial_value(y0, t0).set_f_params(2.0).set_jac_params(2.0)
t1 = 10
dt = 1
while r.successful() and r.t < t1:
r.integrate(r.t+dt)
print("%g %g" % (r.t, r.y))
ERROR MESSAGE
print("%g %g" % (r.t, r.y))
TypeError: only length-1 arrays can be converted to Python scalars
r.y is a length 2 array. And that of complex numbers. Both is incompatible with the formatting instruction for one float.
Implicit conversion to string however works,
print "%g %s" % (r.t, r.y)