Custom loss function in Keras based on the input data - keras

I am trying to create the custom loss function using Keras. I want to compute the loss function based on the input and predicted the output of the neural network.
I tried using the customloss function in Keras. I think y_true is the output that we give for training and y_pred is the predicted output of the neural network. The below loss function is same as "mean_squared_error" loss in Keras.
def customloss(y_true, y_pred):
return K.mean(K.square(y_pred - y_true), axis=-1)
I would like to use the input to the neural network also to compute the custom loss function in addition to mean_squared_error loss. Is there a way to send an input to the neural network as an argument to the customloss function.
Thank you.

I have come across 2 solutions to the question you asked.
You can pass your input (scalar only) as an argument to the custom loss wrapper function.
def custom_loss(i):
def loss(y_true, y_pred):
return K.mean(K.square(y_pred - y_true), axis=-1) + something with i...
return loss
def baseline_model():
# create model
i = Input(shape=(5,))
x = Dense(5, kernel_initializer='glorot_uniform', activation='linear')(i)
o = Dense(1, kernel_initializer='normal', activation='linear')(x)
model = Model(i, o)
model.compile(loss=custom_loss(i), optimizer=Adam(lr=0.0005))
return model
This solution is also mentioned in the accepted answer here
You can pad your label with extra data columns from input and write a custom loss. This is helpful if you just want one/few feature column(s) from your input.
def custom_loss(data, y_pred):
y_true = data[:, 0]
i = data[:, 1]
return K.mean(K.square(y_pred - y_true), axis=-1) + something with i...
def baseline_model():
# create model
i = Input(shape=(5,))
x = Dense(5, kernel_initializer='glorot_uniform', activation='linear')(i)
o = Dense(1, kernel_initializer='normal', activation='linear')(x)
model = Model(i, o)
model.compile(loss=custom_loss, optimizer=Adam(lr=0.0005))
return model
model.fit(X, np.append(Y_true, X[:, 0], axis =1), batch_size = batch_size, epochs=90, shuffle=True, verbose=1)
This solution can be found also here in this thread.
I have only used the 2nd method when I had to use input feature columns in the loss. The first method can be only used with scalar arguments as mentioned in the comments.

You could wrap your custom loss with another function that takes the input tensor as an argument:
def customloss(x):
def loss(y_true, y_pred):
# Use x here as you wish
err = K.mean(K.square(y_pred - y_true), axis=-1)
return err
return loss
And then compile your model as follows:
model.compile('sgd', customloss(x))
where x is your input tensor.
NOTE: Not tested.

Related

Custom adaptive loss function with additional dynamic argument in Keras

I have to use an adaptive custom loss function that takes an additional dynamic argument (eps) in keras. The argument eps is a scalar but changes from one sample to the other : the loss function should be therefore adapted during training. I use a generator and I can pass this argument through every call of the generator during training (generator_train[2]). Based on answers to similar questions I tried to write the following wrapping:
def custom_loss(eps):
def square_err(y_true, y_pred):
nom = K.sum(K.square(y_pred - y_true), axis=-1)
denom = eps**2
loss = nom/denom
return loss
return square_err
But I am struggling with implementing it since eps is a dynamic variable: I don't know how I should pass this argument to the loss function during training (model.fit). Here is a simple version of my model:
model = keras.Sequential()
model.add(layers.LSTM(units=32, input_shape=(32, 4))
model.add(layers.Dense(units=1))
model.add_loss(custom_loss)
opt = keras.optimizers.Adam()
model.compile(optimizer=opt)
history = model.fit(x=generator_train[0], y=generator_train[1],
steps_per_epoch=100
epochs=50,
validation_data=gen_vl,
validation_steps=n_vl)
Your help would be very appreciated.
Simply pass "sample weights", which will be 1/(eps**2) for each sample.
Your generator should just output x, y, sample_weights and that's all.
Your loss can be:
def loss(y_true, y_pred):
return K.sum(K.square(y_pred - y_true), axis=-1)
In fit, you cannot use indexing in the generator, you will pass just generator_train, no x, no y, just generator_train.

Keras custom loss - Combining loss of multiple branch

I have a 2 branch network where one branch outputs regression value and another branch outputs classification label.
model = Model(inputs=inputs, outputs=[output1, output2])
model.compile(loss=[my_loss_reg, my_loss_class], optimizer='adam')
I want to implement a custom loss function (my_loss_reg()) for the regression branch such that at the regression end I want to add a fraction of the classification loss as follows,
def my_loss_reg(y_true, y_pred):
loss_mse=K.mean(K.sum(K.square(y_true-y_pred)))
#loss_reg = calculate_classification_loss() # How to implement this?
final_loss = some_function(loss_mse, loss_reg) # Can calculate only if loss_reg is available
return final_loss
The y_true and y_pred are true and predicted regression values at the regression branch. To calculate the classifcation loss I need the true and predicted classifcation labels, which is not available in my_loss_reg().
My question is how to calculate or access the classifcation loss at the regression end of the network? Similarly, I want to get the regression loss at the classification end while calulating the custom loss function my_loss_class() for the classification.
How can I do that? Any code snippets will be helpful. I found this solution but this is no longer valid with the latest version of Tensorflow and Keras.
All you need is simply available in native keras
you can automatically combine multiple losses using loss_weights parameter
In the example below I tried to reproduce a task where I combined an mse loss for the regression and a sparse_categorical_crossentropy for the classification task
features,n_sample,n_class = 10, 200, 3
X = np.random.uniform(0,1, (n_sample,features))
y = np.random.randint(0,n_class, n_sample)
inp = Input(shape=(features,))
x = Dense(64, activation='relu')(inp)
hidden = Dense(16, activation='relu')(x)
x = Dense(64, activation='relu')(hidden)
out_reg = Dense(features, name='out_reg')(x) # output regression
x = Dense(32, activation='relu')(hidden)
out_class = Dense(n_class, activation='softmax', name='out_class')(x) # output classification
model = Model(inp, [out_reg,out_class])
model.compile(optimizer='adam',
loss = {'out_reg':'mse', 'out_class':'sparse_categorical_crossentropy'},
loss_weights = {'out_reg':1., 'out_class':0.5})
model.fit(X, [X,y], epochs=10)
In this specific case, the loss is the result of 1*out_reg + 0.5*out_class
if you want to put your custom losses you simply have to do in this way
def my_loss_reg(y_true, y_pred):
return ...
def my_loss_class(y_true, y_pred):
return ...
model.compile(optimizer='adam',
loss = {'out_reg':my_loss_reg, 'out_class':my_loss_class},
loss_weights = {'out_reg':1., 'out_class':0.5})
model.fit(X, [X,y], epochs=10)

How to track weights and gradients in a Keras custom training loop

I have defined the following custom model and training loop in Keras:
class CustomModel(keras.Model):
def train_step(self, data):
x, y = data
with tf.GradientTape() as tape:
y_pred = self(x, training=True) # Forward pass
loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)
trainable_vars = self.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
self.optimizer.apply_gradients(zip(gradients, trainable_vars))
self.compiled_metrics.update_state(y, y_pred)
return {m.name: m.result() for m in self.metrics}
And I am using the following code to train the model on a simple toy data set:
inputs = keras.layers.Input(shape=(1,))
hidden = keras.layers.Dense(1, activation='tanh')(inputs)
outputs = keras.layers.Dense(1)(hidden)
x = np.arange(0, 2*np.pi, 2*np.pi/100)
y = np.sin(x)
nnmodel = CustomModel(inputs, outputs)
nnmodel.compile(optimizer=keras.optimizers.SGD(lr=0.1), loss="mse", metrics=["mae"])
nnmodel.fit(x, y, batch_size=100, epochs=2000)
I want to be able to see the values of the gradient and the trainable_vars variables in the train_step function for each training loop, and I am not sure how to do this.
I have tried to set a break point inside the train_step function in my python IDE and expecting it to stop at the break point for each epoch of the training after I call model.fit() but this didn't happen. I also tried to have them print out the values in the log after each epoch but I am not sure how to achieve this.

How to create a custom loss function in Keras that evaluates prediction after each epoch?

I'm working on a neural network in Keras that translates English sentences into a custom language. For this, I'd like to create a custom loss function that takes the prediction for each sentence and evaluates whether it complies with the grammar rules of the custom language and if not adds value to the standard loss function.
How can I evaluate a tensor after each epoch but not during compilation?
Below is my custom loss function. As during compilation of the model there is no batch yet, y_pred has the shape (None, x, y) and can't be evaluated to get the prediction. My idea to circumvent this was to assign a standard loss function during compilation and when batches arrive calculate the custom loss. Unfortunately the custom loss is never reached.
def custom_loss(tokenizer, punishment_rate):
def compile_loss(y_true, y_pred):
shape = K.int_shape(y_pred)
#standard loss function
loss = K.sparse_categorical_crossentropy(y_true, y_pred)
if shape[0] is not None:
#THIS is never reached and that's the problem
prediction = logits_to_text(K.eval(y_pred), tokenizer)
#test if prediction complies to grammar rules
compileable = compiles(prediction) ^ 1
compile_error = compileable * punishment_rate
loss = K.sparse_categorical_crossentropy(y_true, y_pred, axis=-1) * (1 + compile_error)
return loss
return compile_loss
Is there any workaround for evaluating a tensor only when it was filled with a batch? Or alternatively, change the loss function after compilation of the model via a callback without it having to recompile the model?
As per keras source, you can use a Loss Function Wrapper to create a Custom Loss Function class and then pass it to your model seamlessly.
As an example:
#Import the wrapper
from keras.losses import LossFunctionWrapper
#Create your class extending the wrapper
class MyLossFunction(LossFunctionWrapper):
#Implement the constructor - here you can give extended arguments to it.
def __init__(self,
tokenizer,
punishment_rate,
reduction=losses_utils.Reduction.SUM_OVER_BATCH_SIZE,
name='my_custom_text_function'):
super(MyLossFunction, self).__init__(
my_function,
name=name,
reduction=reduction,
tokenizer = tokenizer,
punishment_rate= punishment_rate)
#Now you have to define your function "my_function":
#Please, notice that ALL loss functions that follow keras model needs two arguments:
#y_true (correct result) and y_pred (the result obtained in the network).
def my_function(y_true, y_pred, tokenizer, punishment_rate):
shape = K.int_shape(y_pred)
if shape[0] is not None:
prediction = logits_to_text(K.eval(y_pred), tokenizer)
#test if prediction complies to grammar rules
compileable = compiles(prediction) ^ 1
compile_error = compileable * punishment_rate
return K.sparse_categorical_crossentropy(y_true, y_pred, axis=-1) * (1 + compile_error)
return K.sparse_categorical_crossentropy(y_true, y_pred)
You can then instantiate it and use in your compiler:
custom_loss= MyLossFunction(tokenizer = ..., punishment_rate = ...)
classifier.compile(optimizer=optimizer,
loss=custom_loss,
metrics= ['binary_accuracy'])

How to regularize a layer's kernel weights bias weights in a single regularization function?

The Keras documentation introduces separate classes for weight regularization and bias regularization. These can be subclasses to add a custom regularizer. An example from the Keras docs:
def my_regularizer(x):
return 1e-3 * tf.reduce_sum(tf.square(x))
where x can be either the kernel weights or the bias weights. I however want to regularize my layer with a function that include both the layer weights and the layer bias. Is there a way that incorporates both of these into a single function?
For example I would like to have as regularizer:
def l1_special_reg(weight_matrix, bias_vector):
return 0.01 * K.sum(K.abs(weight_matrix)-K.abs(bias_vector))
Thanks,
You can call layer[idx].trainable_weights, it will return both weights and bias. After that you can manually add that regularization loss in model loss function as follows:
model.layers[-1].trainable_weights
[<tf.Variable 'dense_2/kernel:0' shape=(100, 10) dtype=float32_ref>,
<tf.Variable 'dense_2/bias:0' shape=(10,) dtype=float32_ref>]
Complete example with loss function:
# define model
def l1_reg(weight_matrix):
return 0.01 * K.sum(K.abs(weight_matrix))
wts = model.layers[-1].trainable_weights # -1 for last dense layer.
reg_loss = l1_reg(wts[0]) + l1_reg(wts[1])
def custom_loss(reg_loss):
def orig_loss(y_true, y_pred):
return K.categorical_crossentropy(y_true, y_pred) + reg_loss
return orig_loss
model.compile(loss=custom_loss(reg_loss),
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
In TensorFlow 2 this can be achieved with the model.add_loss() function. Say you have a weights and a bias tensor of some layer:
w, b = layer.trainable_weights()
Then you can regularize this layer by adding the regularization function a loss term to the model object as follows:
def l1_special_reg(weight_matrix, bias_vector):
return 0.01 * K.sum(K.abs(weight_matrix)-K.abs(bias_vector))
model.add_loss(l1_special_reg(w, b))
Naturally, you can do this for each layer independently.

Resources