How do I get the sample loss while training instead of the total loss? The loss history is available which gives the total batch loss but it doesn't provide the loss for individual samples.
If possible I would like to have something like this:
on_batch_end(batch, logs, **sample_losses**)
Is something like this available and if not can you provide some hints how to change the code to support this?
To the best of my knowledge it is not possible to get this information via callbacks since the loss is already computed once the callbacks are called (have a look at keras/engine/training.py). To simply inspect the losses you may override the loss function, e.g.:
def myloss(ytrue, ypred):
x = keras.objectives.mean_squared_error(ytrue, ypred)
return theano.printing.Print('loss for each sample')(x)
model.compile(loss=myloss)
Actually this can be done using a callback. This is now included in the keras documentation on callbacks. Define your own callback like this
class LossHistory(keras.callbacks.Callback):
def on_train_begin(self, logs={}):
self.losses = []
def on_batch_end(self, batch, logs={}):
self.losses.append(logs.get('loss'))
And then pass in this callback to your model. You should get per batch losses appended to the history ojbect.
I have also not found any existing functions in the Keras API that can return individual sample losses, while still computing on a minibatch. It seems you have to hack keras, or maybe access the tensorflow graph directly.
set batch size to 1 and use callbacks in model.evaluate OR manually calculate the loss between prediction (model.predict) and ground truth.
Related
When using a 64 size batch, I need to fine-operate the loss value with respect to every single data point.
I know I can use reduction='none' when creating a loss function object then I can get a fine granularity loss value. But it's better to be a regular loss object without setting reduction='none', to keep consistency with other code.
It there any way to operate finer loss value without reduction='none'?
Why don't you wrap the function with your predefined options?
def custom_loss(*args, **kwargs):
return some_builtin(*args, **kwargs, reduction='none')
Where some_builtin would be a builtin PyTorch loss, e.g. torch.functional.l1_loss, torch.functional.mse_loss, ...
How can one effectively stop the fit process of a training model via callback in keras? Thus far I have tried various approaches including the one below.
class EarlyStoppingCallback(tf.keras.callbacks.Callback):
def __init__(self, threshold):
super(EarlyStoppingCallback, self).__init__()
self.threshold = threshold
def on_epoch_end(self, epoch, logs=None):
accuracy = logs["accuracy"]
if accuracy >= self.threshold:
print("Stopping early!")
self.model.stop_training = True
The callback is executed, however the self.model.stop_training = True does not seem to have an effect. The print succeeds, but the model continues training. Any idea how to resolve this issue?
My tensorflow version is: tensorflow==1.14.0
You're probably affected by the following issue: https://github.com/tensorflow/tensorflow/issues/37587.
In short - whenever model.predict or model.evaluate are called, model.stop_training is reset to False. I was able to reproduce this behavior using your EarlyStoppingCallback followed by another callback which was calling model.predict on some fixed dataset.
The workaround is to put callbacks which are calling model.predict or model.evaluate first before any callbacks which might want to set model.stop_training to True. It also looks like the issue was fixed in TF 2.2.
In Keras I have the following
MyModel.fit_generator(generator=generatorTraining, epochs=self.nofEpochs,
steps_per_epoch=nofBatchesPerTrainingEpoch, callbacks=callbacks,
validation_data=generatorValidation, validation_steps=nofBatchesPerValidationEpoch)
I then add a custom callback to callbacks, to record information about the epoch, training and batches. This I do using the following functions: on_epoch_begin, on_epoch_end, on_train_begin, on_train_end, on_batch_begin, and on_batch_end. I can find other callbacks that I can use in MyModel.evaluate.
But I cannot seem to find a way to get information from the validation_data in a callback, e.g., accuracy. Is this simply not possible or?
You can use sample code like,
def on_epoch_end_validation(self, epoch):
x_test = self.validation_data[0]
and then can use on_epoch_end_validation in your sample code
I use custom loss function in keras. Now, I want to use sample weights in Keras.
I've searched in google and some article suggest model.fit(X,y,sample_weight= custom_weights)
But I want to use sample weight directly in custom loss function. My custom loss function quite complex and for some reason i need to process sample weight directly.
for example:
custom_weights = np.array([1,2,3,4,5,6,7,8,9,10])
#my failed attempt
def custom_loss_function(y_true, y_pred , custom_weights):
return K.mean(K.abs(y_pred - y_true) * custom_weights), axis=-1)
note: my real custom_loss_function is very complex. In this question, I use "MAE" as example to simplify the problem so we can focus to answer "how to use sample weights in custom_loss_function "
how to do this task correctly ?
I am playing around with Keras and try to predict a word from within a context e.g. from a sentence "I have to say the food was tasty!" I hope to get something like this:
[say the ? was tasty] -> food, meals, spaghetti, drinks
However, my problem currently is that the network I am training appears to learn just the probabilities of the single words, and not the probabilities they have in a particular context.
Since the frequency of words is not balanced I thought I might/could/should apply weights to my loss function - which is currently the binary-cross entropy function.
I simply multiply the converse probability of each word with the error:
def weighted_binary_crossentropy(y_true, y_pred):
return K.mean(K.binary_crossentropy(y_pred, y_true) * (1-word_weights), axis=1)
This function is being used by the model as loss function:
model.compile(optimizer='adam', loss=weighted_binary_crossentropy)
However, my results are the exact same and I am not sure if just my model is broken or if I am using the loss paramter/function wrong.
is my weighted_binary_crossentropy() function doing what I just described? I asked because for some reason this works similar:
word_weights), axis=1)
Actually, as one may read in a documentation of a fit function, one may provide sample_weights which seem to be exactly what you want use.