In Keras I have the following
MyModel.fit_generator(generator=generatorTraining, epochs=self.nofEpochs,
steps_per_epoch=nofBatchesPerTrainingEpoch, callbacks=callbacks,
validation_data=generatorValidation, validation_steps=nofBatchesPerValidationEpoch)
I then add a custom callback to callbacks, to record information about the epoch, training and batches. This I do using the following functions: on_epoch_begin, on_epoch_end, on_train_begin, on_train_end, on_batch_begin, and on_batch_end. I can find other callbacks that I can use in MyModel.evaluate.
But I cannot seem to find a way to get information from the validation_data in a callback, e.g., accuracy. Is this simply not possible or?
You can use sample code like,
def on_epoch_end_validation(self, epoch):
x_test = self.validation_data[0]
and then can use on_epoch_end_validation in your sample code
Related
I have an unbalanced dataset. Therefore, I use class_weight for my model. At the same time, I want to weight newer data more than older data. Therefore, I use sample_weight, too.
I wanted to specify weighted_metrics in the compile function. But the Keras documentation says the following about this argument:
List of metrics to be evaluated and weighted by sample_weight or
class_weight during training and testing.
I use both parameter, but the documentation says sample_weight or class_weight. Which one is used now? I could not find out looking at the Keras github repository (training.py). Or are both used in my case?
I can't obtain history from model.fit_generator()
net_history=densenet_model.fit_generator(
generator=train_loader,
steps_per_epoch=len(train_dataset),
max_queue_size=500,
workers=1,
validation_data=val_loader,
epochs=10,
validation_steps=len(val_dataset),
verbose=0,
callbacks=[checkpointer, tensorboard]
)
print(net_history.history)
print(net_history.history.keys())
result:
{}
dict_keys([])
But I can obtain history information from model.fit()
net_history2=densenet_model.fit(
x_train,
y_train,
validation_split=0.3,
epochs=4,
verbose=0,
batch_size=4,
callbacks=[checkpointer, tensorboard]
)
print(net_history2.history)
result:
{'val_loss': [1.0166720993050904, 0.8762002832421633, 0.9079110455290179, 0.8696109705439238],
'val_categorical_crossentropy': [0.8133353590965271, 0.677170991897583, 0.7131518721580505,
0.6792631149291992], 'val_categorical_accuracy': [0.5887850522994995, 0.6355140209197998,
0.5794392228126526, 0.6074766516685486], 'loss': [0.8654926233446067, 0.8416075144219495,
0.8553338176325748, 0.8491003025881192], 'categorical_crossentropy': [0.6599225, 0.6403823,
0.6583931, 0.6564969], 'categorical_accuracy': [0.6396761, 0.659919, 0.5991903, 0.562753]}
Why does this happen?
How can I get the loss and accuracy information from model.fit_generator？
ps: I failed to visualize iterative process through log event because each time I try to open it in tensorboard, the browser gets stuck in 'namespace hierarchy finding similar subgraphs' and then crashes... QAQ
You can stick with model.fit becuase it supports generators.
For TensorBoard you may want to wait a little bit longer (I thought that tensorboard stucked in 'namespace hierarchy finding similar subgraphs' but it was running and it took some time).
Note: for training and validation steps you should get the length of the generators which is equal to trainORval_dataset_size / batch size, not the length of the dataset. You don't want the same images to be repeated in the same epoch.
steps_per_epoch=len(train_loader)
validation_steps=len(val_loader)
As for why exactly this is happening I can only speculate that it is because model.fit_generator is deprecated, in TensorFlow 2.2 and higher you can just use model.fit because this now supports generators.
Source:
https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit_generator
I use custom loss function in keras. Now, I want to use sample weights in Keras.
I've searched in google and some article suggest model.fit(X,y,sample_weight= custom_weights)
But I want to use sample weight directly in custom loss function. My custom loss function quite complex and for some reason i need to process sample weight directly.
for example:
custom_weights = np.array([1,2,3,4,5,6,7,8,9,10])
#my failed attempt
def custom_loss_function(y_true, y_pred , custom_weights):
return K.mean(K.abs(y_pred - y_true) * custom_weights), axis=-1)
note: my real custom_loss_function is very complex. In this question, I use "MAE" as example to simplify the problem so we can focus to answer "how to use sample weights in custom_loss_function "
how to do this task correctly ?
Why does this code work fine for the loss function but the metrics fail after one iteration with "ValueError: operands could not be broadcast together with shapes (32,) (24,) (32,)"?
If I use "categorical_crossentropy" in quotes then it works. And my custom metric looks identical to the one in keras.losses.
import keras.backend as K
def categorical_crossentropy(y_true, y_pred):
return K.categorical_crossentropy(y_pred, y_true)
fc.compile(optimizer=Adam(.01), loss=categorical_crossentropy, metrics=[categorical_crossentropy])
fc.fit(xtrain, ytrain, validation_data=(xvalid, yvalid), verbose=0,
callbacks=[TQDMNotebookCallback(leave_inner=True, leave_outer=True)],
nb_epoch=2)
It works if I import categorical_crossentropy from keras.metrics; rather than importing K. Still no idea why the above doesn't work but at least this is a solution.
Also it looks like the loss function is not necessary in metrics parameter anyway as it is automatically calculated and shown for training and validation.
How do I get the sample loss while training instead of the total loss? The loss history is available which gives the total batch loss but it doesn't provide the loss for individual samples.
If possible I would like to have something like this:
on_batch_end(batch, logs, **sample_losses**)
Is something like this available and if not can you provide some hints how to change the code to support this?
To the best of my knowledge it is not possible to get this information via callbacks since the loss is already computed once the callbacks are called (have a look at keras/engine/training.py). To simply inspect the losses you may override the loss function, e.g.:
def myloss(ytrue, ypred):
x = keras.objectives.mean_squared_error(ytrue, ypred)
return theano.printing.Print('loss for each sample')(x)
model.compile(loss=myloss)
Actually this can be done using a callback. This is now included in the keras documentation on callbacks. Define your own callback like this
class LossHistory(keras.callbacks.Callback):
def on_train_begin(self, logs={}):
self.losses = []
def on_batch_end(self, batch, logs={}):
self.losses.append(logs.get('loss'))
And then pass in this callback to your model. You should get per batch losses appended to the history ojbect.
I have also not found any existing functions in the Keras API that can return individual sample losses, while still computing on a minibatch. It seems you have to hack keras, or maybe access the tensorflow graph directly.
set batch size to 1 and use callbacks in model.evaluate OR manually calculate the loss between prediction (model.predict) and ground truth.