Can keras progress bar show instantaneous metrics rather than running average? - keras

Question

By "progress bar" I mean the standard progress bar that shows up with tf.keras.Model.fit
As I understand, it shows a running average of your selected metrics (over the current epoch), but I want it to show the value at the last completed iteration.
Is there a built-in way to make this change? And if not, what would be the easiest way?

Answer 1

I made a callback a while ago to solve this problem.
class print_on_end(Callback):
def on_batch_end(self, batch, logs={}):
print()
You want to call it like this.
model.fit(training_dataset, steps_per_epoch=num_training_samples, epochs=EPOCHS,validation_data=validation_dataset, callbacks=[print_on_end()])
But this callback prints the everages, just on different lines, so i don't thinks its what you want.
This instead:
class LossAndErrorPrintingCallback(keras.callbacks.Callback):
def on_train_batch_end(self, batch, logs=None):
print("For batch {}, loss is {:7.2f}.".format(batch, logs["loss"]))
)
)
This callback prints the loss of every batch, so it should be what you are looking for.
( if you need the metric just change logs["loss"] with logs["name of the metric"] ex. logs["mean_absolute_error"]
EDIT:
to check the name of the metric inside logs you could print the keys of the log and find the ona you are searching.
class PrintKeys(keras.callbacks.Callback):
def on_train_batch_end(self, batch, logs=None):
keys = list(logs.keys())
print(keys)
)
)
In that method you should only find the keys for the loss and the metric.
source:
https://keras.io/guides/writing_your_own_callbacks/

Answer 2

An example of the class hierarchy of the MeanSquaredError metric is
MeanSquaredError->MeanMetricWrapper->Mean->Reduce->Metric
Is there a built-in way ?
The main problem is that all metrics are subclasses of the Reduce metric, which performs the aggregation, and there is nothing foreseen to change the behaviour of the Reduce base class.
How to achieve this the most easily
Given the schema above you can achieve what you want by means of creating a new metric subclass of the MeanMetricWrapper
which overrides the update_state method of the MeanMetricWrapper
by means of first calling self.reset_state and then MeanMetricWrapper.update_state.
Like this, the aggregates in the underlying Reduce base class will aggregate only a single value. Working Example below:
#! /usr/bin/env python
import numpy as np
import keras
from keras.metrics import MeanMetricWrapper
x=np.linspace(0, 1, 20000)[:,np.newaxis,np.newaxis]
y=np.sin(x*2*np.pi)
model = keras.Sequential()
model.add(keras.layers.Dense(4, activation="tanh", input_shape=(1,1)))
model.add(keras.layers.Dense(4, activation="tanh"))
model.add(keras.layers.Dense(4))
#####
# Here the Instantaneous metric variant
class InstMetric(MeanMetricWrapper):
def __init__(self, fn, **kwargs):
""" fn is the callable loss function you want to use in your metric """
super().__init__(fn=fn, **kwargs)
def update_state(self, y_true, y_pred, sample_weight=None):
self.reset_states()
return super().update_state(y_true, y_pred, sample_weight=sample_weight)
#####
model.compile(optimizer='adam', loss='mean_squared_error',
metrics=[
keras.metrics.MeanSquaredError(name="MSE"),
InstMetric(keras.metrics.mean_squared_error, name="IMSE")
]
)
model.fit(x=x, y=y, epochs=1, batch_size=5, steps_per_epoch=1000)
Storing this script as inst_demo.py and running it through tr to unfold the progress bar in the terminal
you get
$> ./inst_demo.py | tr \\r \\n
1/1000 [..............................] - ETA: 8:07 - loss: 0.4656 - MSE: 0.4656 - IMSE: 0.4656
42/1000 [>.............................] - ETA: 1s - loss: 0.4874 - MSE: 0.4874 - IMSE: 0.4133
87/1000 [=>............................] - ETA: 1s - loss: 0.4685 - MSE: 0.4685 - IMSE: 0.4764
132/1000 [==>...........................] - ETA: 1s - loss: 0.4627 - MSE: 0.4627 - IMSE: 0.5445
175/1000 [====>.........................] - ETA: 0s - loss: 0.4558 - MSE: 0.4558 - IMSE: 0.7689
217/1000 [=====>........................] - ETA: 0s - loss: 0.4443 - MSE: 0.4443 - IMSE: 0.1058
264/1000 [======>.......................] - ETA: 0s - loss: 0.4258 - MSE: 0.4258 - IMSE: 0.4162
311/1000 [========>.....................] - ETA: 0s - loss: 0.4090 - MSE: 0.4090 - IMSE: 0.1716
356/1000 [=========>....................] - ETA: 0s - loss: 0.3889 - MSE: 0.3889 - IMSE: 0.3417
400/1000 [===========>..................] - ETA: 0s - loss: 0.3707 - MSE: 0.3707 - IMSE: 0.1271
445/1000 [============>.................] - ETA: 0s - loss: 0.3532 - MSE: 0.3532 - IMSE: 0.0729
489/1000 [=============>................] - ETA: 0s - loss: 0.3383 - MSE: 0.3383 - IMSE: 0.2310
535/1000 [===============>..............] - ETA: 0s - loss: 0.3248 - MSE: 0.3248 - IMSE: 0.1228
580/1000 [================>.............] - ETA: 0s - loss: 0.3143 - MSE: 0.3143 - IMSE: 0.2670
625/1000 [=================>............] - ETA: 0s - loss: 0.3048 - MSE: 0.3048 - IMSE: 0.1762
671/1000 [===================>..........] - ETA: 0s - loss: 0.2962 - MSE: 0.2962 - IMSE: 0.0751
715/1000 [====================>.........] - ETA: 0s - loss: 0.2896 - MSE: 0.2896 - IMSE: 0.0650
756/1000 [=====================>........] - ETA: 0s - loss: 0.2831 - MSE: 0.2831 - IMSE: 0.2332
799/1000 [======================>.......] - ETA: 0s - loss: 0.2773 - MSE: 0.2773 - IMSE: 0.1026
841/1000 [========================>.....] - ETA: 0s - loss: 0.2721 - MSE: 0.2721 - IMSE: 0.1238
888/1000 [=========================>....] - ETA: 0s - loss: 0.2673 - MSE: 0.2673 - IMSE: 0.1471
936/1000 [===========================>..] - ETA: 0s - loss: 0.2631 - MSE: 0.2631 - IMSE: 0.2242
986/1000 [============================>.] - ETA: 0s - loss: 0.2580 - MSE: 0.2580 - IMSE: 0.2704
1000/1000 [==============================] - 2s 1ms/step - loss: 0.2574 - MSE: 0.2574 - IMSE: 0.2773
So you get an instant value each time the progressbar is updated.
You can also derive the InstMetric from any of the available keras metrics, if you don't want to be able to select the metric you will use.