I am using HDF5Matrix to load a dataset and train my model with it. In the first epoch I obtain about 10% of accuracy.
At the moment, my dataset is not very large, so I can copy the contents of the HDF5Matrix to a numpy array and train with it. I reinitialise the model, and this time, in the first epoch I obtain a 40% accuracy.
For more information about the HDF5Matrix, see this example.
I understand in the fit method, the parameters shuffle must be either False or 'batch'. I get the same behaviour either way.
Does anybody have the same problem? Could you tell me if there is something I am doing wrong?
This is a snippet of the code:
using HDF5Matrix
from keras.utils.io_utils import HDF5Matrix
x_train = HDF5Matrix('../data/default_data.h5', 'data')
y_train = HDF5Matrix('../data/default_data.h5', 'labels') # create the model ...
# train the model
model.fit(x_train, y_train, epochs=200, batch_size=2048, shuffle='batch') # which outputs:
Epoch 1/200
1758510/1758510 [==============================] - 42s - loss: 2.5574 - categorical_accuracy: 0.1032
Epoch 2/200
1758510/1758510 [==============================] - 41s - loss: 2.3145 - categorical_accuracy: 0.1553
Epoch 3/200
1758510/1758510 [==============================] - 41s - loss: 2.1931 - categorical_accuracy: 0.2067
Epoch 4/200
694272/1758510 [==========>...................] - ETA: 24s - loss: 2.1055 - categorical_accuracy: 0.2328
Using numpy array
# create the model again
...
# copy the HDF5Matrix to a numpy array
X_training = x_train[0:1758510]
Y_training = y_train[0:1758510]
# check X_training is equal to x_train
...
# train the model again
model.fit(X_training,
Y_training,
epochs=200,
batch_size=256,
shuffle=True)
# which outputs
Epoch 1/200
1758510/1758510 [==============================] - 27s - loss: 1.5019 - categorical_accuracy: 0.4710
Epoch 2/200
89600/1758510 [>.............................] - ETA: 26s - loss: 1.2786 - categorical_accuracy: 0.5523
Thank you very much
Related
I am training a LSTM model on my current dataset to predict the multiclass categories - there are 18 mutually exclusive categories and the dataset has ~ 500 rows only (a really small dataset). I am handling the class imbalance using the following:
from sklearn.utils import class_weight
class_weights = list(class_weight.compute_class_weight('balanced',
classes = np.unique(df['categories']),
y = df['categories']))
weights = {}
for index, weight in enumerate(class_weights):
weights[index] = weight
Post this I am building my LSTM model and have been evaluating this model using PRC in tf.metrics as this is an imbalanced target classification problem
METRICS = [ tf.metrics.AUC(name='prc', curve='PR'), # precision-recall curve]
model = Sequential()
model.add(Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=X.shape[1]))
model.add(SpatialDropout1D(0.2))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(18, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=METRICS)
print(model.summary())
and finally:
history = model.fit(X_train,
y_train,
batch_size=10,
epochs=10,
verbose=1,
class_weight=weights,
validation_data=(X_test,y_test))
Now when I look at the results, the training prc is coming out to be really high whereas my val_prc is really low. An example with 10 epochs:
Epoch 1/10
30/30 [==============================] - 5s 174ms/step - loss: 2.9951 - prc: 0.0682 -
val_loss: 2.8865 - val_prc: 0.0639
Epoch 2/10
30/30 [==============================] - 5s 169ms/step - loss: 2.9556 - prc: 0.0993 -
val_loss: 2.8901 - val_prc: 0.0523
.....
Epoch 8/10
30/30 [==============================] - 6s 189ms/step - loss: 1.2494 - prc: 0.6415 -
val_loss: 3.0662 - val_prc: 0.0728
Epoch 9/10
30/30 [==============================] - 6s 210ms/step - loss: 0.9237 - prc: 0.8302 -
val_loss: 3.0624 - val_prc: 0.1006
Epoch 10/10
30/30 [==============================] - 6s 184ms/step - loss: 0.7452 - prc: 0.9017 -
val_loss: 3.5035 - val_prc: 0.0821
My questions are:
Is the evaluation metric correct that I am using considering it is an imbalanced class problem?
Am I treating the imbalance correctly with the code that I have written in the first place and most importantly, am I using this correct in the model.fit() ?
How can I resolve this? Is there any alternative approach that you can suggest?
I understand there is a similar question "How to interpret Keras model.fit output?" , but my question is more specific, I am wondering how the loss and acc output of each batch inside one epoch are calculated?
is it calculated upon the validation set?
or on the trained samples so far in each epoch?(I think its this one )
or anything else?
below is a sample output during my training:
Epoch x/20:
...
54320/55200 [============================>.] - ETA: 0s - loss: 1.2083 - acc: 0.9554
54440/55200 [============================>.] - ETA: 0s - loss: 1.2083 - acc: 0.9554
54560/55200 [============================>.] - ETA: 0s - loss: 1.2083 - acc: 0.9555
...
my configuration:
model.fit(x_train, y_train,
batch_size=10,
epochs=20,
verbose=1,
validation_split=0.08)
Thank you!
The values are:
loss - training data, calculated every batch
acc - training data, calculated every batch
val_loss - test data, calculated every epoch
val_acc - test data, calculated every epoch
I'm currently trying to use a pre-trained network and test in on this dataset.
Originally, I used VGG19 and just fine-tuned only the classifier at the end to fit with my 120 classes. I let all layers trainable to maybe improve performance by having a deeper training. The problem is that the model is very slow (even if I let it run for a night, I only got couple of epochs and reach an accuracy of around 45% - I have a GPU GTX 1070).
Then, my thinking was to freeze all layers from this model as I have only 10k images and only train the few last Denses layers but it's still not realy fast.
After watching this video (at around 2 min 30s), I decided to replicate the principle of Transfer-Values with InceptionResnetv2.
I processed every pictures and saved the output in a numpy matrix with the following code.
# Loading pre-trained Model + freeze layers
model = applications.inception_resnet_v2.InceptionResNetV2(
include_top=False,
weights='imagenet',
pooling='avg')
for layer in model.layers:
layer.trainable = False
# Extraction of features and saving
a = True
for filename in glob.glob('train/resized/*.jpg'):
name_img = os.path.basename(filename)[:-4]
class_ = label[label["id"] == name_img]["breed"].values[0]
input_img = np.expand_dims(np.array(Image.open(filename)), 0)
pred = model.predict(input_img)
if a:
X = np.array(pred)
y = np.array(class_)
a = False
else:
X = np.vstack((X, np.array(pred)))
y = np.vstack((y, class_))
np.savez_compressed('preprocessed.npz', X=X, y=y)
X is a matrix of shape (10222, 1536) and y is (10222, 1).
After, I designed my classifier (several topologies) and I have no idea why it is not able to perform any learning.
# Just to One-Hot-Encode labels properly to (10222, 120)
label_binarizer = sklearn.preprocessing.LabelBinarizer()
y = label_binarizer.fit_transform(y)
model = Sequential()
model.add(Dense(512, input_dim=X.shape[1]))
# model.add(Dense(2048, activation="relu"))
# model.add(Dropout(0.5))
# model.add(Dense(256))
model.add(Dense(120, activation='softmax'))
model.compile(
loss = "categorical_crossentropy",
optimizer = "Nadam", # I tried several ones
metrics=["accuracy"]
)
model.fit(X, y, epochs=100, batch_size=64,
callbacks=[early_stop], verbose=1,
shuffle=True, validation_split=0.10)
Below you can find the output from the model :
Train on 9199 samples, validate on 1023 samples
Epoch 1/100
9199/9199 [==============================] - 2s 185us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 2/100
9199/9199 [==============================] - 1s 100us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 3/100
9199/9199 [==============================] - 1s 98us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 4/100
9199/9199 [==============================] - 1s 96us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 5/100
9199/9199 [==============================] - 1s 99us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 6/100
9199/9199 [==============================] - 1s 96us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
I tried to change topologies, activation functions, add dropouts but nothing creates any improvements.
I have no idea what is wrong in my way of doing this. Is the X matrix incorrect ? Isn't it allowed to use the pre-trained model only as feature extractor then perform the classification with a second model ?
Many thanks for your feedbacks,
Regards,
Nicolas
You'll need to call preprocess_input before feeding the image array to the model. It normalizes the values of input_img from [0, 255] into [-1, 1], which is the desired input range for InceptionResNetV2.
input_img = np.expand_dims(np.array(Image.open(filename)), 0)
input_img = applications.inception_resnet_v2.preprocess_input(input_img.astype('float32'))
pred = model.predict(input_img)
I am using Keras to build a neural network model:
model_keras = Sequential()
model_keras.add(Dense(4, input_dim=input_num, activation='relu',kernel_regularizer=regularizers.l2(0.01)))
model_keras.add(Dense(1, activation='linear',kernel_regularizer=regularizers.l2(0.01)))
sgd = optimizers.SGD(lr=0.01, clipnorm=0.5)
model_keras.compile(loss='mean_squared_error', optimizer=sgd)
model_keras.fit(X_norm_train, y_norm_train, batch_size=20, epochs=100)
The output looks like below. I am wondering if it is possible to out the loss, say every 10 epochs instead of every epoch? Thanks!
Epoch 1/200
20/20 [==============================] - 0s - loss: 0.2661
Epoch 2/200
20/20 [==============================] - 0s - loss: 0.2625
Epoch 3/200
20/20 [==============================] - 0s - loss: 0.2590
Epoch 4/200
20/20 [==============================] - 0s - loss: 0.2556
Epoch 5/200
20/20 [==============================] - 0s - loss: 0.2523
Epoch 6/200
20/20 [==============================] - 0s - loss: 0.2490
Epoch 7/200
20/20 [==============================] - 0s - loss: 0.2458
Epoch 8/200
20/20 [==============================] - 0s - loss: 0.2427
Epoch 9/200
20/20 [==============================] - 0s - loss: 0.2397
Epoch 10/200
20/20 [==============================] - 0s - loss: 0.2367
Epoch 11/200
20/20 [==============================] - 0s - loss: 0.2338
Epoch 12/200
20/20 [==============================] - 0s - loss: 0.2309
Epoch 13/200
20/20 [==============================] - 0s - loss: 0.2281
Epoch 14/200
20/20 [==============================] - 0s - loss: 0.2254
Epoch 15/200
20/20 [==============================] - 0s - loss: 0.2228
:
It is not possible to reduce frequency of logging to stdout, however, passing verbose=0 argument to fit() method would turn logging completely off.
Since the loop over epochs is not exposed in the Keras' sequential model, one way to collect scalar variable summaries with a custom frequency would be using Keras callbacks. In particular, you could use TensorBoard (assuming you are running with tensorflow backend) or CSVLogger (any backend) callbacks to collect any scalar variable summaries (training loss, in your case):
from keras.callbacks import TensorBoard
model_keras = Sequential()
model_keras.add(Dense(4, input_dim=input_num, activation='relu',kernel_regularizer=regularizers.l2(0.01)))
model_keras.add(Dense(1, activation='linear',kernel_regularizer=regularizers.l2(0.01)))
sgd = optimizers.SGD(lr=0.01, clipnorm=0.5)
model_keras.compile(loss='mean_squared_error', optimizer=sgd)
TB = TensorBoard(histogram_freq=10, batch_size=20)
model_keras.fit(X_norm_train, y_norm_train, batch_size=20, epochs=100, callbacks=[TB])
Setting histogram_freq=10 will save loss every 10 epochs.
EDIT: passing validation_data=(...) to the fit method will also allow to check validation level metrics.
Create a Keras callback to reduce the number of log lines. By default, Keras print log per every epoch. The following code prints only 10 log lines regardless the number of epochs.
class callback(tf.keras.callbacks.Callback):
def on_epoch_end(this,Epoch,Logs):
L = Logs["loss"];
if Epoch%Lafte==Lafte-1: #Log after a number of epochs
print(f"Average batch loss: {L:.9f}");
if Epoch==Epochs-1:
print(f"Fin-avg batch loss: {L:.9f}"); #Final average
Model = model();
Model.compile(...);
Dsize = ... #Number of samples in training data
Bsize = ... #Number of samples to process in 1 batch
Steps = 1000; #Number of batches to use to train
Epochs = round(Steps/(Dsize/Bsize));
Lafte = round(Epochs/10); #Log 10 times only, regardless of num of Epochs
if Lafte==0: Lafte=1; #Avoid modulus by zero in on_epoch_end
Model.fit(Data, epochs=Epochs, steps_per_epoch=round(Dsize/Bsize),
callbacks=[callback()], verbose=0);
I've just started using Keras. The sample I'm working on has a model and the following snippet is used to run the model
from sklearn.preprocessing import LabelBinarizer
label_binarizer = LabelBinarizer()
y_one_hot = label_binarizer.fit_transform(y_train)
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, nb_epoch=3, validation_split=0.2)
I get the following response:
Using TensorFlow backend. Train on 80 samples, validate on 20 samples Epoch 1/3
32/80 [===========>..................] - ETA: 0s - loss: 1.5831 - acc:
0.4062 80/80 [==============================] - 0s - loss: 1.3927 - acc:
0.4500 - val_loss: 0.7802 - val_acc: 0.8500 Epoch 2/3
32/80 [===========>..................] - ETA: 0s - loss: 0.9300 - acc:
0.7500 80/80 [==============================] - 0s - loss: 0.8490 - acc:
0.8000 - val_loss: 0.5772 - val_acc: 0.8500 Epoch 3/3
32/80 [===========>..................] - ETA: 0s - loss: 0.6397 - acc:
0.8750 64/80 [=======================>......] - ETA: 0s - loss: 0.6867 - acc:
0.7969 80/80 [==============================] - 0s - loss: 0.6638 - acc:
0.8000 - val_loss: 0.4294 - val_acc: 0.8500
The documentation says that fit returns
A History instance. Its history attribute contains all information
collected during training.
Does anyone know how to interpret the history instance?
For example, what does 32/80 mean? I assume 80 is the number of samples but what is 32? ETA: 0s ??
ETA = Estimated Time of Arrival.
80 is the size of your training set, 32/80 and 64/80 mean that your batch size is 32 and currently the first batch (or the second batch respectively) is being processed.
loss and acc refer to the current loss and accuracy of the training set.
At the end of each epoch your trained NN is evaluated against your validation set. This is what val_loss and val_acc refer to.
The history object returned by model.fit() is a simple class with some fields, e.g. a reference to the model, a params dict and, most importantly, a history dict. It stores the values of loss and acc (or any other used metric) at the end of each epoch. For 2 epochs it will look like this:
{
'val_loss': [16.11809539794922, 14.12947562917035],
'val_acc': [0.0, 0.0],
'loss': [14.890108108520508, 12.088571548461914],
'acc': [0.0, 0.25]
}
This comes in very handy if you want to visualize your training progress.
Note: if your validation loss/accuracy starts increasing while your training loss/accuracy is still decreasing, this is an indicator of overfitting.
Note 2: at the very end you should test your NN against some test set that is different from you training set and validation set and thus has never been touched during the training process.
32 is your batch size. 32 is the default value that you can change in your fit function if you wish to do so.
After the first batch is trained Keras estimates the training duration (ETA: estimated time of arrival) of one epoch which is equivalent to one round of training with all your samples.
In addition to that you get the losses (the difference between prediction and true labels) and your metric (in your case the accuracy) for both the training and the validation samples.