def myloss(y_true, y_pred):
b = k.constant([1, 1, 1, 50, 50, 50], shape=[6, 1])
return (k.mean(k.sqrt(k.dot(k.square(y_pred - y_true), b)))
This is our loss function and we got this result.
2800/2799 [==============================] - 245s - loss: 204.2003 - soft_acc: 0.5136 - val_loss: 64.3844 - val_soft_acc: 0.4648
We tried changing the learning rates and optimiser but the loss didn't improve
we refered to this link
Keras Extremely High Loss
epoch 1/200 ===========================] - 254s - loss: 4.0631 - rmse: 5.1670 - val_loss: 4.6882 - val_rmse: 4.7807
and added logarithmic error and got the above loss value. How to reduce the loss further?
I tried normalising the data. It worked for me.
Related
I am training a LSTM model on my current dataset to predict the multiclass categories - there are 18 mutually exclusive categories and the dataset has ~ 500 rows only (a really small dataset). I am handling the class imbalance using the following:
from sklearn.utils import class_weight
class_weights = list(class_weight.compute_class_weight('balanced',
classes = np.unique(df['categories']),
y = df['categories']))
weights = {}
for index, weight in enumerate(class_weights):
weights[index] = weight
Post this I am building my LSTM model and have been evaluating this model using PRC in tf.metrics as this is an imbalanced target classification problem
METRICS = [ tf.metrics.AUC(name='prc', curve='PR'), # precision-recall curve]
model = Sequential()
model.add(Embedding(MAX_NB_WORDS, EMBEDDING_DIM, input_length=X.shape[1]))
model.add(SpatialDropout1D(0.2))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(18, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=METRICS)
print(model.summary())
and finally:
history = model.fit(X_train,
y_train,
batch_size=10,
epochs=10,
verbose=1,
class_weight=weights,
validation_data=(X_test,y_test))
Now when I look at the results, the training prc is coming out to be really high whereas my val_prc is really low. An example with 10 epochs:
Epoch 1/10
30/30 [==============================] - 5s 174ms/step - loss: 2.9951 - prc: 0.0682 -
val_loss: 2.8865 - val_prc: 0.0639
Epoch 2/10
30/30 [==============================] - 5s 169ms/step - loss: 2.9556 - prc: 0.0993 -
val_loss: 2.8901 - val_prc: 0.0523
.....
Epoch 8/10
30/30 [==============================] - 6s 189ms/step - loss: 1.2494 - prc: 0.6415 -
val_loss: 3.0662 - val_prc: 0.0728
Epoch 9/10
30/30 [==============================] - 6s 210ms/step - loss: 0.9237 - prc: 0.8302 -
val_loss: 3.0624 - val_prc: 0.1006
Epoch 10/10
30/30 [==============================] - 6s 184ms/step - loss: 0.7452 - prc: 0.9017 -
val_loss: 3.5035 - val_prc: 0.0821
My questions are:
Is the evaluation metric correct that I am using considering it is an imbalanced class problem?
Am I treating the imbalance correctly with the code that I have written in the first place and most importantly, am I using this correct in the model.fit() ?
How can I resolve this? Is there any alternative approach that you can suggest?
I'm running a multiclass classification problem using the below resnet model:
resnet = tf.keras.applications.ResNet50(
include_top=False ,
weights='imagenet' ,
input_shape=(96, 96, 3) ,
pooling="avg"
)
for layer in resnet.layers:
layer.trainable = True
model_resnet = tf.keras.Sequential()
model_resnet.add(resnet)
model_resnet.add(tf.keras.layers.Flatten())
model_resnet.add(tf.keras.layers.Dense(8, activation='softmax',name='output') )
model_resnet.compile( loss="sparse_categorical_crossentropy" , optimizer=tf.keras.optimizers.Adam(learning_rate=0.001) ,metrics=['accuracy'])
I also used a train and a test generator as below:
train_generator=img_gen.flow_from_dataframe(dataframe=train_dataset,x_col="file_loc",y_col='expr',target_size=(96, 96),batch_size=91,class_mode="raw")
test_generator=img_gen.flow_from_dataframe(dataframe=test_dataset,x_col="file_loc",target_size=(96, 96),batch_size=93,y_col=None,shuffle=False,class_mode=None)
when I am running the code below I get the wanted results and everything works fine
model_resnet.fit_generator(train_generator,
steps_per_epoch=STEP_SIZE_TRAIN_resnet,
epochs=20
)
I wanted to compute the validation accuracy of every epoch so I wrote something like this
model_path = f"/content/weights" + "{val_accuracy:.4f}.hdf5"
checkpoint = tf.keras.callbacks.ModelCheckpoint(
model_path,
monitor='val_accuracy',
save_best_only=True,
mode='max',
verbose=1
)
history = model_resnet.fit_generator(
train_generator,
epochs=5,
steps_per_epoch=STEP_SIZE_TRAIN_resnet,
validation_data=test_generator,
validation_steps=STEP_SIZE_TEST_resnet,
max_queue_size=1,
shuffle=True,
callbacks=[checkpoint],
verbose=1
)
The problem is that for every epoch the validation loss and validation accuracy remain zero even though the training loss and accuracy change. I ran this code for over 20 epochs and it doesn't change at all. I can't find what am I doing wrong since without this it works perfectly,does anyone have any idea?
Epoch 1: val_accuracy improved from -inf to 0.00000, saving model to /content/weights0.0000.hdf5
500/500 [==============================] - 30s 60ms/step - loss: 1.0213 - accuracy: 0.6546 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 2/5
500/500 [==============================] - ETA: 0s - loss: 0.9644 - accuracy: 0.6672
Epoch 2: val_accuracy did not improve from 0.00000
500/500 [==============================] - 29s 58ms/step - loss: 0.9644 - accuracy: 0.6672 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Edit: I didn't specify the test labels of the test dataset because I used to compute the accuracy score as below:
y_pred = model_resnet.predict(test_generator)
y_pred_max = np.argmax(y_pred, axis=1)
y_true = test_dataset["expr"].to_numpy()
print("accuracy",accuracy_score(y_true, y_pred_max))
I changed the test_generator as below:
test_generator=img_gen.flow_from_dataframe(dataframe=test_dataset,x_col="file_loc",target_size=(96, 96),batch_size=93,y_col='expr',shuffle=False,class_mode=None)
but nothing has changed, it still results in zero
As #Dr.Snoopy said, the problems were that I didn't specify the test labels in these generator (which are required to compute accuracy) and I had different class modes in the generator,the correct was "raw" in both.
I need to compute a weighted F1-score in such a way to penalize more errors over my least popular label (typical binary classification problem with an unbalanced dataset).
Unfortunately, I don't get a valid F1-score.
The followings are my metrics functions:
def sensitivity(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
return true_positives / (possible_positives + K.epsilon())
def specificity(y_true, y_pred):
true_negatives = K.sum(K.round(K.clip((1-y_true) * (1-y_pred), 0, 1)))
possible_negatives = K.sum(K.round(K.clip(1-y_true, 0, 1)))
return true_negatives / (possible_negatives + K.epsilon())
def f1(y_true, y_pred):
def recall(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
def precision(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
return precision
precision = precision(y_true, y_pred)
recall = recall(y_true, y_pred)
return 2*((precision*recall)/(precision+recall))
model.compile(loss='binary_crossentropy',
optimizer=RMSprop(0.001),
metrics=[sensitivity, specificity, 'accuracy', f1])
and here I train the model and do evaluation:
model.fit(x_train, y_train, epochs=12, batch_size=32, verbose=1, class_weight=class_weights_dict, validation_split=0.3)
classes = model.predict(x_test)
loss_and_metrics = model.evaluate(x_test, y_test, batch_size=128, verbose=1)
I always get nan as f1score - is something wrong conceptually or programmatically? Because data are the same I used with another classifier of the scikit-learn library (SVM) and it succeeded.
These are results:
Epoch 1/12
5133/5133 [==============================] - 5s 976us/step - loss: 0.6955 - sensitivity: 0.0561 - specificity: 0.9377 - acc: 0.8712 - f1: nan - val_loss: 0.6884 - val_sensitivity: 0.8836 - val_specificity: 0.0000e+00 - val_acc: 0.0723 - val_f1: nan
Epoch 2/12
5133/5133 [==============================] - 5s 894us/step - loss: 0.6954 - sensitivity: 0.3865 - specificity: 0.5548 - acc: 0.5398 - f1: nan - val_loss: 0.6884 - val_sensitivity: 0.0000e+00 - val_specificity: 1.0000 - val_acc: 0.9277 - val_f1: nan
Epoch 3/12
5133/5133 [==============================] - 5s 925us/step - loss: 0.6953 - sensitivity: 0.3928 - specificity: 0.5823 - acc: 0.5696 - f1: nan - val_loss: 0.6884 - val_sensitivity: 0.0000e+00 - val_specificity: 1.0000 - val_acc: 0.9277 - val_f1: nan
Epoch 4/12
5133/5133 [==============================] - 5s 935us/step - loss: 0.6954 - sensitivity: 0.1309 - specificity: 0.8504 - acc: 0.7976 - f1: nan - val_loss: 0.6884 - val_sensitivity: 0.0000e+00 - val_specificity: 1.0000 - val_acc: 0.9277 - val_f1: nan
etc.
Final result:
[0.6859536773606656, 0.0, 1.0, 0.9321705426356589, nan]
Regarding the nan in your f1 metric:
If you look at the log, your validation sensitivity is 0. Which means your precision and recall are both zero as well. So in the f1 calculation you are dividing by zero and getting a nan.
Add K.epsilon(), as you have done in the other functions.
On a side note, judging by your loss, which had a negligible improvement on the train set, your network had learnt nothing. I'd advice you to start by increasing the number of epochs, make the network deeper and don't pass anything to the class_weight argument (you mention not using weighted computation yet, but your code does set some class weight).
Check also if one of the batches has f1_score equals to nan.
I'm currently trying to use a pre-trained network and test in on this dataset.
Originally, I used VGG19 and just fine-tuned only the classifier at the end to fit with my 120 classes. I let all layers trainable to maybe improve performance by having a deeper training. The problem is that the model is very slow (even if I let it run for a night, I only got couple of epochs and reach an accuracy of around 45% - I have a GPU GTX 1070).
Then, my thinking was to freeze all layers from this model as I have only 10k images and only train the few last Denses layers but it's still not realy fast.
After watching this video (at around 2 min 30s), I decided to replicate the principle of Transfer-Values with InceptionResnetv2.
I processed every pictures and saved the output in a numpy matrix with the following code.
# Loading pre-trained Model + freeze layers
model = applications.inception_resnet_v2.InceptionResNetV2(
include_top=False,
weights='imagenet',
pooling='avg')
for layer in model.layers:
layer.trainable = False
# Extraction of features and saving
a = True
for filename in glob.glob('train/resized/*.jpg'):
name_img = os.path.basename(filename)[:-4]
class_ = label[label["id"] == name_img]["breed"].values[0]
input_img = np.expand_dims(np.array(Image.open(filename)), 0)
pred = model.predict(input_img)
if a:
X = np.array(pred)
y = np.array(class_)
a = False
else:
X = np.vstack((X, np.array(pred)))
y = np.vstack((y, class_))
np.savez_compressed('preprocessed.npz', X=X, y=y)
X is a matrix of shape (10222, 1536) and y is (10222, 1).
After, I designed my classifier (several topologies) and I have no idea why it is not able to perform any learning.
# Just to One-Hot-Encode labels properly to (10222, 120)
label_binarizer = sklearn.preprocessing.LabelBinarizer()
y = label_binarizer.fit_transform(y)
model = Sequential()
model.add(Dense(512, input_dim=X.shape[1]))
# model.add(Dense(2048, activation="relu"))
# model.add(Dropout(0.5))
# model.add(Dense(256))
model.add(Dense(120, activation='softmax'))
model.compile(
loss = "categorical_crossentropy",
optimizer = "Nadam", # I tried several ones
metrics=["accuracy"]
)
model.fit(X, y, epochs=100, batch_size=64,
callbacks=[early_stop], verbose=1,
shuffle=True, validation_split=0.10)
Below you can find the output from the model :
Train on 9199 samples, validate on 1023 samples
Epoch 1/100
9199/9199 [==============================] - 2s 185us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 2/100
9199/9199 [==============================] - 1s 100us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 3/100
9199/9199 [==============================] - 1s 98us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 4/100
9199/9199 [==============================] - 1s 96us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 5/100
9199/9199 [==============================] - 1s 99us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
Epoch 6/100
9199/9199 [==============================] - 1s 96us/step - loss: 15.9639 - acc: 0.0096 - val_loss: 15.8975 - val_acc: 0.0137
I tried to change topologies, activation functions, add dropouts but nothing creates any improvements.
I have no idea what is wrong in my way of doing this. Is the X matrix incorrect ? Isn't it allowed to use the pre-trained model only as feature extractor then perform the classification with a second model ?
Many thanks for your feedbacks,
Regards,
Nicolas
You'll need to call preprocess_input before feeding the image array to the model. It normalizes the values of input_img from [0, 255] into [-1, 1], which is the desired input range for InceptionResNetV2.
input_img = np.expand_dims(np.array(Image.open(filename)), 0)
input_img = applications.inception_resnet_v2.preprocess_input(input_img.astype('float32'))
pred = model.predict(input_img)
I've just started using Keras. The sample I'm working on has a model and the following snippet is used to run the model
from sklearn.preprocessing import LabelBinarizer
label_binarizer = LabelBinarizer()
y_one_hot = label_binarizer.fit_transform(y_train)
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, nb_epoch=3, validation_split=0.2)
I get the following response:
Using TensorFlow backend. Train on 80 samples, validate on 20 samples Epoch 1/3
32/80 [===========>..................] - ETA: 0s - loss: 1.5831 - acc:
0.4062 80/80 [==============================] - 0s - loss: 1.3927 - acc:
0.4500 - val_loss: 0.7802 - val_acc: 0.8500 Epoch 2/3
32/80 [===========>..................] - ETA: 0s - loss: 0.9300 - acc:
0.7500 80/80 [==============================] - 0s - loss: 0.8490 - acc:
0.8000 - val_loss: 0.5772 - val_acc: 0.8500 Epoch 3/3
32/80 [===========>..................] - ETA: 0s - loss: 0.6397 - acc:
0.8750 64/80 [=======================>......] - ETA: 0s - loss: 0.6867 - acc:
0.7969 80/80 [==============================] - 0s - loss: 0.6638 - acc:
0.8000 - val_loss: 0.4294 - val_acc: 0.8500
The documentation says that fit returns
A History instance. Its history attribute contains all information
collected during training.
Does anyone know how to interpret the history instance?
For example, what does 32/80 mean? I assume 80 is the number of samples but what is 32? ETA: 0s ??
ETA = Estimated Time of Arrival.
80 is the size of your training set, 32/80 and 64/80 mean that your batch size is 32 and currently the first batch (or the second batch respectively) is being processed.
loss and acc refer to the current loss and accuracy of the training set.
At the end of each epoch your trained NN is evaluated against your validation set. This is what val_loss and val_acc refer to.
The history object returned by model.fit() is a simple class with some fields, e.g. a reference to the model, a params dict and, most importantly, a history dict. It stores the values of loss and acc (or any other used metric) at the end of each epoch. For 2 epochs it will look like this:
{
'val_loss': [16.11809539794922, 14.12947562917035],
'val_acc': [0.0, 0.0],
'loss': [14.890108108520508, 12.088571548461914],
'acc': [0.0, 0.25]
}
This comes in very handy if you want to visualize your training progress.
Note: if your validation loss/accuracy starts increasing while your training loss/accuracy is still decreasing, this is an indicator of overfitting.
Note 2: at the very end you should test your NN against some test set that is different from you training set and validation set and thus has never been touched during the training process.
32 is your batch size. 32 is the default value that you can change in your fit function if you wish to do so.
After the first batch is trained Keras estimates the training duration (ETA: estimated time of arrival) of one epoch which is equivalent to one round of training with all your samples.
In addition to that you get the losses (the difference between prediction and true labels) and your metric (in your case the accuracy) for both the training and the validation samples.