I have built a common Unet to train my own dataset by using Keras. I have set the EarlyStopping option as follows. However, during training, it keeps prompt out the precision value dis not change but in the next line, it is apparently changing. Has anyone meet this problem before or know how to solve the problem?
enter image description here
train_iterator = create_one_shot_iterator(train_files, batch_size=train_batch_size, num_epoch=epochs)
train_images, train_masks = train_iterator.get_next()
train_images, train_masks = augment_dataset(train_images, train_masks,
augment=True,
resize=True,
scale=1 / 255.,
hue_delta=0.1,
horizontal_flip=True,
width_shift_range=0.1,
height_shift_range=0.1,
rotate=15)
val_iterator = create_initializable_iterator(val_files, batch_size=val_batch_size)
val_images, val_masks = val_iterator.get_next()
val_images, val_masks = augment_dataset(val_images, val_masks,
augment=True,
resize=True,
scale=1 / 255.,
)
model_input = tf.keras.layers.Input(tensor=train_images)
model_output = Unet.u_net_256(model_input)
# Model definition
model = models.Model(inputs=model_input, outputs=model_output)
precision = tf.keras.metrics.Precision()
model.compile(optimizer='adam',
loss=bce_dice_loss,
metrics=[precision],
target_tensors=[train_masks])
model.summary()
cp = [tf.keras.callbacks.ModelCheckpoint(filepath=os.path.join(hdf5_dir, class_name) + '.hdf5',
monitor='val_precision',
save_best_only=True,
verbose=1),
tf.keras.callbacks.TensorBoard(log_dir=log_dir,
write_graph=True,
wr`enter code here`ite_images=True),
tf.keras.callbacks.EarlyStopping(monitor='val_precision', patience=10, verbose=2, mode='max')]
History = model.fit(train_images, train_masks,
steps_per_epoch=int(np.ceil(num_train_samples / float(train_batch_size))),
epochs=epochs,
validation_data=(val_images, val_masks),
validation_steps=int(np.ceil(num_val_samples / float(val_batch_size))),
callbacks=cp,
)
The feedback message you are getting is letting you know that for the epoch just completed, no improvement in validation precision occurred. This is probably happening because you have set verbose=2 in the callback settings, which is intended to give you a heads up that if you see the message for 10 consecutive epochs, your training will end.
Related
I was trying to make image augmentation and see how it will affect the model but for some reason I got this error
TypeError: '>' not supported between instances of 'int' and 'ImageDataGenerator'
I'm using efficientNetb4 with adding my own classifier layer.
augment = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rotation_range=30, validation_split=0.15)
train = augment.flow_from_directory(path, target_size=(380,380), batch_size=35, subset='training')
valid = augment.flow_from_directory(path, target_size=(380,380), batch_size=35, subset='validation')
base_model = keras.applications.EfficientNetB4(weights="imagenet",include_top=False, input_shape=(380, 380,3))
for layer in base_model.layers:
layer.trainable = False
avg = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(3, activation="softmax")(avg)
model = keras.Model(inputs=base_model.input, outputs=output)
earlystopping = keras.callbacks.EarlyStopping(monitor='loss', patience=3)
optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9, decay=0.0001)
model.compile(loss="sparse_categorical_crossentropy",optimizer=optimizer,metrics=["accuracy"])
history = model.fit_generator(train, augment, validation_data=valid, epochs=25, verbose=2, callbacks=[earlystopping])
I think the problem is the batch_size i specified but couldn't understand shy it caused this error
Please check documentation for fit_generator (https://faroit.com/keras-docs/1.2.0/models/model/):
fit_generator(self, generator, samples_per_epoch, nb_epoch, verbose=1, callbacks=[], validation_data=None, nb_val_samples=None, class_weight={}, max_q_size=10, nb_worker=1, pickle_safe=False, initial_epoch=0)
The second argument is 'samples_per_epoch', which is int, but you are passing ImageDataGenerator instead. Hence the error message. I don't see why you need to pass 'augment' here. The following should work:
history = model.fit_generator(train, validation_data=valid, epochs=25, verbose=2, callbacks=[earlystopping])
I'm attempting to find variable importance on a Neural Network I've built. Using tensorflow, it seems you can use either the tensorflow.keras way, or the kerasRegressor way. Admittedly, I have been reading documentation / stack overflow for hours and am confused on the differences. They seem to perform similarly but have slightly different pros/cons.
One issue I'm running into is when I use tf.keras to build the model, I am able to clearly compare my training data to my validation/testing data, and get an 'accuracy score'. But, when using kerasRegressor, I am not.
The difference here is the .evaluate() function, which kerasRegressor doesn't seem to have.
Questions:
How to evaluate performance of kerasRegressor model w/ same output as tf.keras.evaluate()?
kerasRegressor Code:
K.clear_session()
def base_model():
# 1- Instantiate Model
modelNEW = keras.Sequential()
# 2- Specify Shape of First Layer
modelNEW.add(layers.Dense(512, activation = 'relu', input_shape = ourInputShape))
# 3- Add the layers
modelNEW.add(layers.Dense(3, activation= 'softmax')) #softmax returns array of probability scores (num prior), and in this case we have to predict either CSCANCEL, MEMBERCANCEL, ACTIVE)
modelNEW.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
return modelNEW
# *** THIS IS SUPPOSED TO PREVENT OVERFITTING ***
from tensorflow.keras.callbacks import EarlyStopping
callbacks = [
EarlyStopping(patience=2)
]
yTrain = keras.utils.to_categorical(yTrain, 3)
yValidation = keras.utils.to_categorical(yValidation, 3)
currentModel = KerasRegressor(build_fn=base_model, epochs=100, batch_size=50, shuffle='True')
history = currentModel.fit(xTrain, yTrain)
Now if I want to test the accuracy, I have to use .predict()
prediction = currentModel.predict(xValidation)
# print(prediction)
# train_error = np.abs(yValidation - prediction)
# mean_error = np.mean(train_error)
# min_error = np.min(train_error)
# max_error = np.max(train_error)
# std_error = np.std(train_error)```
tf.Keras neural Network:
modelNEW = keras.Sequential()
modelNEW.add(layers.Dense(512, activation = 'relu', input_shape = ourInputShape))
modelNEW.add(layers.Dense(3, activation= 'softmax')) #softmax returns array of probability scores (num prior), and in this case we have to predict either CSCANCEL, MEMBERCANCEL, ACTIVE)
modelNEW.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
*** THIS IS SUPPOSED TO PREVENT OVERFITTING ***
from tensorflow.keras.callbacks import EarlyStopping
callbacks = [
EarlyStopping(patience=2)
]
yTrain = keras.utils.to_categorical(yTrain, 3)
yValidation = keras.utils.to_categorical(yValidation, 3)
history = modelNEW.fit(xTrain, yTrain, epochs=100, batch_size=50, shuffle="True")
This is the evaluation I need to see, and cannot with kerasRegressor:
# 6- Model evaluation with test data
test_loss, test_acc = modelNEW.evaluate(xValidation, yValidation)
print('test_acc:', test_acc)
Possible Workaround, still error:
# predictionTrain = currentModel.predict(xTrain)
predictionValidation = currentModel.predict(xValidation)
# print('Train Accuracy = ',accuracy_score(yTrain,np.argmax(pred_train, axis=1)))
print('Test Accuracy = ',accuracy_score(yValidation,np.argmax(predictionValidation, axis=1)))
: Classification metrics can't handle a mix of multilabel-indicator and binary targets
I'm using a pre trained InceptionV3 on Keras to retrain the model to make a binary image classification (data labeled with 0's and 1's).
I'm reaching about 65% of accuracy on my k-fold validation with never seen data, but the problem is the model is overfitting to soon. I need to improve this average accuracy, and I guess there is something related to this overfitting problem.
Here are the loss values on epochs:
Here is the code. The dataset and label variables are Numpy Arrays.
dataset = joblib.load(path_to_dataset)
labels = joblib.load(path_to_labels)
le = LabelEncoder()
labels = le.fit_transform(labels)
labels = to_categorical(labels, 2)
X_train, X_test, y_train, y_test = sk.train_test_split(dataset, labels, test_size=0.2)
X_train, X_val, y_train, y_val = sk.train_test_split(X_train, y_train, test_size=0.25) # 0.25 x 0.8 = 0.2
X_train = np.array(X_train)
y_train = np.array(y_train)
X_val = np.array(X_val)
y_val = np.array(y_val)
X_test = np.array(X_test)
y_test = np.array(y_test)
aug = ImageDataGenerator(
rotation_range=20,
zoom_range=0.15,
horizontal_flip=True,
fill_mode="nearest")
pre_trained_model = InceptionV3(input_shape = (299, 299, 3),
include_top = False,
weights = 'imagenet')
for layer in pre_trained_model.layers:
layer.trainable = False
x = layers.Flatten()(pre_trained_model.output)
x = layers.Dense(1024, activation = 'relu')(x)
x = layers.Dropout(0.2)(x)
x = layers.Dense(2, activation = 'softmax')(x) #already tried with sigmoid activation, same behavior
model = Model(pre_trained_model.input, x)
model.compile(optimizer = RMSprop(lr = 0.0001),
loss = 'binary_crossentropy',
metrics = ['accuracy']) #Already tried with Adam optimizer, same behavior
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=100)
mc = ModelCheckpoint('best_model_inception_rmsprop.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
history = model.fit(x=aug.flow(X_train, y_train, batch_size=32),
validation_data = (X_val, y_val),
epochs = 100,
callbacks=[es, mc])
The training dataset has 2181 images and validation has 727 images.
Something is wrong, but I can't tell what...
Any thoughts of what can be done to improve it?
One way to avoid overfitting is to use a lot of data. The main reason overfitting happens is because you have a small dataset and you try to learn from it. The algorithm will have greater control over this small dataset and it will make sure it satisfies all the datapoints exactly. But if you have a large number of datapoints, then the algorithm is forced to generalize and come up with a good model that suits most of the points.
Suggestions:
Use a lot of data.
Use less deep network if you have a small number of data samples.
If 2nd satisfies then don't use huge number of epochs - Using many epochs leads is kinda forcing your model to learn that and your model will learn it well but can not generalize.
From your loss graph , i see that the model is generalized at early epoch ( where there is intersection of both the train & val score) so plz try to use the model saved at that epoch ( and not the later epochs which seems to overfit)
Second option what you have is use lot of training samples..
If you have less no. of training samples then use data augmentations
Have you tried following?
Using a higher dropout value
Lower Learning Rate (lr=0.00001 or lr=0.000001 ...)
More data augmentation you can use.
It seems to me your data amount is low. You may use a lower ratio for test and validation (10%, 10%).
on my project related to machine learning,flower identification using CNN keras..showing following errors
ERRORS :
Anaconda3\lib\site-packages\keras\callbacks\callbacks.py:846: RuntimeWarning: Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: val_loss,val_accuracy,loss,accuracy
(self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
on the code:
#using Grid Search and Early stopping
es = EarlyStopping(monitor='val_acc', verbose=2, patience=25)
mc = ModelCheckpoint('./best_model_1.h5', monitor='val_acc', verbose=2, save_best_only=True)
Hyp_Model_1 = KerasClassifier(build_fn=Revised_1_fn)
#You need to pick the right hyper-parameters for your training (try with different ones)
learn_rate = [0.01]
batch_size = [32,75,100]
epochs = [5]
param_grid = dict(batch_size=batch_size, epochs=epochs, learn_rate = learn_rate)
randSearch_1 = GridSearchCV(estimator = Hyp_Model_1, param_grid=param_grid, cv=5)
new_grid_1 = randSearch_1.fit(X_train,y_train, validation_data = (X_val, y_val), verbose=2,callbacks=[es,mc])
As hinted in the error message, Keras does not recognize val_acc as a metric; you will need to type in val_accuracy as listed in the available options.
I ran this code but it didn't work. it was not error. in Jupyter notebook, It is still running (marked as In [*])
This is my jupyter notebook
_________________________________
In [*]: model.fit(
img_vector, y,
batch_size=128,
epochs=10,
verbose=2
)
___________________________
WARNING:tensorflow:From C:\Users\il357\Anaconda3\envs\mlbook\lib\site-packages\tensorflow\python\ops\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/10
Epoch 1/10 is marked but that's all.
I will write down previous code. I will omit importing code. (It is Deeplearning cookbook's code. you can find it in github.
and This code is running on my laptop.)
# The number of image files is very large.. I think this maybe is the problem.
pet_images_fn = [fn for fn in os.listdir('pet_images') if fn.endswith('.jpg')]
# make image vector
labels = []
idx_to_labels = []
label_to_idx = {}
for fn in pet_images_fn:
label, _ = fn.rsplit('_', 1)
if not label in label_to_idx:
label_to_idx[label] = len(idx_to_labels)
idx_to_labels.append(label)
labels.append(label_to_idx[label])
len(idx_to_labels)
def fetch_pet(pet):
img = image.load_img('pet_images/' + pet, target_size=(299, 299))
return image.img_to_array(img)
img_vector = np.asarray([fetch_pet(pet) for pet in pet_images_fn])
# make model with InceptionV3
base_model = InceptionV3(weights='imagenet', include_top=False, input_shape=(299, 299, 3))
for layer in base_model.layers:
layer.trainable = False
pool_2d = GlobalAveragePooling2D(name='pool_2d')(base_model.output)
dense = Dense(1024, name='dense', activation='relu')(pool_2d)
predictions = Dense(len(idx_to_labels), activation='softmax')(dense)
model = Model(inputs=base_model.input, outputs=predictions)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
# set label to one hot encoding
y = np.zeros((len(labels), len(idx_to_labels)))
for idx, label in enumerate(labels):
y[idx][label] = 1
model.fit(
img_vector, y,
batch_size=128,
epochs=15,
verbose=2
)
verbose=2in model.fit means it will only print one line after the epoch finishes, if you want the progress bar you should setverbose=1.