I am doing image classification by using Keras , I have 8k images(input) in training sample and 2k images(input) in test sample , defined epoch as 25 . I noticed that epoch is very slow (approx takes an hour for first iteration) .
can any one suggest how can I overcome this , and what is the reason it takes hell lot of time?
code below..
PART-1
initialise neural network
from keras.models import Sequential
#package to perfom first layer , which is convolution , using 2d as it is for image , for video it will be 3d
from keras.layers import Convolution2D
#to perform max pooling on convolved layer
from keras.layers import MaxPool2D
#to convert the pool feature map into large feature vector, will be input for ANN
from keras.layers import Flatten
#to add layeres on ANN
from keras.layers import Dense
#STEP -1
#Initializing CNN
classifier = Sequential()
#add convolution layer
classifier.add(Convolution2D(filters=32,kernel_size=(3,3),strides=(1, 1),input_shape= (64,64,3),activation='relu'))
#filters - Number of feature detecters that we are going to apply in image
#kernel_size - dimension of feature detector
#strides moving thru one unit at a time
#input shape - shape of the input image on which we are going to apply filter thru convolution opeation,
#we will have to covert the image into that shape in image preprocessing before feeding it into convolution
#channell 3 for rgb and 1 for bw , and dimension of pixels
#activation - function we use to avoid non linearity in image
#STEP -2
#add pooling
#this step will significantly reduce the size of feature map , and makes it easier for computation
classifier.add(MaxPool2D(pool_size=(2,2)))
#pool_size - factor by which to downscale
#STEP -3
#flattern the feature map
classifier.add(Flatten())
#STEP -4
#hidden layer
classifier.add(Dense(units=128,activation='relu',kernel_initializer='uniform'))
#output layer
classifier.add(Dense(units=1,activation='sigmoid'))
#Compiling the CNN using stochastic gradient descend
classifier.compile(optimizer='adam',loss = 'binary_crossentropy',
metrics=['accuracy'])
#loss function should be categorical_crossentrophy if output is more than 2 class
#PART2 - Fitting CNN to image
#copied from keras documentation
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
training_set = train_datagen.flow_from_directory(
'/Users/arunramji/Downloads/Sourcefiles/CNN_Imageclassification/Convolutional_Neural_Networks/dataset/training_set',
target_size=(64, 64),
batch_size=32,
class_mode='binary')
test_set = test_datagen.flow_from_directory(
'/Users/arunramji/Downloads/Sourcefiles/CNN_Imageclassification/Convolutional_Neural_Networks/dataset/test_set',
target_size=(64, 64),
batch_size=32,
class_mode='binary')
classifier.fit_generator(
training_set,
steps_per_epoch=8000, #number of input (image)
epochs=25,
validation_data=test_set,
validation_steps=2000) # number of training sample
classifier.fit(
training_set,
steps_per_epoch=8000, #number of input (image)
epochs=25,
validation_data=test_set,
validation_steps=2000)
You are setting steps_per_epoch to the wrong value (this is why it takes longer than necessary): it is not set to the number of data points. steps_per_epoch should be set to the size of the dataset divided by the batch size, which should be 8000/32 = 250 for your training set, and 63 for your validation set.
Update:
As Matias in his answer pointed out, your steps_per_epoch parameter setting in your fit method led for the huge slowing down per epoch.
From the fit_generator documentation:
steps_per_epoch:
Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size) Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
validation_steps: Only relevant if validation_data is a generator.
Total number of steps (batches of samples) to yield from
validation_data generator before stopping at the end of every epoch.
It should typically be equal to the number of samples of your
validation dataset divided by the batch size. Optional for Sequence:
if unspecified, will use the len(validation_data) as a number of
steps.
Actually Keras has an inconsistency at handling the two parameters, as fit method raises an Valuerror if you uses a simple dataset instead of datagenerator and set the parameters like batch_size=batch_size, steps_per_epoch=num_samples:
ValueError: Number of samples 60000 is less than samples required for specified batch_size 200 and steps 60000
But when data comes from datagenerator it doesn't handle the same problem letting you to have an issue like the current one.
I made a little example code to check these up.
The fit method with steps_per_epoch=num_samples:
Number of samples: 60000
Number of samples per batch: 200
Train for 60000 steps, validate for 50 steps
Epoch 1/5
263/60000 [..............................] - ETA: 4:07:09 - loss: 0.2882 - accuracy: 0.9116
with ETA (estimated time): 4:07:09,
as this is for 60000 steps, each of 200 samples per batch.
The same fit with steps_per_epoch=num_samples // batch_size:
Number of samples: 60000
Number of samples per batch: 200
Train for 300 steps, validate for 50 steps
Epoch 1/5
28/300 [=>............................] - ETA: 1:15 - loss: 1.0946 - accuracy: 0.6446
with ETA: 1:15
Solution:
steps_per_epoch=(training_set.shape[0] // batch_size)
validation_steps=(validation_set.shape[0] // batch_size)
Further possible issues regarding performance:
As #SajanGohil wrote in his comment train_datagen.flow_from_director make some tasks like file operations, preprocessings before actual traning process which sometimes takes more time as the traning itself.
So to avoid these extratime, you can do the preprocessing task before the whole traning process separately only once. Then you can use these preprocessed data at traning time.
Anyway CNNs with vast images are rather time and resource consuming tasks, which assumes GPU usage for this reason.
Related
Problem
I am training a deep learning model in PyTorch for binary classification, and I have a dataset containing unbalanced class proportions. My minority class makes up about 10% of the given observations. To avoid the model learning to just predict the majority class, I want to use the WeightedRandomSampler from torch.utils.data in my DataLoader.
Let's say I have 1000 observations (900 in class 0, 100 in class 1), and a batch size of 100 for my dataloader.
Without weighted random sampling, I would expect each training epoch to consist of 10 batches.
Questions
Will only 10 batches be sampled per epoch when using this sampler - and consequently, would the model 'miss' a large portion of the majority class during each epoch, since the minority class is now overrepresented in the training batches?
Will using the sampler result in more than 10 batches being sampled per epoch (meaning the same minority class observations may appear many times, and also that training would slow down)?
A small snippet of code to use WeightedRandomSampler
First, define the function:
def make_weights_for_balanced_classes(images, nclasses):
n_images = len(images)
count_per_class = [0] * nclasses
for _, image_class in images:
count_per_class[image_class] += 1
weight_per_class = [0.] * nclasses
for i in range(nclasses):
weight_per_class[i] = float(n_images) / float(count_per_class[i])
weights = [0] * n_images
for idx, (image, image_class) in enumerate(images):
weights[idx] = weight_per_class[image_class]
return weights
And after this, use it in the following way:
import torch
dataset_train = datasets.ImageFolder(traindir)
# For unbalanced dataset we create a weighted sampler
weights = make_weights_for_balanced_classes(dataset_train.imgs, len(dataset_train.classes))
weights = torch.DoubleTensor(weights)
sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, len(weights))
train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=args.batch_size, shuffle = True,
sampler = sampler, num_workers=args.workers, pin_memory=True)
It depends on what you're after, check torch.utils.data.WeightedRandomSampler documentation for details.
There is an argument num_samples which allows you to specify how many samples will actually be created when Dataset is combined with torch.utils.data.DataLoader (assuming you weighted them correctly):
If you set it to len(dataset) you will get the first case
If you set it to 1800 (in your case) you will get the second case
Will only 10 batches be sampled per epoch when using this sampler - and consequently, would the model 'miss' a large portion of the majority class during each epoch [...]
Yes, but new samples will be returned after this epoch passes
Will using the sampler result in more than 10 batches being sampled per epoch (meaning the same minority class observations may appear many times, and also that training would slow down)?
Training would not slow down, each epoch would take longer, but convergence should be approximately the same (as less epochs will be necessary due to more data in each).
I am trying to fit BERT text classifier. My training and test data looks as follows.
x_train = data["TEXT"].head(4500).tolist()
y_train= [label2id[label] for label in data["EMOTION"].head(4500).values.tolist()]
x_test = data["TEXT"].tail(500).tolist()
y_test = [label2id[label] for label in data["EMOTION"].tail(500).values.tolist()]
Then, I download the pretrained BERT model (uncased_L-12_H-768_A-12.zip)...
(x_train, y_train), (x_test, y_test), preproc = text.texts_from_array(x_train=x_train, y_train=y_train,
x_test=x_test, y_test=y_test,
class_names=data['EMOTION'].unique().tolist(),
preprocess_mode='bert',
ngram_range=1,
maxlen=350,
max_features=35000)
For classification, we set the bert model as
model = text.text_classifier('bert', train_data=(x_train, y_train), preproc=preproc)
learner = ktrain.get_learner(model, train_data=(x_train, y_train), batch_size=6)
Finally, I try fit to the model using 1cycle policy rate
hist = learner.fit_onecycle(2e-5, 1)
I get the result with 750 samples rather than 4500 samples. I also tested this with various data. So there is always variations in data items. Can you give an idea what is behind it?
begin training using onecycle policy with max lr of 2e-05...
750/750 [==============================] - 875s 1s/step - loss: 0.3740 - accuracy: 0.8544
Thank you for your response in Advance.
My personal idea is that when you instantiate the learner with ktrain.get_learner you give it a batch size = 6 as input parameter.
So when you try to train the learner by simply doing learner.fit_onecycle (2e-5, 1), it takes exactly one batch for training, in fact 4500 training data / batch size (6) = 750 data to train on.
At this point either try to change the batch size, or do a for loop like this:
for epoch in range(X):
....
for batch in chunker(train, batch_size):
....
where chunker() could be something like:
def chunker(sequence, size):
"""useful for splitting a sequence into minibatches"""
for i in range(0, len(sequence), size):
chunk = sequence[i:i+size]
# sort sentences in batch by length in descending order
chunk.sort(key=lambda x: len(x), reverse=True)
yield chunk
In a nutshell the idea is that you have to do a loop in which you go to select each time a set of data (batch) that you want to use to train your model.
I have a simple model for demonstration:
input_layer = Input(shape=(100,))
encoded = Dense(2, activation='relu')(input_layer)
X = np.ones((1000, 100))
Y = np.ones((1000, 2))
print(X.shape)
model = Model(input_layer, encoded)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(x=X, y=Y, batch_size = 2)
Output is:
2.2.4
(1000, 100)
Epoch 1/1
1000/1000 [==============================] - 3s 3ms/step - loss: 1.3864
Why there are 1000 iterations in one epoch(as shown in the output).
I tried changing this but does not changes the output. I guess it should have been 1000/2 = 500. Please explain what is wrong with my understanding and how can i set the batch size appropriately.
Thanks
In model.fit the numbers in the left part of the progress bar count samples, so it is always the current samples / total number of samples.
Maybe you are confused because it works different in model.fit_generator. There you actually see iterations or batches being counted.
It changes the batch size, the bar progresses faster although you do not explicitly see it as a step. I had the same question in my mind some time ago.
If you want to explicitly see each step, you can use steps_per_epoch and validation_steps.
An example is listed below.
model.fit_generator(training_generator,
steps_per_epoch=steps_per_epoch,
epochs=epochs,
validation_data=validation_generator,
validation_steps=validation_steps)
In this case, steps_per_epoch = number_of_training_samples / batch_size, while validation_steps = number_of_training_samples / batch_size.
During the training, you will see 500 steps instead of 1000 (provided that you have 1000 training samples and your batch_size is 2).
I have pre-stored bottleneck features (.npy files) obtained from VGG16 for around 10k images. Training a SVM classifier (3-class classification) on these features gave me an accuracy of 90% on the test set. These images are obtained from videos. I want to train an LSTM in keras on top of these features. My code snippet can be found below. The issue is that the training accuracy is not going above 43%, which is unexpected. Please help me in debugging the issue. I have tried with different learning rates.
#Asume all necessary imports done
classes = 3
frames = 5
channels = 3
img_height = 224
img_width = 224
epochs = 20
#Model definition
model = Sequential()
model.add(TimeDistributed(Flatten(),input_shape=(frames,7,7,512)))
model.add(LSTM(256,return_sequences=False))
model.add(Dense(1024,activation="relu"))
model.add(Dense(3,activation="softmax"))
optimizer = Adam(lr=0.1,beta_1=0.9,beta_2=0.999,epsilon=None,decay=0.0)
model.compile (loss="categorical_crossentropy",optimizer=optimizer,metrics=["accuracy"])
model.summary()
train_data = np.load(open('bottleneck_features_train.npy','rb'))
#final_img_data shape --> 2342,5,7,7,512
#one_hot_labels shape --> 2342,3
model.fit(final_img_data,one_hot_labels,epochs=epochs,batch_size=2)
You are probably missing the local minimum, because learning rate is too high. Try to decrease learning rate to 0.01 -- 0.001 and increase number of epochs. Also, decrease Dense layer neurons from 1024 to half. Otherwise you may overfit.
How do I get the weights of all filters (like 32 ,64, etc.) of a Conv2D layer in Keras after each epoch? I mention that, because initial weights are random but after optimization they will change.
I checked this answer but did not understand. Please help me find a solution of getting the weights of all the filter and after every epoch.
And one more question is that in Keras documentation for the Conv2D layer input shape is (samples, channels, rows, cols). What exactly does samples mean? Is it the total number of inputs we have (like in MNIST data set it is 60.000 training images) or the batch size (like 128 or other)?
Samples = batch size = number of images in a batch
Keras will often use None for this dimension, meaning it can vary and you don't have to set it.
Although this dimension actually exists, when you create a layer, you pass input_shape without it:
Conv2D(64,(3,3), input_shape=(channels,rows,cols))
#the standard it (rows,cols,channels), depending on your data_format
To have actions done after each epoch (or batch), you can use a LambdaCallback, passing the on_epoch_end function:
#the function to call back
def get_weights(epoch,logs):
wsAndBs = model.layers[indexOfTheConvLayer].get_weights()
#or model.get_layer("layerName").get_weights()
weights = wsAndBs[0]
biases = wsAndBs[1]
#do what you need to do with them
#you can see the epoch and the logs too:
print("end of epoch: " + str(epoch)) for instance
#the callback
from keras.callbacks import LambdaCallback
myCallback = LambdaCallback(on_epoch_end=get_weights)
Pass this callback to the training function:
model.fit(...,...,... , callbacks=[myCallback])