I have a simple model for demonstration:
input_layer = Input(shape=(100,))
encoded = Dense(2, activation='relu')(input_layer)
X = np.ones((1000, 100))
Y = np.ones((1000, 2))
print(X.shape)
model = Model(input_layer, encoded)
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(x=X, y=Y, batch_size = 2)
Output is:
2.2.4
(1000, 100)
Epoch 1/1
1000/1000 [==============================] - 3s 3ms/step - loss: 1.3864
Why there are 1000 iterations in one epoch(as shown in the output).
I tried changing this but does not changes the output. I guess it should have been 1000/2 = 500. Please explain what is wrong with my understanding and how can i set the batch size appropriately.
Thanks
In model.fit the numbers in the left part of the progress bar count samples, so it is always the current samples / total number of samples.
Maybe you are confused because it works different in model.fit_generator. There you actually see iterations or batches being counted.
It changes the batch size, the bar progresses faster although you do not explicitly see it as a step. I had the same question in my mind some time ago.
If you want to explicitly see each step, you can use steps_per_epoch and validation_steps.
An example is listed below.
model.fit_generator(training_generator,
steps_per_epoch=steps_per_epoch,
epochs=epochs,
validation_data=validation_generator,
validation_steps=validation_steps)
In this case, steps_per_epoch = number_of_training_samples / batch_size, while validation_steps = number_of_training_samples / batch_size.
During the training, you will see 500 steps instead of 1000 (provided that you have 1000 training samples and your batch_size is 2).
Related
I am trying to fit BERT text classifier. My training and test data looks as follows.
x_train = data["TEXT"].head(4500).tolist()
y_train= [label2id[label] for label in data["EMOTION"].head(4500).values.tolist()]
x_test = data["TEXT"].tail(500).tolist()
y_test = [label2id[label] for label in data["EMOTION"].tail(500).values.tolist()]
Then, I download the pretrained BERT model (uncased_L-12_H-768_A-12.zip)...
(x_train, y_train), (x_test, y_test), preproc = text.texts_from_array(x_train=x_train, y_train=y_train,
x_test=x_test, y_test=y_test,
class_names=data['EMOTION'].unique().tolist(),
preprocess_mode='bert',
ngram_range=1,
maxlen=350,
max_features=35000)
For classification, we set the bert model as
model = text.text_classifier('bert', train_data=(x_train, y_train), preproc=preproc)
learner = ktrain.get_learner(model, train_data=(x_train, y_train), batch_size=6)
Finally, I try fit to the model using 1cycle policy rate
hist = learner.fit_onecycle(2e-5, 1)
I get the result with 750 samples rather than 4500 samples. I also tested this with various data. So there is always variations in data items. Can you give an idea what is behind it?
begin training using onecycle policy with max lr of 2e-05...
750/750 [==============================] - 875s 1s/step - loss: 0.3740 - accuracy: 0.8544
Thank you for your response in Advance.
My personal idea is that when you instantiate the learner with ktrain.get_learner you give it a batch size = 6 as input parameter.
So when you try to train the learner by simply doing learner.fit_onecycle (2e-5, 1), it takes exactly one batch for training, in fact 4500 training data / batch size (6) = 750 data to train on.
At this point either try to change the batch size, or do a for loop like this:
for epoch in range(X):
....
for batch in chunker(train, batch_size):
....
where chunker() could be something like:
def chunker(sequence, size):
"""useful for splitting a sequence into minibatches"""
for i in range(0, len(sequence), size):
chunk = sequence[i:i+size]
# sort sentences in batch by length in descending order
chunk.sort(key=lambda x: len(x), reverse=True)
yield chunk
In a nutshell the idea is that you have to do a loop in which you go to select each time a set of data (batch) that you want to use to train your model.
I am training my CNN using the following code:
history = model.fit_generator(
train_data_gen,
steps_per_epoch=int(np.ceil(train_data_gen.n / float(batch_size))),
epochs=num_epochs,
validation_data=val_data_gen,
validation_steps=int(np.ceil(val_data_gen.n / float(batch_size))),
verbose=1,
)
and these are the output:
The accuracy is okay but why is epoch 1/20 repeating itself multiple times?
Your steps per epoch is not managed well. You are trying to validate twice too..
Try without the float and int:
history = model.fit_generator(
train_data_gen,
steps_per_epoch=np.ceil(train_data_gen.n / batch_size),
epochs=num_epochs,
validation_data=val_data_gen,
validation_steps=np.ceil(val_data_gen.n / batch_size),
verbose=1,
)
print out the values of generator size, the steps_per_epoch should reflect number of training steps (usually determined by generator function)
And make sure train_data_gen.n = total_training_samples and same for val_data_gen.n = total_validation_samples
OR, just comment out steps_per_epoch and validation_steps and the model will take on the generator length as training steps.
Consult: https://keras.io/models/model/#fit_generator
I am doing image classification by using Keras , I have 8k images(input) in training sample and 2k images(input) in test sample , defined epoch as 25 . I noticed that epoch is very slow (approx takes an hour for first iteration) .
can any one suggest how can I overcome this , and what is the reason it takes hell lot of time?
code below..
PART-1
initialise neural network
from keras.models import Sequential
#package to perfom first layer , which is convolution , using 2d as it is for image , for video it will be 3d
from keras.layers import Convolution2D
#to perform max pooling on convolved layer
from keras.layers import MaxPool2D
#to convert the pool feature map into large feature vector, will be input for ANN
from keras.layers import Flatten
#to add layeres on ANN
from keras.layers import Dense
#STEP -1
#Initializing CNN
classifier = Sequential()
#add convolution layer
classifier.add(Convolution2D(filters=32,kernel_size=(3,3),strides=(1, 1),input_shape= (64,64,3),activation='relu'))
#filters - Number of feature detecters that we are going to apply in image
#kernel_size - dimension of feature detector
#strides moving thru one unit at a time
#input shape - shape of the input image on which we are going to apply filter thru convolution opeation,
#we will have to covert the image into that shape in image preprocessing before feeding it into convolution
#channell 3 for rgb and 1 for bw , and dimension of pixels
#activation - function we use to avoid non linearity in image
#STEP -2
#add pooling
#this step will significantly reduce the size of feature map , and makes it easier for computation
classifier.add(MaxPool2D(pool_size=(2,2)))
#pool_size - factor by which to downscale
#STEP -3
#flattern the feature map
classifier.add(Flatten())
#STEP -4
#hidden layer
classifier.add(Dense(units=128,activation='relu',kernel_initializer='uniform'))
#output layer
classifier.add(Dense(units=1,activation='sigmoid'))
#Compiling the CNN using stochastic gradient descend
classifier.compile(optimizer='adam',loss = 'binary_crossentropy',
metrics=['accuracy'])
#loss function should be categorical_crossentrophy if output is more than 2 class
#PART2 - Fitting CNN to image
#copied from keras documentation
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
training_set = train_datagen.flow_from_directory(
'/Users/arunramji/Downloads/Sourcefiles/CNN_Imageclassification/Convolutional_Neural_Networks/dataset/training_set',
target_size=(64, 64),
batch_size=32,
class_mode='binary')
test_set = test_datagen.flow_from_directory(
'/Users/arunramji/Downloads/Sourcefiles/CNN_Imageclassification/Convolutional_Neural_Networks/dataset/test_set',
target_size=(64, 64),
batch_size=32,
class_mode='binary')
classifier.fit_generator(
training_set,
steps_per_epoch=8000, #number of input (image)
epochs=25,
validation_data=test_set,
validation_steps=2000) # number of training sample
classifier.fit(
training_set,
steps_per_epoch=8000, #number of input (image)
epochs=25,
validation_data=test_set,
validation_steps=2000)
You are setting steps_per_epoch to the wrong value (this is why it takes longer than necessary): it is not set to the number of data points. steps_per_epoch should be set to the size of the dataset divided by the batch size, which should be 8000/32 = 250 for your training set, and 63 for your validation set.
Update:
As Matias in his answer pointed out, your steps_per_epoch parameter setting in your fit method led for the huge slowing down per epoch.
From the fit_generator documentation:
steps_per_epoch:
Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size) Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
validation_steps: Only relevant if validation_data is a generator.
Total number of steps (batches of samples) to yield from
validation_data generator before stopping at the end of every epoch.
It should typically be equal to the number of samples of your
validation dataset divided by the batch size. Optional for Sequence:
if unspecified, will use the len(validation_data) as a number of
steps.
Actually Keras has an inconsistency at handling the two parameters, as fit method raises an Valuerror if you uses a simple dataset instead of datagenerator and set the parameters like batch_size=batch_size, steps_per_epoch=num_samples:
ValueError: Number of samples 60000 is less than samples required for specified batch_size 200 and steps 60000
But when data comes from datagenerator it doesn't handle the same problem letting you to have an issue like the current one.
I made a little example code to check these up.
The fit method with steps_per_epoch=num_samples:
Number of samples: 60000
Number of samples per batch: 200
Train for 60000 steps, validate for 50 steps
Epoch 1/5
263/60000 [..............................] - ETA: 4:07:09 - loss: 0.2882 - accuracy: 0.9116
with ETA (estimated time): 4:07:09,
as this is for 60000 steps, each of 200 samples per batch.
The same fit with steps_per_epoch=num_samples // batch_size:
Number of samples: 60000
Number of samples per batch: 200
Train for 300 steps, validate for 50 steps
Epoch 1/5
28/300 [=>............................] - ETA: 1:15 - loss: 1.0946 - accuracy: 0.6446
with ETA: 1:15
Solution:
steps_per_epoch=(training_set.shape[0] // batch_size)
validation_steps=(validation_set.shape[0] // batch_size)
Further possible issues regarding performance:
As #SajanGohil wrote in his comment train_datagen.flow_from_director make some tasks like file operations, preprocessings before actual traning process which sometimes takes more time as the traning itself.
So to avoid these extratime, you can do the preprocessing task before the whole traning process separately only once. Then you can use these preprocessed data at traning time.
Anyway CNNs with vast images are rather time and resource consuming tasks, which assumes GPU usage for this reason.
I'm new to Keras so I get confused between Keras documentation and other people's examples of using fit_generator. When I test-ran this code with 100 samples (for the sake of speedy output. The actual training sample is more than 10k) in batch size of 32 for 2 epochs:
# Create a generator that generates an image and a label one at a time (because loading all data into memory will freeze the laptop)
def generate_transform(imgs, lbls):
while 1:
for i in range(len(imgs)):
img = np.array(cv2.resize(imgs[i], (224, 224)))
lbl = to_categorical(lbls[i], num_classes=10)
yield (img, lbl)
history = model.fit_generator(generate_transform(x[:100], y[:100]),
steps_per_epoch=100/32,
samples_per_epoch=100,
nb_epoch=2,
validation_data=generate_transform(x_test[:100], y_test[:100]),
validation_steps=100)
# nb_val_samples=100?)
I got this UserWarning:
D:\Users\jason\AppData\Local\Continuum\Anaconda3\lib\site-packages\ipykernel_launcher.py:8: UserWarning: The semantics of the Keras 2 argument `steps_per_epoch` is not the same as the Keras 1 argument `samples_per_epoch`. `steps_per_epoch` is the number of batches to draw from the generator at each epoch. Basically steps_per_epoch = samples_per_epoch/batch_size. Similarly `nb_val_samples`->`validation_steps` and `val_samples`->`steps` arguments have changed. Update your method calls accordingly.
D:\Users\jason\AppData\Local\Continuum\Anaconda3\lib\site-packages\ipykernel_launcher.py:8: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<generator..., steps_per_epoch=100, validation_data=<generator..., validation_steps=100, epochs=2)`
And the output looked like this:
Epoch 1/2
100/100 [==============================] - 84s 836ms/step - loss: 3.0745 - acc: 0.4500 - val_loss: 2.3886 - val_acc: 0.0300
Epoch 2/2
100/100 [==============================] - 86s 864ms/step - loss: 0.3654 - acc: 0.9000 - val_loss: 2.4644 - val_acc: 0.0900
My questions are:
Was my call correct with those arguments and their supplied values?
Was my model trained with 32 images and labels at each step; and it was trained with 100/32 steps per epoch?
Am I required to use the argument steps_per_epoch?
Which argument should I use: validation_steps or nb_val_samples?
Did my model validate all 100 samples of the validation generator (as indicated by x_test[:100]) for 100 times (as indicated by validation_steps=100) or it's only validating 100 times of one sample each (because validation generator only yield one sample at a time)? Why didn't the output show the number of steps?
Did my model use the trained weight from the first epoch to re-train the same data again, that's why the training accuracy jumped from 0.45 in the first epoch to 0.9 in the second epoch?
Could you please help me with the above questions?
Thanks in advance.
I ran into this problem & solved it in my code below {before in Keras 1.1.2 ==> after in Keras 2.2.4}:
294 # Old Keras==1.1.2 fit_generator
295 # history = model.fit_generator(
296 # train_data_generator.get_data(),
297 # samples_per_epoch=train_data_generator.get_num_files(),
298 # nb_epoch=config["num_epochs"],
300 # verbose=1,
301 # validation_data=validation_data_generator.get_data(should_shuffle=False),
302 # nb_val_samples=validation_data_generator.get_num_files(),
303 # nb_worker=2,
304 # max_q_size=batch_size,
305 # pickle_safe=True)
306
307 # New working! Keras 2.2.4 fit_generator
309 history = model.fit_generator(
310 train_data_generator.get_data(),
312 verbose=1,
313 validation_data=validation_data_generator.get_data(should_shuffle=False),
314 steps_per_epoch=train_data_generator.get_num_files() // batch_size,
315 epochs=config["num_epochs"],
316 validation_steps=validation_data_generator.get_num_files() // batch_size,
317 workers=2, use_multiprocessing=True,
318 max_queue_size=batch_size)
Looking at your code, you need just steps_per_epoch not samples_per_epoch, change nb_epoch to epochs. I don't fully understand your code or the training/validation setup (100 train & validation samples?) and it's best to ask one question per post, but I'll take a stab at fixing your code (untested of course):
Keep in mind that number_of_steps == number_of_samples // batch_size and if 100 is num_training_samples, you'll have to have a pretty small batch_size for number_of_steps to make sense:
history = model.fit_generator(
generate_transform(x[:100], y[:100]), # training data generator
verbose=1,
val_data=generate_transform(x_test[:100], y_test[:100]), # validation data generator
steps_per_epoch=100 // batch_size, # 100 is num_training_samples, divided by batch_size == steps_per_epoch
epochs=2,
val_steps=100 // batch_size # 100 is num_val_samples, divided by batch_size == val_steps
)
I am working on a basic Classification task with Keras and I seem to have stumbled upon a problem where I need some assistance.
I have 200 samples for training and a 100 for validation, I intend to use a ImageDataGenerator to increase the number of training samples for my task. I want to make sure of the total number of training images that are passed to the fit_generator().
I know that the steps_per_epoch defines the total number of batches we get from a generator and ideally it should be number of samples divided by the batch size.
However, this is where things do not add up for me. Here is a snippet of my code:
num_samples = 200
batch_size = 10
gen = ImageDataGenerator(horizontal_flip = True,
vertical_flip = True,
width_shift_range = 0.1,
height_shift_range = 0.1,
zoom_range = 0.1,
rotation_range = 10
)
x,y = shuffle(img_data,img_label, random_state=2)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.333, random_state=2)
generator = gen.flow(x_train, y_train, save_to_dir='check_images/sample_run')
new_network.fit_generator(generator, steps_per_epoch=len(x_train)/batch_size, validation_data=(x_test, y_test), epochs=1, verbose=2)
I am saving the augmented images to see how the images turn out from the ImageDataGenerator and also to ascertain the number of images that are generated from it.
After running this code for a single epoch, I get 600 images in my directory, a number which I cannot arrive at, or maybe I am making a mistake.
Any assistance in making me understand the calculation in this code would be deeply appreciated. Has anyone come across similar problems ?
TIA
gen.flow() creates a NumpyArrayIterator internally and that in turn uses Iterator to calculate the steps_per_epoch. Ideally if steps_per_epoch is None, then the calculation is done as steps_per_epoch = (x.shape[0] + batch_size - 1) // batch_size which is approximately same as your calculation.
Not sure why you see more number of samples. Could you compute the x.shape[0] and double check if your code is as per what you explained?