This is my first time asking a question here (that's mean I'm really need help) and sorry for my bad English. I want to make a cnn-lstm layer for video classification in Keras but I have a problem on making my y_train. I will describe my problem after this.
I have videos dataset (1 video has 10 frames) and I converted the videos to images.
First I splited the dataset to xtrain, xtest, ytrain, and ytest (20% test, 80% train) and I did it.
X_train, X_test = img_data[:trainco], img_data[trainco:]
y_train, y_test = y[:trainco], y[trainco:]
X_train shape : (2280, 64, 64, 1) -> I have 2280 images, 64x64 height x widht, 1 channel
y_train shape : (2280, 26) -> 26 classes
And then I must reshape them before entering the cnn-lstm process. *note : I do the same thing with x_test and y_test
time_steps = 10 (because I have 10 frames per video)
X_train = X_train.reshape(int(X_train.shape[0] / time_steps), time_steps, X_train.shape[1], X_train.shape[2], X_train.shape[3])
y_train = y_train.reshape(int(y_train.shape[0] / time_steps), time_steps, y_train.shape[1])
X_train shape : (228, 10, 64, 64, 1), y_train shape : (228, 10, 26)
And then this is my model :
model = Sequential()
model.add(TimeDistributed(Conv2D(32, (3, 3), strides=(2, 2), activation='relu', padding='same'), input_shape=X_train.shape[1:]))
model.add(TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2))))
model.add(TimeDistributed(Conv2D(32, (3, 3), padding='same', activation='relu')))
model.add(TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2))))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(256, return_sequences=False, input_shape=(64, 64)))
model.add(Dense(128))
model.add(Dense(64))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=["accuracy"])
checkpoint = ModelCheckpoint(fname, monitor='acc', verbose=1, save_best_only=True, mode='max', save_weights_only=True)
hist = model.fit(X_train, y_train, batch_size=num_batch, nb_epoch=num_epoch, verbose=1, validation_data=(X_test, y_test), callbacks=[checkpoint])
But I got an error that says
ValueError: Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (228, 10, 26)
Like it says expected to have 2 dimensions. I changed the code to
y_train = y_train.reshape(int(y_train.shape[0] / time_steps), y_train.shape[1])
And I got an error again that says
ValueError: cannot reshape array of size 59280 into shape (228,26)
And then I change the code again to
y_train = y_train.reshape(y_train.shape[0], y_train.shape[1])
And I still got an error
ValueError: Input arrays should have the same number of samples as target arrays. Found 228 input samples and 2280 target samples.
What should I do? I know the problem but I don't know how to solve it. Please help me.
I recreated a slightly simplified version of your situation to reproduce the problem. Basically, it appears that the LSTM layer is only putting out one result for the entire sequence of time steps, thereby reducing the dimension from 3 to 2 in the output. If you run my program below, I've added the model.summary() which provides details of the architecture.
from keras import Sequential
from keras.layers import TimeDistributed, Dense, Conv2D, MaxPooling2D, Flatten, LSTM
import numpy as np
X_train = np.random.random((228, 10, 64, 64, 1))
y_train = np.random.randint(2, size=(228, 10, 26))
num_classes = 26
# Create the model
model = Sequential()
model.add(TimeDistributed(Conv2D(32, (3, 3), strides=(2, 2), activation='relu', padding='same'), input_shape=X_train.shape[1:]))
model.add(TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2))))
model.add(TimeDistributed(Conv2D(32, (3, 3), padding='same', activation='relu')))
model.add(TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2))))
model.add(TimeDistributed(Flatten(),name='Flatten'))
model.add(LSTM(256, return_sequences=False, input_shape=(64, 64)))
model.add(Dense(128))
model.add(Dense(64))
model.add(Dense(num_classes, activation='softmax', name='FinalDense'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=["accuracy"])
#
model.summary()
# hist = model.fit(X_train, y_train, epochs=1)
I believe you'll need to decide if you want to reduce the dimension of the y_train (target) data to be consistent with the model, or change the model. I hope this helps.
Related
I am building a CNN for non image data in Keras 2.1.0 on Window 10.
My input feature is a 3x12 matrix of non negative number and my output is a binary multi-label vector with length 6x1
And I was running into this error expected conv2d_14_input to have shape (3, 12, 1) but got array with shape (3, 12, 6500)
Here is my code below
import tensorflow as tf
from scipy.io import loadmat
import numpy as np
from tensorflow.keras.layers import BatchNormalization
import matplotlib.pyplot as plt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout
from tensorflow.keras.layers import Conv2D, MaxPool2D, Flatten
reshape_channel_train = loadmat('reshape_channel_train')
reshape_channel_test = loadmat('reshape_channel_test.mat')
reshape_label_train = loadmat('reshape_label_train')
reshape_label_test = loadmat('reshape_label_test')
X_train = reshape_channel_train['store_train']
X_test = reshape_channel_test['store_test']
X_train = np.expand_dims(X_train,axis = 0)
X_test = np.expand_dims(X_test, axis = 0)
Y_train = reshape_label_train['label_train']
Y_test = reshape_label_test['label_test']
classifier = Sequential()
classifier.add(Conv2D(8, kernel_size=(3,3) , input_shape=(3, 12, 1), padding="same"))
classifier.add(BatchNormalization())
classifier.add(Activation('relu'))
classifier.add(Conv2D(8, kernel_size=(3,3), input_shape=(3, 12, 1), padding="same"))
classifier.add(BatchNormalization())
classifier.add(Activation('relu'))
classifier.add(Flatten())
classifier.add(Dense(8, activation='relu'))
classifier.add(Dense(6, activation='sigmoid'))
classifier.compile(optimizer='nadam', loss='binary_crossentropy', metrics=['accuracy'])
history = classifier.fit(X_train, Y_train, batch_size = 32, epochs=100,
validation_data=(X_test, Y_test), verbose=2)
After some searching, I have use the dimension expanding trick but it seem not to work
X_train = np.expand_dims(X_train,axis = 0)
X_test = np.expand_dims(X_test, axis = 0)
The X_train variable containing 6500 training instances is loaded from a Matlab .mat file with dimension 3x12x6500.
Where each training instance is a 3x12 matrix.
Before using the expand_dim tricks, the k-th training sample could be invoke by X_train[:,:,k] and X_train[:,:,k].shape would return (3,12). Also X_train.shape would return (3, 12, 6500)
After using the expand_dim tricks the command X_train[:,:,k].shape would return (1, 3, 6500)
Please help me with this !
Thank you
you manage your data wrongly. A Conv2D layer accepts data in this format (n_sample, height, width, channels) which in your case (for your X_train) became (6500,3,12,1). you need to simply reconduct to this case
# create data as in your matlab data
n_class = 6
n_sample = 6500
X_train = np.random.uniform(0,1, (3,12,n_sample)) # (3,12,n_sample)
Y_train = tf.keras.utils.to_categorical(np.random.randint(0,n_class, n_sample)) # (n_sample, n_classes)
# reshape your data for conv2d
X_train = X_train.transpose(2,0,1) # (n_sample,3,12)
X_train = np.expand_dims(X_train, -1) # (n_sample,3,12,1)
classifier = Sequential()
classifier.add(Conv2D(8, kernel_size=(3,3) , input_shape=(3, 12, 1), padding="same"))
classifier.add(BatchNormalization())
classifier.add(Activation('relu'))
classifier.add(Conv2D(8, kernel_size=(3,3), padding="same"))
classifier.add(BatchNormalization())
classifier.add(Activation('relu'))
classifier.add(Flatten())
classifier.add(Dense(8, activation='relu'))
classifier.add(Dense(n_class, activation='softmax'))
classifier.compile(optimizer='nadam', loss='categorical_crossentropy', metrics=['accuracy'])
history = classifier.fit(X_train, Y_train, batch_size = 32, epochs=2, verbose=2)
# get predictions
pred = np.argmax(classifier.predict(X_train), 1)
I also use a softmax activation with categorical_crossentropy which is more suited for multiclass problem but you can also modify this. remember to applicate the same data manipulation also on your test data
you need to pass data_format="channels_last" argument, bcoz your channels are at last
you try this:
x_train=x_train.reshape((6500,3,12,1))
x_test=x_test.reshape((-1,3,12,1))
and in each of conv2d layer conv2D(<other args>, data_format="channels_last")
I have a sample data of page visits of one page for 803 days. I have extracted features from data like mean, median etc and final shape of data is (803, 25). I had taken a train set of 640 and a test set of 160. I am trying to use CNN+LSTM model using Keras. But I am getting an error in model.fit method.
I have tried permute layer and changed input shapes but still not able to fix it.
trainX.shape = (642, 1, 25)
trainY.shape = (642,)
testX.shape = (161, 1, 25)
testY.shape = (161,)
'''python
# Basic layer
model = Sequential()
model.add(TimeDistributed(Convolution2D(filters = 32, kernel_size = (3, 3), strides=1, padding='SAME', input_shape = (642, 25, 1), activation = 'relu')))
model.add(TimeDistributed(Convolution2D(filters = 32, kernel_size = (3, 3), activation = 'relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size = (2, 2))))
model.add(TimeDistributed(Convolution2D(32, 3, 3, activation = 'relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size = (2, 2))))
model.add(TimeDistributed(Flatten()))
model.add(Permute((2, 3), input_shape=(1, 25)))
model.add(LSTM(units=54, return_sequences=True))
# To avoid overfitting
model.add(Dropout(0.2))
# Adding 6 more layers
model.add(LSTM(units=25, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=54))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(units = 1, activation='relu', kernel_regularizer=regularizers.l1(0.0001))))
model.add(PReLU(weights=None, alpha_initializer="zero")) # add an advanced activation
model.compile(optimizer = 'adam', loss = customSmapeLoss, metrics=['mae'])
model.fit(trainX, trainY, epochs = 50, batch_size = 32)
predictions = model.predict(testX)
'''
#Runtime Error
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-218-86932db86d0b> in <module>()
42
43 model.compile(optimizer = 'adam', loss = customSmapeLoss, metrics=['mae'])
---> 44 model.fit(trainX, trainY, epochs = 50, batch_size = 32)
Error - IndexError: list index out of range
The input_shape requires a tuple of length 4 when it using TimeDistributed with Conv2D
see https://keras.io/layers/wrappers/
input_shape=(10, 299, 299, 3))
Dont you think your data field is too little? usually, CNN+LSTM is meant for a more complicated task with thousands of sequential images/videos.
I'm training an CNN with LSTM, where I use TimeDistributed but apparently it wants an extra dimension for the data. I don't know how to add it.
My thought is that the problem is in ImageGenerator, but I don't know how to reshape images generated from it.
cnn_model = Sequential()
cnn_model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(128,128,3)))
cnn_model.add(MaxPooling2D(pool_size=(2, 2)))
cnn_model.add(Conv2D(32, (3, 3), activation='relu'))
cnn_model.add(MaxPooling2D(pool_size=(2, 2)))
cnn_model.add(Conv2D(64, (3, 3), activation='relu'))
cnn_model.add(MaxPooling2D(pool_size=(2, 2)))
cnn_model.add(Conv2D(128, (3, 3), activation='relu'))
cnn_model.add(MaxPooling2D(pool_size=(2, 2)))
cnn_model.add(Flatten())
model = Sequential()
model.add(TimeDistributed(cnn_model, input_shape=(16, 128, 128,3)))
model.add(LSTM(128, return_sequences=True, dropout=0.5))
# model.add(Dropout(0.2)) #added
model.add(Dense(4, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
batch_size = 16
train_datagen = ImageDataGenerator(rescale=1. / 255)
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
'train/', # this is the target directory
target_size=(128,128),
batch_size=batch_size,
class_mode='categorical',
shuffle=True,
classes=['class_0', 'class_1','class_2','class_3'])
validation_generator = test_datagen.flow_from_directory(
'test/',
target_size=(128,128),
batch_size=batch_size,
class_mode='categorical',
shuffle=True,
classes=['class_0', 'class_1','class_2','class_3'])
model.fit_generator(
train_generator,
steps_per_epoch=47549 // batch_size,
epochs=5,
validation_data=validation_generator,
validation_steps=5444 // batch_size)
But I'm getting the following error message
ValueError: Error when checking input: expected time_distributed_136_input to have 5 dimensions, but got array with shape (16, 128, 128, 3)
Data folder is as follows:
-- train
-- class 0
-- vid 1
-- frame1.jpg
-- frame2.jpg
-- frame3.jpg
-- class 1
-- frame1.jpg
-- frame2.jpg
-- frame3.jpg
-- class 2
-- class 3
-- test
(same as train)
Thanks for any help.
You're fogetting the first dimension of every tensor, which is the batch size. You don't define a batch size unless it's absolutely necessary, thus the input shapes don't consider it.
When you defined input_shape=(16,128,128,3), this means that your data must have five dimensions: (examples, 16, 128, 128, 3)
And the examples dimension is missing in your data.
If you say they're movies, you should have data like (movies, frames, height, width, channels), probably. Then this would be accepted by input_shape=(frames, height, width, channels).
After several trials, I ended up using the same code, but with a tweaked version of the Keras "ImageDataGenerator" Class to add an extra dimension to the data, so that it becomes 5D.
(This is also valid for the use of Conv3D)
For anyone who face the same problem, you can find my tweaked version of the ImageDataGenerator class here.
It's the same as the main Keras ImageDataGenerator but I added an option to take more than one image/frame on each iteration. This is by changing the parameter frames_per_step to specify the number of frames/images you want to include in each iteration.
So here's how to use it:
from tweaked_ImageGenerator_v2 import ImageDataGenerator
datagen = ImageDataGenerator()
train_data=datagen.flow_from_directory('path/to/data', target_size=(x, y), batch_size=32, frames_per_step=4)
I think your problem is with your model. You define the input shape of your TimeDistributed in your model as input_shape=(16, 128, 128,3) which I guess it should be input_shape=(128, 128,3).
change this line :
model.add(TimeDistributed(cnn_model, input_shape=(16, 128, 128,3)))
to:
model.add(TimeDistributed(cnn_model, input_shape=(128, 128,3)))
And I hope it will work.
I have a dataset of about 160k images of 160 classes and I'm trying to classify them using CNN. Training on 120k images for 20 epochs i start with loss ~ 4.9 and val_loss ~ 4.6 which improves to about 3.3 and 3.2 after 20 epochs. I Really tried to read the documentation of Keras and understand what does that mean, but i couldn't, so I'm asking if someone would explain to me in the context of my model what does that mean for my model. I mean what does the loss score represent? What does it say for the model?
num_classes = 154
batch_size = 64
input_shape = (50,50,3)
epochs = 20
X, y = load_data()
# input image dimensions
img_rows, img_cols = 50, 50
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
model = Sequential()
model.add(Conv2D(64, kernel_size=(5, 5),
activation='relu',
padding = 'same',
input_shape=input_shape))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.20))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
I think it will help to watch a few tutorials online on CNN, to begin with. Basically, you want your loss to reduce with the training epochs which is what is observed in your case. Typically we look at how both losses are evolving over the entire training period. Observing how the train and validation loss is changing helps us have an understand whether the model is overfitting or not. You can check This link for a basic explanation to detect overfitting.
Ideally, you would want both your training as well as test loss to reduce with iterations. It is a measure of the error committed by the model in classification (as accuracy increase you expect the loss to reduce)
I am having an issue when trying to train my model in Keras 2.0.8, Python 3.6.1, and a Tensorflow Backend.
Error Message:
ValueError: Error when checking target: expected dense_4 to have shape (None, 2) but got array with shape (2592, 1)
X_train = numpy.swapaxes(X_train, 1, 3)
X_test = numpy.swapaxes(X_test, 1, 3)
print("X_train shape: ") --> size = (2592, 1, 1366, 96)
print("-----")
print("X_test shape") --> size = (648, 1, 1366, 96)
print("-----")
print(Y_train.shape) --> size = (2592,)
print("-----")
print("Y_test shape") --> size = (648,)
Relevant Code snippets:
K.set_image_dim_ordering('th')
K.set_image_data_format('channels_first')
def create_model(weights_path=None):
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu', padding="same", input_shape=(1, 1366, 96)))
model.add(Conv2D(64, (3, 3), activation='relu', dim_ordering="th"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(16, activation='relu'))
model.add(Dense(2, activation='softmax'))
if weights_path:
model.load_weights(weights_path)
return model
model = create_model()
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.SGD(lr=0.01),
metrics=['accuracy'])
history = model.fit(X_train, Y_train,
batch_size=32,
epochs=100,
verbose=1,
validation_data=(X_test, Y_test))
Line 142, where I call model.fit() is where I am getting this error
Things I have tried to fix this error
Referenced these stack overflow posts:
I tried to reshape the Y_test and Y_train numpy arrays using the following code:
Y_train.reshape(2592, 2)
Y_test.reshape(648, 2)
However, I get the following error:
ValueError: cannot reshape array of size 2592 into shape (2592,2)
It seems to me you need to change the last layer of the NN:
def create_model(weights_path=None):
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu', padding="same", input_shape=(1, 1366, 96)))
model.add(Conv2D(64, (3, 3), activation='relu', dim_ordering="th"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
if weights_path:
model.load_weights(weights_path)
return model
As you are using the categorical_crossentropy loss, you have to use one-hot encoded labels. For this you can use the function to_categorical from keras.utils.np_utils
from keras.utils import np_utils
y_train_onehot = np_utils.to_categorical(y_train)
y_test_onehot = np_utils.to_categorical(y_test)
Then use the one-hot encoded labels to train your model.