my simple regression model(by keras) doesn't work - keras

i'm beginner studying neural network.
i'm doing kaggle project - bike sharing demand.
i want to use simple neural network by keras, but loss is not decreasing.
what should i do?
-----------code--------------
# dataset from pandas
feature_names = ["season", "holiday", "workingday", "weather",
"temp", "atemp", "humidity", "windspeed",
"datetime_year", "datetime_hour", "datetime_dayofweek"]
label_name = ["count"]
X_train = train[feature_names] #shape (10886, 11)
Y_train = train[label_name] #shape (10886, 1)
X_test = test[feature_names]
# layers
model = Sequential()
model.add(Dense(units = 50, kernel_initializer = 'uniform', activation = 'relu', input_dim=11))
model.add(Dropout(0.3))
model.add(Dense(units = 50, kernel_initializer = 'uniform', activation = 'relu'))
model.add(Dropout(0.3))
model.add(Dense(units = 5, kernel_initializer = 'uniform', activation = 'relu'))
model.add(Dropout(0.3))
model.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics = ['accuracy'])
# Train
model.fit(X_train, Y_train, batch_size = 100, epochs = 200)
---------result------------
Epoch 1/200
10886/10886 [==============================] - 4s 325us/step - loss: 69206.2478 - acc: 0.0094
Epoch 2/200
10886/10886 [==============================] - 1s 93us/step - loss: 69184.5435 - acc: 0.0096
Epoch 3/200
10886/10886 [==============================] - 1s 89us/step - loss: 69181.6330 - acc: 0.0096
Epoch 4/200
10886/10886 [==============================] - 1s 93us/step - loss: 69179.0222 - acc: 0.0096
Epoch 5/200
10886/10886 [==============================] - 1s 91us/step - loss: 69175.7442 - acc: 0.0096
Epoch 6/200
10886/10886 [==============================] - 1s 109us/step - loss: 69171.9052 - acc: 0.0096
Epoch 7/200
10886/10886 [==============================] - 1s 122us/step - loss: 69171.6164 - acc: 0.0096
Epoch 8/200
10886/10886 [==============================] - 1s 92us/step - loss: 69167.6923 - acc: 0.0096
Epoch 9/200
10886/10886 [==============================] - 1s 91us/step - loss: 69166.2911 - acc: 0.0096
Epoch 10/200
10886/10886 [==============================] - 1s 94us/step - loss: 69164.1145 - acc: 0.0096
...


Try setting the learning rate in the model.compile section. Start with 0.0001, then 0.001 and 0.01 and see what happens.

Related

While training image segmentation network UNET- Validation loss is not getting decreased after some epochs, remains constant

I am training the UNET image segmentation network on brain tumor dataset from figshare, it is training well, training loss and training dice score both are changing accordingly or in the same tone with validation loss and validation dice score. Means no question of Overfitting. But after approximately 40 epochs no improvement in performance measures. It's toggling around loss 0.58 and dice score of 0.47. How to solve this? Please suggest me.
Below is my UNET network-
def unet(pretrained_weights = None,input_size = (512,512,3)):
inputs = Input(input_size)
conv1 = Convolution2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
conv1 = BatchNormalization()(conv1)
#conv1 = Dropout(0.2)(conv1)
conv1 = Convolution2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
conv1 = BatchNormalization()(conv1)
#conv1 = Dropout(0.2)(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Convolution2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)
conv2 = BatchNormalization()(conv2)
#conv2 = Dropout(0.1)(conv2)
conv2 = Convolution2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)
conv2 = BatchNormalization()(conv2)
#conv2 = Dropout(0.1)(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Convolution2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)
conv3 = BatchNormalization()(conv3)
#conv3 = Dropout(0.1)(conv3)
conv3 = Convolution2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)
conv3 = BatchNormalization()(conv3)
#conv3 = Dropout(0.1)(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
conv4 = Convolution2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)
conv4 = BatchNormalization()(conv4)
#conv4 = Dropout(0.1)(conv4)
conv4 = Convolution2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)
conv4 = BatchNormalization()(conv4)
#conv4 = Dropout(0.5)(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
conv5 = Convolution2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)
conv5 = BatchNormalization()(conv5)
#conv5 = Dropout(0.1)(conv5)
conv5 = Convolution2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)
conv5 = BatchNormalization()(conv5)
#conv5 = Dropout(0.5)(conv5)
up6 = Convolution2D(512, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv5))
merge6 = concatenate([conv4,up6], axis = 3)
conv6 = Convolution2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)
conv6 = BatchNormalization()(conv6)
#conv6 = Dropout(0.1)(conv6)
conv6 = Convolution2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)
conv6 = BatchNormalization()(conv6)
#conv6 = Dropout(0.1)(conv6)
up7 = Convolution2D(256, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))
merge7 = concatenate([conv3,up7], axis = 3)
conv7 = Convolution2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)
conv7 = BatchNormalization()(conv7)
#conv7 = Dropout(0.1)(conv7)
conv7 = Convolution2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)
conv7 = BatchNormalization()(conv7)
#conv7 = Dropout(0.1)(conv7)
up8 = Convolution2D(128, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))
merge8 = concatenate([conv2,up8], axis = 3)
conv8 = Convolution2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)
conv8 = BatchNormalization()(conv8)
#conv8 = Dropout(0.1)(conv8)
conv8 = Convolution2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)
conv8 = BatchNormalization()(conv8)
#conv8 = Dropout(0.1)(conv8)
up9 = Convolution2D(64, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))
merge9 = concatenate([conv1,up9], axis = 3)
conv9 = Convolution2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)
conv9 = BatchNormalization()(conv9)
#conv9 = Dropout(0.2)(conv9)
conv9 = Convolution2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
conv9 = BatchNormalization()(conv9)
#conv9 = Dropout(0.2)(conv9)
conv9 = Convolution2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
conv9 = BatchNormalization()(conv9)
#conv9 = Dropout(0.2)(conv9)
conv10 = Convolution2D(1, 1, activation = 'sigmoid')(conv9)
model = Model(input = inputs, output = conv10)
#model.summary()
if(pretrained_weights):
model.load_weights(pretrained_weights)
return model
Callback details initialized. Staring LR= 1e-4
callbacks = [EarlyStopping(monitor='val_loss',mode="min", patience=30,verbose=1,min_delta=1e-4),
ReduceLROnPlateau(monitor='val_loss',mode="min", factor=0.1,patience=8,verbose=1),
ModelCheckpoint(monitor='val_loss',mode="min",
filepath='weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-{epoch:03d}-{dice_coef:.6f}--{val_loss:.6f}.hdf5',save_weights_only=True, verbose=1),
CSVLogger('weights/anmol/1/UNET_mixed_loss_monitor_DC_new.csv')]
My user-defined Dice Score and loss functions. I have used dice_coef_loss here.
def dice_coef(y_true, y_pred, smooth=1):
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f * y_pred_f)
return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
def dice_loss(y_true, y_pred):
loss = 1 - dice_coef(y_true, y_pred)
return loss
def dice_coef_loss(y_true, y_pred):
return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)
Used 2605 images for training, 306 images for validation.
Train_image Train_mask
img_size = 512
image_args = dict(seed=seed,
batch_size=2,
shuffle=True,
class_mode=None,
target_size=(img_size, img_size),
color_mode='rgb')
mask_args = dict(seed=seed,
batch_size=2,
class_mode=None,
shuffle=True,
target_size=(img_size, img_size),
color_mode='grayscale')
DIR = 'raw/brain/'
image = 'images'
masks = 'masks'
# combine generators into one which yields image and masks
train_generator = zip(image_datagen.flow_from_directory(**image_args, directory=DIR+'train_'+image),
mask_datagen.flow_from_directory(**mask_args, directory=DIR+'train_'+masks))
validation_generator = zip(image_datagen.flow_from_directory(**image_args, directory=DIR+'validation_'+image),
mask_datagen.flow_from_directory(**mask_args, directory=DIR+'validation_'+masks))
model.fit_generator(train_generator, steps_per_epoch=1302, epochs=100, validation_data=validation_generator,validation_steps=153, callbacks=callbacks)
some training log shown below
Epoch 00041: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-041-0.466533--0.511900.hdf5
Epoch 42/100
1302/1302 [==============================] - 1063s 817ms/step - loss: 0.5939 - dice_coef: 0.4658 - val_loss: 0.5076 - val_dice_coef: 0.5430
Epoch 00042: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-042-0.465990--0.507603.hdf5
Epoch 43/100
1302/1302 [==============================] - 1062s 816ms/step - loss: 0.5928 - dice_coef: 0.4678 - val_loss: 0.5191 - val_dice_coef: 0.5270
Epoch 00043: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-043-0.467685--0.519115.hdf5
Epoch 44/100
1302/1302 [==============================] - 1063s 817ms/step - loss: 0.5966 - dice_coef: 0.4632 - val_loss: 0.5158 - val_dice_coef: 0.5364
Epoch 00044: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-044-0.463308--0.515760.hdf5
Epoch 45/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5892 - dice_coef: 0.4702 - val_loss: 0.4993 - val_dice_coef: 0.5507
Epoch 00045: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-045-0.470134--0.499294.hdf5
Epoch 46/100
1302/1302 [==============================] - 1063s 816ms/step - loss: 0.5960 - dice_coef: 0.4636 - val_loss: 0.5166 - val_dice_coef: 0.5329
Epoch 00046: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-046-0.463810--0.516552.hdf5
Epoch 47/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5920 - dice_coef: 0.4672 - val_loss: 0.5062 - val_dice_coef: 0.5427
Epoch 00047: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-047-0.467146--0.506242.hdf5
Epoch 48/100
1302/1302 [==============================] - 1063s 816ms/step - loss: 0.5938 - dice_coef: 0.4657 - val_loss: 0.5239 - val_dice_coef: 0.5277
Epoch 00048: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-048-0.465866--0.523923.hdf5
Epoch 49/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5962 - dice_coef: 0.4639 - val_loss: 0.5035 - val_dice_coef: 0.5434
Epoch 00049: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-049-0.463924--0.503518.hdf5
Epoch 50/100
1302/1302 [==============================] - 1063s 816ms/step - loss: 0.5854 - dice_coef: 0.4743 - val_loss: 0.5463 - val_dice_coef: 0.5066
Epoch 00050: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-050-0.474530--0.546343.hdf5
Epoch 51/100
1302/1302 [==============================] - 1063s 816ms/step - loss: 0.5840 - dice_coef: 0.4749 - val_loss: 0.5146 - val_dice_coef: 0.5360
Epoch 00051: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-051-0.475072--0.514581.hdf5
Epoch 52/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5852 - dice_coef: 0.4742 - val_loss: 0.5257 - val_dice_coef: 0.5256
Epoch 00052: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-052-0.474234--0.525729.hdf5
Epoch 53/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5857 - dice_coef: 0.4736 - val_loss: 0.5157 - val_dice_coef: 0.5315
Epoch 00053: ReduceLROnPlateau reducing learning rate to 9.999999747378752e-07.
Epoch 00053: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-053-0.473557--0.515651.hdf5
Epoch 54/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5852 - dice_coef: 0.4737 - val_loss: 0.5067 - val_dice_coef: 0.5421
Epoch 00054: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-054-0.473682--0.506671.hdf5
Epoch 55/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5903 - dice_coef: 0.4696 - val_loss: 0.4910 - val_dice_coef: 0.5571
Epoch 00055: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-055-0.469478--0.491024.hdf5
Epoch 56/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5876 - dice_coef: 0.4711 - val_loss: 0.5154 - val_dice_coef: 0.5340
Epoch 00056: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-056-0.471110--0.515441.hdf5
Epoch 57/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5897 - dice_coef: 0.4703 - val_loss: 0.5263 - val_dice_coef: 0.5258
Epoch 00057: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-057-0.470255--0.526310.hdf5
Epoch 58/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5849 - dice_coef: 0.4741 - val_loss: 0.5067 - val_dice_coef: 0.5451
Epoch 00058: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-058-0.474262--0.506664.hdf5
Epoch 59/100
1302/1302 [==============================] - 1062s 816ms/step - loss: 0.5816 - dice_coef: 0.4769 - val_loss: 0.5160 - val_dice_coef: 0.5348
Epoch 00059: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-059-0.476830--0.516005.hdf5
Epoch 60/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5891 - dice_coef: 0.4709 - val_loss: 0.5179 - val_dice_coef: 0.5318
Epoch 00060: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-060-0.470746--0.517893.hdf5
Epoch 61/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5873 - dice_coef: 0.4727 - val_loss: 0.5064 - val_dice_coef: 0.5431
Epoch 00061: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-061-0.472722--0.506373.hdf5
Epoch 62/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5803 - dice_coef: 0.4793 - val_loss: 0.5187 - val_dice_coef: 0.5319
Epoch 00062: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-062-0.479199--0.518674.hdf5
Epoch 63/100
1302/1302 [==============================] - 1066s 819ms/step - loss: 0.5843 - dice_coef: 0.4738 - val_loss: 0.5052 - val_dice_coef: 0.5459
Epoch 00063: ReduceLROnPlateau reducing learning rate to 9.999999974752428e-08.
Epoch 00063: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-063-0.473731--0.505171.hdf5
Epoch 64/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5859 - dice_coef: 0.4731 - val_loss: 0.5064 - val_dice_coef: 0.5419
Epoch 00064: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-064-0.473008--0.506380.hdf5
Epoch 65/100
1302/1302 [==============================] - 1064s 817ms/step - loss: 0.5836 - dice_coef: 0.4752 - val_loss: 0.4997 - val_dice_coef: 0.5508
Epoch 00065: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-065-0.475424--0.499673.hdf5
Epoch 66/100
1302/1302 [==============================] - 1063s 817ms/step - loss: 0.5932 - dice_coef: 0.4660 - val_loss: 0.5168 - val_dice_coef: 0.5338
Epoch 00066: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-066-0.465829--0.516758.hdf5
Epoch 67/100
1302/1302 [==============================] - 1063s 816ms/step - loss: 0.5820 - dice_coef: 0.4765 - val_loss: 0.5179 - val_dice_coef: 0.5323
Epoch 00067: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-067-0.476715--0.517926.hdf5
Epoch 68/100
1302/1302 [==============================] - 1062s 816ms/step - loss: 0.5912 - dice_coef: 0.4689 - val_loss: 0.5125 - val_dice_coef: 0.5375
Epoch 00068: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-068-0.468950--0.512456.hdf5
Epoch 69/100
1302/1302 [==============================] - 1062s 816ms/step - loss: 0.5820 - dice_coef: 0.4769 - val_loss: 0.5282 - val_dice_coef: 0.5237
Epoch 00069: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-069-0.476976--0.528154.hdf5
Epoch 70/100
1302/1302 [==============================] - 1062s 816ms/step - loss: 0.5845 - dice_coef: 0.4743 - val_loss: 0.5204 - val_dice_coef: 0.5303
Epoch 00070: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-070-0.474195--0.520356.hdf5
Epoch 71/100
1302/1302 [==============================] - 1063s 816ms/step - loss: 0.5886 - dice_coef: 0.4708 - val_loss: 0.5230 - val_dice_coef: 0.5270
Epoch 00071: ReduceLROnPlateau reducing learning rate to 1.0000000116860975e-08.
Epoch 00071: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-071-0.470715--0.523011.hdf5
Epoch 72/100
1302/1302 [==============================] - 1062s 816ms/step - loss: 0.5837 - dice_coef: 0.4759 - val_loss: 0.5216 - val_dice_coef: 0.5303
Epoch 00072: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-072-0.475787--0.521631.hdf5
Epoch 73/100
1302/1302 [==============================] - 1062s 815ms/step - loss: 0.5804 - dice_coef: 0.4780 - val_loss: 0.5333 - val_dice_coef: 0.5171
Epoch 00073: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-073-0.478063--0.533321.hdf5
Epoch 74/100
1302/1302 [==============================] - 1065s 818ms/step - loss: 0.5842 - dice_coef: 0.4747 - val_loss: 0.5126 - val_dice_coef: 0.5393
Epoch 00074: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-074-0.474628--0.512649.hdf5
Epoch 75/100
1302/1302 [==============================] - 1069s 821ms/step - loss: 0.5836 - dice_coef: 0.4755 - val_loss: 0.5103 - val_dice_coef: 0.5386
Epoch 00075: saving model to weights/anmol/1/UNET_sigmoid_focus_DC_2605_R_B_t-075-0.475690--0.510267.hdf5
Epoch 76/100
160/1302 [==>...........................] - ETA: 15:02 - loss: 0.6069 - dice_coef: 0.4548
​

LSTM time series data classification model low accuracy

I'm trying to create a model for time series analysis using LSTM layer, however accuracy is very low even when using Dense layers and no LSTM.
The data is time series (synthetic spectrum), which depends on 4 parameters. Changing the parameters enables to use different size datasets, where each sample is more or less different from the other. But no matter the size of dataset accuracy is always as low as 0.0 - 0.32 %.
Model with LSTM:
print(trainset.shape)
print(testset.shape)
print(trainlabels.shape)
model = Sequential()
model.add(Masking(mask_value=0.0, input_shape=(trainset.shape[1], trainset.shape[2])))
model.add(LSTM(10, activation='relu', stateful=False, return_sequences=False))
model.add(Dropout(0.3))
model.add(Dense(len(trainlabels), activation='relu'))
model.compile(loss='sparse_categorical_crossentropy',
optimizer='Adam', metrics=['accuracy'])
print(model.summary())
model.fit(trainset, trainlabels, validation_data=(testset, testlabels),
epochs=3, batch_size=10)
scores = model.evaluate(testset, testlabels, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))
OUTPUT:
(2478, 600, 1)
(620, 600, 1)
(2478,)
Model: "sequential_7"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
masking_7 (Masking) (None, 600, 1) 0
_________________________________________________________________
lstm_7 (LSTM) (None, 10) 480
_________________________________________________________________
dropout_7 (Dropout) (None, 10) 0
_________________________________________________________________
dense_7 (Dense) (None, 2478) 27258
=================================================================
Total params: 27,738
Trainable params: 27,738
Non-trainable params: 0
_________________________________________________________________
None
Train on 2478 samples, validate on 620 samples
Epoch 1/3
2478/2478 [==============================] - 53s 22ms/step - loss: 8.9022 - accuracy: 4.0355e-04 - val_loss: 7.8152 - val_accuracy: 0.0016
Epoch 2/3
2478/2478 [==============================] - 54s 22ms/step - loss: 7.8152 - accuracy: 4.0355e-04 - val_loss: 7.8152 - val_accuracy: 0.0016
Epoch 3/3
2478/2478 [==============================] - 53s 21ms/step - loss: 7.8152 - accuracy: 4.0355e-04 - val_loss: 7.8152 - val_accuracy: 0.0016
Accuracy: 0.16%
Some values in training data are 0.0, therefore Masking is used.
I have tried playing around with different loss, optimizer, activations, Dropout and layer parameters. The result is always the same even after adding more Dense layers, or changing batch size.
Model with Dense:
Data format is 2D instead of 3D.
model=keras.Sequential([
keras.layers.Masking(mask_value=0.0, input_shape=(trainset.shape[1],)),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dropout(0.05),
keras.layers.Dense(64, activation='relu'),
keras.layers.Dropout(0.05),
keras.layers.Dense(len(labels), activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(np.uint8(trainset), np.uint8(trainlabels), epochs=100)
test_loss, test_acc=model.evaluate(np.uint8(testset), np.uint8(testlabels),
verbose=2)
print(test_acc)
OUTPUT:
Train on 1239 samples
Epoch 1/100
1239/1239 [==============================] - 1s 1ms/sample - loss: 8.5421 - accuracy: 0.0033
Epoch 2/100
1239/1239 [==============================] - 0s 371us/sample - loss: 6.2039 - accuracy: 0.0025
Epoch 3/100
1239/1239 [==============================] - 0s 347us/sample - loss: 5.6502 - accuracy: 0.0033
****
Epoch 97/100
1239/1239 [==============================] - 0s 380us/sample - loss: 0.1472 - accuracy: 0.9746
Epoch 98/100
1239/1239 [==============================] - 0s 364us/sample - loss: 0.1562 - accuracy: 0.9680
Epoch 99/100
1239/1239 [==============================] - 1s 408us/sample - loss: 0.1511 - accuracy: 0.9721
Epoch 100/100
1239/1239 [==============================] - 0s 378us/sample - loss: 0.1719 - accuracy: 0.9680
310/1 - 0s - loss: 18.6845 - accuracy: 0.0000e+00
0.0
With this model loss is very low but so is accuracy.
What kind of model architecture should be used for my data?
Thanks in advance for helping to learn this stuff!

InceptionV3+LSTM activity recognition, accuracy grows for 10 epochs and then drops down

I'm trying to build model to do activity recognition.
Using InceptionV3 and backbone and LSTM for the detection, using pre-trained weights.
train_generator = datagen.flow_from_directory(
'dataset/train',
target_size=(1,224, 224),
batch_size=batch_size,
class_mode='categorical', # this means our generator will only yield batches of data, no labels
shuffle=True,
classes=['PlayingPiano','HorseRiding','Skiing', 'Basketball','BaseballPitch'])
validation_generator = datagen.flow_from_directory(
'dataset/validate',
target_size=(1,224, 224),
batch_size=batch_size,
class_mode='categorical', # this means our generator will only yield batches of data, no labels
shuffle=True,
classes=['PlayingPiano','HorseRiding','Skiing', 'Basketball','BaseballPitch'])
return train_generator,validation_generator
I train 5 classes so split my data into folders for train and validate.
This is my CNN+LSTM architecture
image = Input(shape=(None,224,224,3),name='image_input')
cnn = applications.inception_v3.InceptionV3(
weights='imagenet',
include_top=False,
pooling='avg')
cnn.trainable = False
encoded_frame = TimeDistributed(Lambda(lambda x: cnn(x)))(image)
encoded_vid = LSTM(256)(encoded_frame)
layer1 = Dense(512, activation='relu')(encoded_vid)
dropout1 = Dropout(0.5)(layer1)
layer2 = Dense(256, activation='relu')(dropout1)
dropout2 = Dropout(0.5)(layer2)
layer3 = Dense(64, activation='relu')(dropout2)
dropout3 = Dropout(0.5)(layer3)
outputs = Dense(5, activation='softmax')(dropout3)
model = Model(inputs=[image],outputs=outputs)
sgd = SGD(lr=0.001, decay = 1e-6, momentum=0.9, nesterov=True)
model.compile(optimizer=sgd,loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(train_generator,validation_data = validation_generator,steps_per_epoch=300, epochs=nb_epoch,callbacks=callbacks,shuffle=True,verbose=1)
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
image_input (InputLayer) (None, None, 224, 224, 3) 0
_________________________________________________________________
time_distributed_1 (TimeDist (None, None, 2048) 0
_________________________________________________________________
lstm_1 (LSTM) (None, 256) 2360320
_________________________________________________________________
dense_1 (Dense) (None, 512) 131584
_________________________________________________________________
dropout_1 (Dropout) (None, 512) 0
_________________________________________________________________
dense_2 (Dense) (None, 256) 131328
_________________________________________________________________
dropout_2 (Dropout) (None, 256) 0
_________________________________________________________________
dense_3 (Dense) (None, 64) 16448
_________________________________________________________________
dropout_3 (Dropout) (None, 64) 0
_________________________________________________________________
dense_4 (Dense) (None, 5) 325
_________________________________________________________________
Model compiles normally without problem.
Problem starts during the training. It reaches val_acc=0.50 and then drops back to val_acc=0.30 and the loss just freeze on 0.80 and mostly don't move.
Here the logs from training, as you see the model for some tome improves and then just slowly drops down and later just freeze.
Any idea what can be the reason?
Epoch 00002: val_loss improved from 1.56471 to 1.55652, saving model to ./weights_inception/Inception_V3.02-0.28.h5
Epoch 3/500
300/300 [==============================] - 66s 219ms/step - loss: 1.5436 - acc: 0.3281 - val_loss: 1.5476 - val_acc: 0.2981
Epoch 00003: val_loss improved from 1.55652 to 1.54757, saving model to ./weights_inception/Inception_V3.03-0.30.h5
Epoch 4/500
300/300 [==============================] - 66s 220ms/step - loss: 1.5109 - acc: 0.3593 - val_loss: 1.5284 - val_acc: 0.3588
Epoch 00004: val_loss improved from 1.54757 to 1.52841, saving model to ./weights_inception/Inception_V3.04-0.36.h5
Epoch 5/500
300/300 [==============================] - 66s 221ms/step - loss: 1.4167 - acc: 0.4167 - val_loss: 1.4945 - val_acc: 0.3553
Epoch 00005: val_loss improved from 1.52841 to 1.49446, saving model to ./weights_inception/Inception_V3.05-0.36.h5
Epoch 6/500
300/300 [==============================] - 66s 221ms/step - loss: 1.2941 - acc: 0.4683 - val_loss: 1.4735 - val_acc: 0.4443
Epoch 00006: val_loss improved from 1.49446 to 1.47345, saving model to ./weights_inception/Inception_V3.06-0.44.h5
Epoch 7/500
300/300 [==============================] - 66s 221ms/step - loss: 1.2096 - acc: 0.5116 - val_loss: 1.3738 - val_acc: 0.5186
Epoch 00007: val_loss improved from 1.47345 to 1.37381, saving model to ./weights_inception/Inception_V3.07-0.52.h5
Epoch 8/500
300/300 [==============================] - 66s 221ms/step - loss: 1.1477 - acc: 0.5487 - val_loss: 1.2337 - val_acc: 0.5788
Epoch 00008: val_loss improved from 1.37381 to 1.23367, saving model to ./weights_inception/Inception_V3.08-0.58.h5
Epoch 9/500
300/300 [==============================] - 66s 221ms/step - loss: 1.0809 - acc: 0.5831 - val_loss: 1.2247 - val_acc: 0.5658
Epoch 00009: val_loss improved from 1.23367 to 1.22473, saving model to ./weights_inception/Inception_V3.09-0.57.h5
Epoch 10/500
300/300 [==============================] - 66s 221ms/step - loss: 1.0362 - acc: 0.6089 - val_loss: 1.1704 - val_acc: 0.5774
Epoch 00010: val_loss improved from 1.22473 to 1.17035, saving model to ./weights_inception/Inception_V3.10-0.58.h5
Epoch 11/500
300/300 [==============================] - 66s 221ms/step - loss: 0.9811 - acc: 0.6317 - val_loss: 1.1612 - val_acc: 0.5616
Epoch 00011: val_loss improved from 1.17035 to 1.16121, saving model to ./weights_inception/Inception_V3.11-0.56.h5
Epoch 12/500
300/300 [==============================] - 66s 221ms/step - loss: 0.9444 - acc: 0.6471 - val_loss: 1.1533 - val_acc: 0.5613
Epoch 00012: val_loss improved from 1.16121 to 1.15330, saving model to ./weights_inception/Inception_V3.12-0.56.h5
Epoch 13/500
300/300 [==============================] - 66s 221ms/step - loss: 0.9072 - acc: 0.6650 - val_loss: 1.1843 - val_acc: 0.5361
Epoch 00013: val_loss did not improve from 1.15330
Epoch 14/500
300/300 [==============================] - 66s 221ms/step - loss: 0.8747 - acc: 0.6744 - val_loss: 1.2135 - val_acc: 0.5258
Epoch 00014: val_loss did not improve from 1.15330
Epoch 15/500
300/300 [==============================] - 67s 222ms/step - loss: 0.8666 - acc: 0.6829 - val_loss: 1.1585 - val_acc: 0.5443
Epoch 00015: val_loss did not improve from 1.15330
Epoch 16/500
300/300 [==============================] - 66s 222ms/step - loss: 0.8386 - acc: 0.6926 - val_loss: 1.1503 - val_acc: 0.5482
Epoch 00016: val_loss improved from 1.15330 to 1.15026, saving model to ./weights_inception/Inception_V3.16-0.55.h5
Epoch 17/500
300/300 [==============================] - 66s 221ms/step - loss: 0.8199 - acc: 0.7023 - val_loss: 1.2162 - val_acc: 0.5288
Epoch 00017: val_loss did not improve from 1.15026
Epoch 18/500
300/300 [==============================] - 66s 222ms/step - loss: 0.8018 - acc: 0.7150 - val_loss: 1.1995 - val_acc: 0.5179
Epoch 00018: val_loss did not improve from 1.15026
Epoch 19/500
300/300 [==============================] - 66s 221ms/step - loss: 0.7923 - acc: 0.7186 - val_loss: 1.2218 - val_acc: 0.5137
Epoch 00019: val_loss did not improve from 1.15026
Epoch 20/500
300/300 [==============================] - 67s 222ms/step - loss: 0.7748 - acc: 0.7268 - val_loss: 1.2880 - val_acc: 0.4574
Epoch 00020: val_loss did not improve from 1.15026
Epoch 21/500
300/300 [==============================] - 66s 221ms/step - loss: 0.7604 - acc: 0.7330 - val_loss: 1.2658 - val_acc: 0.4861

The model is starting to overfit. Ideally as you increase number of epochs training loss will decrease(depends on learning rate), if its not able to decrease may be your model can have a high bias for the data. You can use bigger model(more parameters or deeper model).
you can also to reduce the learning rate, if it still freezes then model may have a low bias.

Thank you for the help. Yes, the problem was overfitting, so i made more aggresive dropout on LSTM, and it helped. But the accuracy on val_loss and acc_val still very low
video = Input(shape=(None, 224,224,3))
cnn_base = VGG16(input_shape=(224,224,3),
weights="imagenet",
include_top=False)
cnn_out = GlobalAveragePooling2D()(cnn_base.output)
cnn = Model(inputs=cnn_base.input, outputs=cnn_out)
cnn.trainable = False
encoded_frames = TimeDistributed(cnn)(video)
encoded_sequence = LSTM(32, dropout=0.5, W_regularizer=l2(0.01), recurrent_dropout=0.5)(encoded_frames)
hidden_layer = Dense(units=64, activation="relu")(encoded_sequence)
dropout = Dropout(0.2)(hidden_layer)
outputs = Dense(5, activation="softmax")(dropout)
model = Model([video], outputs)
Here the logs
Epoch 00033: val_loss improved from 1.62041 to 1.57951, saving model to
./weights_inception/Inception_V3.33-0.76.h5
Epoch 34/500
100/100 [==============================] - 54s 537ms/step - loss: 0.6301 - acc:
0.9764 - val_loss: 1.6190 - val_acc: 0.7627
Epoch 00034: val_loss did not improve from 1.57951
Epoch 35/500
100/100 [==============================] - 54s 537ms/step - loss: 0.5907 - acc:
0.9840 - val_loss: 1.5927 - val_acc: 0.7608
Epoch 00035: val_loss did not improve from 1.57951
Epoch 36/500
100/100 [==============================] - 54s 537ms/step - loss: 0.5783 - acc:
0.9812 - val_loss: 1.3477 - val_acc: 0.7769
Epoch 00036: val_loss improved from 1.57951 to 1.34772, saving model to
./weights_inception/Inception_V3.36-0.78.h5
Epoch 37/500
100/100 [==============================] - 54s 537ms/step - loss: 0.5618 - acc:
0.9802 - val_loss: 1.6545 - val_acc: 0.7384
Epoch 00037: val_loss did not improve from 1.34772
Epoch 38/500
100/100 [==============================] - 54s 537ms/step - loss: 0.5382 - acc:
0.9818 - val_loss: 1.8298 - val_acc: 0.7421
Epoch 00038: val_loss did not improve from 1.34772
Epoch 39/500
100/100 [==============================] - 54s 536ms/step - loss: 0.5080 - acc:
0.9844 - val_loss: 1.7948 - val_acc: 0.7290
Epoch 00039: val_loss did not improve from 1.34772
Epoch 40/500
100/100 [==============================] - 54s 537ms/step - loss: 0.4800 - acc:
0.9892 - val_loss: 1.8036 - val_acc: 0.7522

Conv1D and batch_size questions

I face two problems when I implement 1D convnet for multi-channel sequential data.
(224 samples x 300 time sequential x 19 channels)
1) I set batch_size as 7 but it jumps with 5 times of that.
not 7 14 21 28, but 7, 56, 105, 147... what's wrong with mine?
2) when I look at the records of accuracy, it looks like to learn NOTHING.
is it impossible implement classifier for multi-channel sequential data with Conv1D?
If possible can you give me some advice from my code?
#result
x_train shape: (224, 300, 19)
224 train samples
28 test samples
Train on 224 samples, validate on 28 samples
Epoch 1/50
7/224 [..............................] - ETA: 68s - loss: 0.6945 - acc: 0.5714
56/224 [======>.......................] - ETA: 6s - loss: 0.6993 - acc: 0.4464
105/224 [=============>................] - ETA: 2s - loss: 0.6979 - acc: 0.4381
147/224 [==================>...........] - ETA: 1s - loss: 0.6968 - acc: 0.4422
189/224 [========================>.....] - ETA: 0s - loss: 0.6953 - acc: 0.4444
224/224 [==============================] - 2s - loss: 0.6953 - acc: 0.4420 - val_loss: 0.6956 - val_acc: 0.5000
Epoch 2/50
7/224 [..............................] - ETA: 0s - loss: 0.6759 - acc: 0.5714
63/224 [=======>......................] - ETA: 0s - loss: 0.6924 - acc: 0.5556
133/224 [================>.............] - ETA: 0s - loss: 0.6905 - acc: 0.5338
203/224 [==========================>...] - ETA: 0s - loss: 0.6903 - acc: 0.5567
224/224 [==============================] - 0s - loss: 0.6923 - acc: 0.5357 - val_loss: 0.6968 - val_acc: 0.5000
# code
from __future__ import print_function
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Activation
from keras.layers import Conv2D, MaxPooling2D, Conv1D, MaxPooling1D
import numpy as np
batch_size = 7
num_classes = 2
epochs = 50
# input data dimensions : 300 sequential x 19 channels
eeg_rows, num_ch = 300, 19
x_train = np.load('eeg_train.npy')
y_train = np.load('label_train.npy')
x_test = np.load('eeg_test.npy')
y_test = np.load('label_test.npy')
x_valid = np.load('eeg_valid.npy')
y_valid = np.load('label_valid.npy')
x_train = x_train.reshape(x_train.shape[0], eeg_rows, num_ch)
x_test = x_test.reshape(x_test.shape[0], eeg_rows,num_ch)
x_valid = x_valid.reshape(x_valid.shape[0], eeg_rows, num_ch)
input_shape = (eeg_rows, num_ch)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_valid = x_test.astype('float32')
x_train /= 100
x_test /= 100
x_valid /= 100
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
# model
conv = Sequential()
conv.add(Conv1D(32, 3, input_shape=input_shape, activation='relu', padding='same'))
conv.add(Conv1D(32, 3, activation='relu', padding='same'))
conv.add(MaxPooling1D(pool_size=2, strides=2, padding='same'))
conv.add(Dropout(0.2))
conv.add(Flatten())
conv.add(Dense(16, activation='relu'))
conv.add(Dropout(0.5))
conv.add(Dense(2, activation='softmax'))
conv.compile(loss='categorical_crossentropy',
optimizer=keras.optimizers.Adam(),
metrics=['accuracy'])
# train
conv.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_valid, y_valid))
score = conv.evaluate(x_valid, y_valid, verbose=0)
print(conv.summary())
print(conv.input_shape)
print(conv.output_shape)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

TimeDistributed Model Not learning

I am trying to train model to play Chrome Dino (the offline game).
The idea was to have 6 last screenshots of the game, use CNN on each separately (to extract features) and then put those features as timesteps into LSTM.
My training data is X = [6 timestep game screenshots] -> y=[1,0] (keep running, jump)
Timestep example
I have even split the database so it has 50% jump examples and 50% keep running examples.
Sadly I am stuck at 50% accuracy and loss is stuck too.
198/198 [==============================] - 0s - loss: 0.6944 - acc: 0.4596
Epoch 91/100
198/198 [==============================] - 0s - loss: 0.6932 - acc: 0.5000
Epoch 92/100
198/198 [==============================] - 0s - loss: 0.6932 - acc: 0.5000
Epoch 93/100
198/198 [==============================] - 0s - loss: 0.6932 - acc: 0.5000
Epoch 94/100
198/198 [==============================] - 0s - loss: 0.6933 - acc: 0.5000
Epoch 95/100
198/198 [==============================] - 0s - loss: 0.6942 - acc: 0.5000
Epoch 96/100
198/198 [==============================] - 0s - loss: 0.6939 - acc: 0.5000
Epoch 97/100
198/198 [==============================] - 0s - loss: 0.6935 - acc: 0.5000
I have tried many model hyperparams with different layers, but I always get the same result.
Current model
model = Sequential()
model.add(TimeDistributed(Convolution2D(64, 3, 3, activation='relu'), input_shape=(FRAMES_TO_PROCESS, FRAME_HEIGHT,FRAME_WIDTH, FRAME_FILTERS )))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(TimeDistributed(ZeroPadding2D((1,1))))
model.add(TimeDistributed(Convolution2D(64, 3, 3, activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides=(2,2))))
model.add(TimeDistributed(ZeroPadding2D((1,1))))
model.add(TimeDistributed(Convolution2D(128, 3, 3, activation='relu')))
model.add(TimeDistributed(ZeroPadding2D((1,1))))
model.add(TimeDistributed(Convolution2D(128, 3, 3, activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides=(2,2))))
model.add(Activation('relu'))
model.add(TimeDistributed(Flatten()))
model.add(Dropout(0.1))
model.add(LSTM(120, return_sequences=False))
model.add(Dense(2, activation='softmax'))
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=['accuracy'])
Any idea what went wrong?

Resources