I am developing a LSTM model to predict stock movement. It seems that the model is well trained, with low val_loss and loss. But it doesn't work well to predict real data.
The following is the output of the training:
Epoch 75: val_loss improved from 0.03284 to 0.03071, saving model to /data/nn/python/us/stock/idx1/model/dow_257.h5
572/572 - 50s - loss: 0.0379 - accuracy: 0.9485 - val_loss: 0.0307 - val_accuracy: 0.9593 - 50s/epoch - 88ms/step
Epoch 76/100
Epoch 85: val_loss did not improve from 0.04519
580/580 - 47s - loss: 0.0546 - accuracy: 0.9173 - val_loss: 0.0500 - val_accuracy: 0.9284 - 47s/epoch - 82ms/step
Epoch 86/100
Epoch 76: val_loss did not improve from 0.03071
572/572 - 46s - loss: 0.0562 - accuracy: 0.9246 - val_loss: 0.0309 - val_accuracy: 0.9580 - 46s/epoch - 81ms/step
Epoch 77/100
Epoch 86: val_loss improved from 0.04519 to 0.04331, saving model to /data/nn/python/us/stock/idx1/model/dow_260.h5
580/580 - 47s - loss: 0.0563 - accuracy: 0.9192 - val_loss: 0.0433 - val_accuracy: 0.9383 - 47s/epoch - 81ms/step
Epoch 87/100
Epoch 77: val_loss did not improve from 0.03071
572/572 - 46s - loss: 0.0380 - accuracy: 0.9468 - val_loss: 0.0324 - val_accuracy: 0.9558 - 46s/epoch - 81ms/step
Epoch 78/100
Epoch 87: val_loss did not improve from 0.04331
580/580 - 47s - loss: 0.0566 - accuracy: 0.9194 - val_loss: 0.0440 - val_accuracy: 0.9348 - 47s/epoch - 81ms/step
Epoch 88/100
Epoch 78: val_loss improved from 0.03071 to 0.03060, saving model to /data/nn/python/us/stock/idx1/model/dow_257.h5
572/572 - 46s - loss: 0.0356 - accuracy: 0.9496 - val_loss: 0.0306 - val_accuracy: 0.9580 - 46s/epoch - 81ms/step
Epoch 79/100
Epoch 88: val_loss did not improve from 0.04331
580/580 - 47s - loss: 0.0536 - accuracy: 0.9221 - val_loss: 0.0446 - val_accuracy: 0.9340 - 47s/epoch - 81ms/step
Epoch 89/100
Epoch 79: val_loss did not improve from 0.03060
572/572 - 46s - loss: 0.0379 - accuracy: 0.9456 - val_loss: 0.0395 - val_accuracy: 0.9501 - 46s/epoch - 81ms/step
Epoch 80/100
Epoch 89: val_loss did not improve from 0.04331
580/580 - 47s - loss: 0.0528 - accuracy: 0.9264 - val_loss: 0.0444 - val_accuracy: 0.9391 - 47s/epoch - 81ms/step
Epoch 90/100
Epoch 80: val_loss improved from 0.03060 to 0.03018, saving model to /data/nn/python/us/stock/idx1/model/dow_257.h5
572/572 - 46s - loss: 0.0358 - accuracy: 0.9504 - val_loss: 0.0302 - val_accuracy: 0.9580 - 46s/epoch - 81ms/step
Epoch 81/100
Epoch 90: val_loss did not improve from 0.04331
580/580 - 47s - loss: 0.0732 - accuracy: 0.8934 - val_loss: 0.0714 - val_accuracy: 0.9007 - 47s/epoch - 81ms/step
Epoch 91/100
Epoch 81: val_loss improved from 0.03018 to 0.03003, saving model to /data/nn/python/us/stock/idx1/model/dow_257.h5
572/572 - 46s - loss: 0.0347 - accuracy: 0.9515 - val_loss: 0.0300 - val_accuracy: 0.9584 - 46s/epoch - 80ms/step
I have over 10,000 instances in training set covering the past five year period of US stock market. I have another 3000 for validation and 3000 for testing. The val_loss decreases over time and accuracy increases at the same time. But when I did testing, the accuracy is actually below 50%.
Does anyone have the same experience and can share some insights?
Thanks.
Related
I am trying to classify the severity of COVID XRay using 426 256x256 xray images and 4 classes present. However the validation accuracy doesnt improve at all. The validation loss also barely decreases from the start
This is the model I am using
from keras.models import Sequential
from keras.layers import Dense,Conv2D,MaxPooling2D,Dropout,Flatten
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras import regularizers
model=Sequential()
model.add(Conv2D(filters=64,kernel_size=(4,4),input_shape=image_shape,activation="relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(filters=128,kernel_size=(6,6),input_shape=image_shape,activation="relu"))
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(64,activation="relu"))
model.add(Dense(16,activation="relu"))
model.add(Dense(4,activation="softmax"))
model.compile(loss="categorical_crossentropy",optimizer="adam",metrics=["accuracy"])
These are the outputs I get
epochs = 20
batch_size = 8
model.fit(X_train, y_train, validation_data=(X_test, y_test),
epochs=epochs,
batch_size=batch_size
)
Epoch 1/20
27/27 [==============================] - 4s 143ms/step - loss: 0.1776 - accuracy: 0.9528 - val_loss: 3.7355 - val_accuracy: 0.2717
Epoch 2/20
27/27 [==============================] - 4s 142ms/step - loss: 0.1152 - accuracy: 0.9481 - val_loss: 4.0038 - val_accuracy: 0.2283
Epoch 3/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0875 - accuracy: 0.9858 - val_loss: 4.1756 - val_accuracy: 0.2391
Epoch 4/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0521 - accuracy: 0.9906 - val_loss: 4.1034 - val_accuracy: 0.2717
Epoch 5/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0496 - accuracy: 0.9858 - val_loss: 4.8433 - val_accuracy: 0.3152
Epoch 6/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0170 - accuracy: 0.9953 - val_loss: 5.6027 - val_accuracy: 0.3043
Epoch 7/20
27/27 [==============================] - 4s 142ms/step - loss: 0.2307 - accuracy: 0.9245 - val_loss: 4.2759 - val_accuracy: 0.3152
Epoch 8/20
27/27 [==============================] - 4s 142ms/step - loss: 0.6493 - accuracy: 0.7830 - val_loss: 3.8390 - val_accuracy: 0.3478
Epoch 9/20
27/27 [==============================] - 4s 142ms/step - loss: 0.2563 - accuracy: 0.9009 - val_loss: 5.0250 - val_accuracy: 0.2500
Epoch 10/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0286 - accuracy: 1.0000 - val_loss: 4.6475 - val_accuracy: 0.2391
Epoch 11/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0097 - accuracy: 1.0000 - val_loss: 5.2198 - val_accuracy: 0.2391
Epoch 12/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0037 - accuracy: 1.0000 - val_loss: 5.7914 - val_accuracy: 0.2500
Epoch 13/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0048 - accuracy: 1.0000 - val_loss: 5.4341 - val_accuracy: 0.2391
Epoch 14/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0044 - accuracy: 1.0000 - val_loss: 5.6364 - val_accuracy: 0.2391
Epoch 15/20
27/27 [==============================] - 4s 143ms/step - loss: 0.0019 - accuracy: 1.0000 - val_loss: 5.8504 - val_accuracy: 0.2391
Epoch 16/20
27/27 [==============================] - 4s 143ms/step - loss: 0.0013 - accuracy: 1.0000 - val_loss: 5.9604 - val_accuracy: 0.2500
Epoch 17/20
27/27 [==============================] - 4s 149ms/step - loss: 0.0023 - accuracy: 1.0000 - val_loss: 6.0851 - val_accuracy: 0.2717
Epoch 18/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0134 - accuracy: 0.9953 - val_loss: 4.9783 - val_accuracy: 0.2717
Epoch 19/20
27/27 [==============================] - 4s 141ms/step - loss: 0.0068 - accuracy: 1.0000 - val_loss: 5.7421 - val_accuracy: 0.2500
Epoch 20/20
27/27 [==============================] - 4s 142ms/step - loss: 0.0024 - accuracy: 1.0000 - val_loss: 5.8480 - val_accuracy: 0.2283
Any tips on how i can solve this or If i am doing something wrong?
I am training a model with the following code
model=Sequential()
model.add(Dense(100, activation='relu',input_shape=(n_cols,)))
model.add(Dense(100, activation='relu'))
model.add(Dense(2,activation='softmax'))
model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])
early_stopping_monitor = EarlyStopping(patience=3)
model.fit(X_train_np,target,validation_split=0.3, epochs=100, callbacks=[early_stopping_monitor])
This is designed to stop the training if the val_loss: parameter does not improve after 3 epochs. The result is shown below. My question is will the model stop with weights of epoch 8 or 7. Because the performance got bad in epoch 8 so it stopped. But the model went ahead by 1 epoch with a bad performing parameter as earlier one (epoch 7) was better. Do I need to retrain the model now with 7 epochs?
Train on 623 samples, validate on 268 samples
Epoch 1/100
623/623 [==============================] - 1s 1ms/step - loss: 4.0365 - accuracy: 0.5923 - val_loss: 1.2208 - val_accuracy: 0.6231
Epoch 2/100
623/623 [==============================] - 0s 114us/step - loss: 1.4412 - accuracy: 0.6356 - val_loss: 0.7193 - val_accuracy: 0.7015
Epoch 3/100
623/623 [==============================] - 0s 103us/step - loss: 1.4335 - accuracy: 0.6260 - val_loss: 1.3778 - val_accuracy: 0.7201
Epoch 4/100
623/623 [==============================] - 0s 106us/step - loss: 3.5732 - accuracy: 0.6324 - val_loss: 2.7310 - val_accuracy: 0.6194
Epoch 5/100
623/623 [==============================] - 0s 111us/step - loss: 1.3116 - accuracy: 0.6372 - val_loss: 0.5952 - val_accuracy: 0.7351
Epoch 6/100
623/623 [==============================] - 0s 98us/step - loss: 0.9357 - accuracy: 0.6645 - val_loss: 0.8047 - val_accuracy: 0.6828
Epoch 7/100
623/623 [==============================] - 0s 105us/step - loss: 0.7671 - accuracy: 0.6934 - val_loss: 0.9918 - val_accuracy: 0.6679
Epoch 8/100
623/623 [==============================] - 0s 126us/step - loss: 2.2968 - accuracy: 0.6629 - val_loss: 1.7789 - val_accuracy: 0.7425
Use restore_best_weights with monitor value set to target quantity. So, the best weights will be restored after training automatically.
early_stopping_monitor = EarlyStopping(patience=3,
monitor='val_loss', # assuming it's val_loss
restore_best_weights=True )
From docs:
restore_best_weights: whether to restore model weights from the epoch with the best value of the monitored quantity ('val_loss' here). If False, the model weights obtained at the last step of training are used (default False).
Docmentation link
All the code that I have placed is in TensorFlow 2.0
file path: Is a string that can have formatting options such as the epoch number. For example the following is a common filepath (weights.{epoch:02d}-{val_loss:.2f}.hdf5)
monitor: (typically it is‘val_loss’or ‘val_accuracy’)
mode: Should it be minimizing or maximizing the monitor value
(typically either ‘min’ or ‘max’)
save_best_only: If this is set to true then it will only save the
model for the current epoch, if it’s metric values, is better than
what has gone before. However, if you set save_best_only to
false it will save every model after each epoch (regardless of
whether that model was better than previous models or not).
Code
model=Sequential()
model.add(Dense(100, activation='relu',input_shape=(n_cols,)))
model.add(Dense(100, activation='relu'))
model.add(Dense(2,activation='softmax'))
model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])
fname = "weights.{epoch:02d}-{val_loss:.2f}.hdf5"
checkpoint = tf.keras.callbacks.ModelCheckpoint(fname, monitor="val_loss",mode="min", save_best_only=True, verbose=1)
model.fit(X_train_np,target,validation_split=0.3, epochs=100, callbacks=[checkpoint])
As the title clearly describes, the accuracy of my simple CNN model is not being affected by the hyper-parameters or even the existence of layers such as Dropout, and MaxPooling. I implemented the model using Keras. What could be the reason behind this odd situation? I added the regarding part of the code below:
input_dim = X_train.shape[1]
nb_classes = Y_train.shape[1]
model = Sequential()
model.add(Conv1D(filters=64, kernel_size=3, activation='relu', input_shape=(input_dim, 1)))
model.add(Dropout(0.5))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(40, activation='relu'))
model.add(Dense(nb_classes, activation='softmax'))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
p.s. The input data (X_train and X_test) contains vectors which were reproduced by Word2Vec. The output is binary.
Edit: You may find a sample training log below:
Sample training log:
Train on 3114 samples, validate on 347 samples
Epoch 1/10
- 1s - loss: 0.6917 - accuracy: 0.5363 - val_loss: 0.6901 - val_accuracy: 0.5476
Epoch 2/10
- 1s - loss: 0.6906 - accuracy: 0.5369 - val_loss: 0.6896 - val_accuracy: 0.5476
Epoch 3/10
- 1s - loss: 0.6908 - accuracy: 0.5369 - val_loss: 0.6895 - val_accuracy: 0.5476
Epoch 4/10
- 1s - loss: 0.6908 - accuracy: 0.5369 - val_loss: 0.6903 - val_accuracy: 0.5476
Epoch 5/10
- 1s - loss: 0.6908 - accuracy: 0.5369 - val_loss: 0.6899 - val_accuracy: 0.5476
Epoch 6/10
- 1s - loss: 0.6909 - accuracy: 0.5369 - val_loss: 0.6901 - val_accuracy: 0.5476
Epoch 7/10
- 1s - loss: 0.6905 - accuracy: 0.5369 - val_loss: 0.6896 - val_accuracy: 0.5476
Epoch 8/10
- 1s - loss: 0.6909 - accuracy: 0.5369 - val_loss: 0.6897 - val_accuracy: 0.5476
Epoch 9/10
- 1s - loss: 0.6905 - accuracy: 0.5369 - val_loss: 0.6892 - val_accuracy: 0.5476
Epoch 10/10
- 1s - loss: 0.6909 - accuracy: 0.5369 - val_loss: 0.6900 - val_accuracy: 0.5476
First you need to change the last layer to
model.add(Dense(1, activation='sigmoid'))
You also need to change the loss function to
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
I assume that you have multi-class classification, right?
Then your loss is not appropriate: you should use 'categorical_crossentropy' not 'mean_squared_error'.
Also, try adding several Conv+Drop+MaxPool (3 sets) in order to clearly verify the robustness of your network.
I want to build a model using Keras to predict the price of the natural gas.
The dataset contains the price for the gas daily and monthly since 1997 and it is available Here.
The following graph shows the prices during a sequence of days. X is days and Y is the price.
I have tried LSTM with 4,50,100 cell in hidden layer but the accuracy still not was bad and the model failed to predict future price.
I have added another two hidden layers (full connected) with 100 and 128 cell but it did not work too.
This is the model and the result form training process:
num_units = 100
activation_function = 'sigmoid'
optimizer = 'adam'
loss_function = 'mean_squared_error'
batch_size = 5
num_epochs = 10
log_file_name = f"{SEQ_LEN}-SEQ-{1}-PRED-{int(time.time())}"
# Initialize the model (of a Sequential type)
model = Sequential()
# Adding the input layer and the LSTM layer
model.add(LSTM(units = num_units, activation = activation_function,input_shape=(None, 1)))
# Adding the output layer
model.add(Dense(units = 1))
# Compiling the RNN
model.compile(optimizer = optimizer, loss = loss_function, metrics=['accuracy'])
# Using the training set to train the model
history = model.fit(train_x, train_y, batch_size = batch_size, epochs =num_epochs,validation_data=(test_x, test_y))
and the output is :
Train on 4362 samples, validate on 1082 samples
Epoch 1/10
4362/4362 [==============================] - 11s 3ms/step - loss: 0.0057 - acc: 2.2925e-04 - val_loss: 0.0016 - val_acc: 0.0018
Epoch 2/10
4362/4362 [==============================] - 9s 2ms/step - loss: 6.2463e-04 - acc: 4.5851e-04 - val_loss: 0.0013 - val_acc: 0.0018
Epoch 3/10
4362/4362 [==============================] - 9s 2ms/step - loss: 6.1073e-04 - acc: 2.2925e-04 - val_loss: 0.0014 - val_acc: 0.0018
Epoch 4/10
4362/4362 [==============================] - 8s 2ms/step - loss: 5.4403e-04 - acc: 4.5851e-04 - val_loss: 0.0014 - val_acc: 0.0018
Epoch 5/10
4362/4362 [==============================] - 7s 2ms/step - loss: 5.4765e-04 - acc: 4.5851e-04 - val_loss: 0.0012 - val_acc: 0.0018
Epoch 6/10
4362/4362 [==============================] - 8s 2ms/step - loss: 5.1991e-04 - acc: 4.5851e-04 - val_loss: 0.0013 - val_acc: 0.0018
Epoch 7/10
4362/4362 [==============================] - 7s 2ms/step - loss: 5.7324e-04 - acc: 2.2925e-04 - val_loss: 0.0011 - val_acc: 0.0018
Epoch 8/10
4362/4362 [==============================] - 7s 2ms/step - loss: 4.4248e-04 - acc: 4.5851e-04 - val_loss: 0.0011 - val_acc: 0.0018
Epoch 9/10
4362/4362 [==============================] - 7s 2ms/step - loss: 4.3868e-04 - acc: 4.5851e-04 - val_loss: 0.0011 - val_acc: 0.0018
Epoch 10/10
4362/4362 [==============================] - 7s 2ms/step - loss: 4.6654e-04 - acc: 4.5851e-04 - val_loss: 0.0011 - val_acc: 0.0018
How to know the number of layers and cells for problem like this? Anyone can suggest a netwrok structure that can solve this problem?
I have 3D CNN U-net architecture to solve segmentation problem. I am using Adam optimisation together with binary cross entropy and the metric is "accuracy". I try to understand why it does not improve.
Train on 2774 samples, validate on 694 samples
Epoch 1/20
2774/2774 [==============================] - 166s 60ms/step - loss: 0.5189 - acc: 0.7928 - val_loss: 0.5456 - val_acc: 0.7674
Epoch 00001: val_loss improved from inf to 0.54555, saving model to model-tgs-salt-1.h5
Epoch 2/20
2774/2774 [==============================] - 170s 61ms/step - loss: 0.5170 - acc: 0.7928 - val_loss: 0.5485 - val_acc: 0.7674
Epoch 00002: val_loss did not improve from 0.54555
Epoch 3/20
2774/2774 [==============================] - 169s 61ms/step - loss: 0.5119 - acc: 0.7928 - val_loss: 0.5455 - val_acc: 0.7674
Epoch 00003: val_loss improved from 0.54555 to 0.54549, saving model to model-tgs-salt-1.h5
Epoch 4/20
2774/2774 [==============================] - 170s 61ms/step - loss: 0.5117 - acc: 0.7928 - val_loss: 0.5715 - val_acc: 0.7674
Epoch 00004: val_loss did not improve from 0.54549
Epoch 5/20
2774/2774 [==============================] - 169s 61ms/step - loss: 0.5126 - acc: 0.7928 - val_loss: 0.5566 - val_acc: 0.7674
Epoch 00005: val_loss did not improve from 0.54549
Epoch 6/20
2774/2774 [==============================] - 169s 61ms/step - loss: 0.5138 - acc: 0.7928 - val_loss: 0.5503 - val_acc: 0.7674
Epoch 00006: val_loss did not improve from 0.54549
Epoch 7/20
2774/2774 [==============================] - 170s 61ms/step - loss: 0.5103 - acc: 0.7928 - val_loss: 0.5444 - val_acc: 0.7674
Epoch 00007: val_loss improved from 0.54549 to 0.54436, saving model to model-tgs-salt-1.h5
Epoch 8/20
2774/2774 [==============================] - 169s 61ms/step - loss: 0.5137 - acc: 0.7928 - val_loss: 0.5454 - val_acc: 0.7674
If you use batch size in your network. let's try to increase that. I think it could be useful in speed of train.