I am a newbie in artificial intelligence and I have a task to do.
I want to create a Model in order to predict the outcome of a boolean AND operator.
So my Idea was to create an Input-Tensor:
[0,0,0] => false and false = false
[0,0.9999,0] false and true = false
[0.9999,0,0] true and false = false
[0.9999,0.9999,0.9999] true and true = true
Below is the python code which should predict the outcome.... but the prediction is very bad.
I want to understand how can I continue.
Can you help me please?
Thx
import tensorflow as tf
import datetime, os
x_train = [ [0,0], [0,0],[0,0.9999], [0,0.9999] ]
y_train = [0,0,0,0.9999]
x_test = x_train
y_test = y_train
mymodel = tf.keras.models.Sequential()
mymodel.add(tf.keras.Input(shape=(4,2)))
mymodel.add(tf.keras.layers.Dense(1, activation='relu'))
mymodel.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
mymodel.fit(x=x_train, y=y_train, epochs=300, validation_data=(x_test, y_test))
print(mymodel.predict([[0,0],[0,0.9999],[0.9999,0],[0.9999,0.9999]]))
Related
This question already has answers here:
How to save/restore a model after training?
(29 answers)
Closed 6 months ago.
I am new to Neural networks and I have successfully trained an RNN but it takes a while to train the data. It would not be feasible for me to train the data every time I want to make a prediction. So the question is, how do I make the training data persistent in that the RNN does not have to train every time every time I make a prediction?
This is the code I am using...
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense
from keras.utils.data_utils import pad_sequences
class Predict:
TrainingData = None
XScale = None
def __init__(self,PredData,TrainingData):
PreparedData = self.PrepareData(PredData,TrainingData)
#set the training data
self.Train(PreparedData)
#split all of the training data into their respective variables
X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(self.XScale, self.Y, test_size = 0.6)
X_val, X_test, Y_val, Y_test = train_test_split(X_val_and_test, Y_val_and_test, test_size = 0.6)
#define the model
self.DefineModel(X_train, Y_train, X_val, Y_val)
#do the prediction
return self.predict(PredData)
def PrepareData(self, PredData,TrainingData):
LNumber = 0
for I in TrainingData:
if(len(I) > LNumber):
LNumber = len(I)
PadData = pad_sequences(TrainingData, maxlen = LNumber, padding = 'post', truncating = 'post')
return PadData
def Train(self,TrainData):
min_max_scaler = preprocessing.MinMaxScaler()
self.X = TrainData[0:10]
self.Y = TrainData[-10:]
self.XScale = min_max_scaler.fit_transform(self.X)
def DefineModel(self,X_train, T_train, X_val, Y_val):
self.model = Sequential([
Dense(32, activation = 'relu', input_shape = (10,)),
Dense(32, activation = 'relu'),
Dense(1, activation = 'sigmoid'),
])
self.model.compile( optimizer = 'sgd',
loss = 'binary_crossentropy',
metrics = ['accuracy']
)
self.model.fit( X_train, Y_train,
batch_size = 32, epochs = 100,
validation_data = (X_val, Y_val))
def predict(self,PredData):
Self.Prediction = model.predict(PredData)
As suggested in the comments you can use SavedModel to save your entire architecture + weights. I suggest you having a look at this page on how to Save and load Keras models.
Basically you just need to save like this:
self.model.save('path/to/location')
And to restore later on:
from tensorflow import keras
model = keras.models.load_model('path/to/location')
Also if your training is quite long, you can also think about saving checkpoints of the best model so far, using the callback tf.keras.callbacks.ModelCheckpoint. You create the callback and add it to your fit:
checkpoint_filepath = '/tmp/checkpoint'
model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_filepath,
save_weights_only=True,
monitor='val_accuracy',
mode='max',
save_best_only=True)
# Model weights are saved at the end of every epoch, if it's the best seen so far.
model.fit(X_train, Y_train, batch_size=32, epochs=100,
validation_data=(X_val, Y_val), callbacks=[model_checkpoint_callback])
If your train crashes or got interrupted you can re-load the best weights like this:
# The model weights (that are considered the best) are loaded into the model.
model.load_weights(checkpoint_filepath)
Note that if you are re-starting your script you will have to re-create the model object: load_weights does not re-create the architecture for you like it does load_model. But it is just a matter, in your case, of doing first:
self.model = Sequential([
Dense(32, activation = 'relu', input_shape = (10,)),
Dense(32, activation = 'relu'),
Dense(1, activation = 'sigmoid'),
])
I built a simple categorical time series model to predict the next number of a random sequence, but the accuracy hardly moved even I trained it for 10000 epochs. The validation loss started to take off after a few hundred epochs. Could anyone make suggestions for improvement? Here's the model:
import os
import sys
import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM
DEVICE = 'CPU'
if DEVICE == 'CPU':
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
else:
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
print(tf.test.gpu_device_name())
TOTAL_CATALOG=4
POSSIBLE_OUTCOME_COL=4
LOOK_BACK_WINDOW=1
TRAINING_DATA_RATIO=0.8
TRAINING_EPOCHS=10000
sys.path.insert(0, '/DataScience/MyModules')
from m6data import getDrawData, series_to_supervised, Split_data, get_all_categories
def get_all_categories_local(last_combination):
all_category = np.arange(1, last_combination+1)
return all_category.reshape(1,all_category.shape[0])
All_categories=get_all_categories_local(TOTAL_CATALOG)
data_sequence = [1,1,2,4,2,3,1,2,3,3,4,1,2,3,4,2,2,3,1,3]
raw_df = pd.DataFrame(data_sequence, columns=['NE'])
values = raw_df.values
# 05-Apr-2022: One-Hot Encoding
oh_encoder = OneHotEncoder(categories=All_categories, sparse=False)
encoded_input = oh_encoder.fit_transform(values)
FEATURES = encoded_input.shape[1]
POSSIBLE_OUTCOME_COL = FEATURES
draw_reframe = series_to_supervised(encoded_input, LOOK_BACK_WINDOW,1)
train, test = Split_data(draw_reframe, TRAINING_DATA_RATIO)
# Total input = all possible One-Hot Encoding outcome * number of look-back samples.
ALL_INPUT = POSSIBLE_OUTCOME_COL * LOOK_BACK_WINDOW
# split into input and outputs
train_X, train_y = train.iloc[:,:ALL_INPUT], train.iloc[:,ALL_INPUT:]
test_X, test_y = test.iloc[:,:ALL_INPUT], test.iloc[:,ALL_INPUT:]
train_X = train_X.values.reshape((train_X.shape[0], LOOK_BACK_WINDOW , FEATURES))
test_X = test_X.values.reshape((test_X.shape[0], LOOK_BACK_WINDOW, FEATURES))
print(train_X.shape, train_y.shape)
print(test_X.shape, test_y.shape)
def create_model():
model = Sequential()
model.add(LSTM(10,
return_sequences=False,
input_shape=(train_X.shape[1], train_X.shape[2]),
activation='relu'
)
)
#model.add(LSTM(20))
model.add(Dense(units=train_y.shape[1], activation='softmax'))
model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=0.00005),
loss = 'categorical_crossentropy',
metrics=['accuracy'])
return model
model=create_model()
history = model.fit(
train_X, train_y,
epochs=TRAINING_EPOCHS,
batch_size=8,
validation_data=(test_X, test_y),
verbose=1,
)
Here are the plots of accuracies and losses (red=training, blue=validation).
Accuracies
Losses
Thank you in advance for any suggestions.
Update (13-Jun-2022)
I changed my model to the following
def create_model():
model = Sequential()
model.add(LSTM(50,
return_sequences=True,
input_shape=(train_X.shape[1], train_X.shape[2]),
activation='relu'
)
)
model.add(LSTM(units=1000, kernel_regularizer=regularizers.l1(0.05), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=1000, kernel_regularizer=regularizers.l1(0.05), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=1000, kernel_regularizer=regularizers.l1(0.05), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=1000, kernel_regularizer=regularizers.l1(0.05), activation='relu'))
model.add(Dropout(0.3))
model.add(BatchNormalization())
model.add(Dense(1000))
model.add(Dense(units=train_y.shape[1], activation='softmax'))
model.compile(optimizer = tf.keras.optimizers.SGD(learning_rate=1e-2, nesterov=True),
#tf.keras.optimizers.Adam(learning_rate=0.001),
loss = 'categorical_crossentropy',
metrics=['accuracy'])
return model
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
patience=20,min_lr=1e-10)
early_stop = EarlyStopping(monitor='loss', patience=100)
history = model.fit(
train_X, train_y,
epochs=TRAINING_EPOCHS,
batch_size=16,
validation_split=0.1,
validation_data=(test_X, test_y),
verbose=1,
shuffle=False,
callbacks=([reduce_lr], [early_stop])
Accuracy was bouncing around and Val_accuracy was zero all the way. The loss and val_loss were almost the same and dropping together.
Can anyone advise what I can do in this scenario?
I'm training the VGG16 block5 with code below over 100 iterations/epochs, adding up two dense and dropout layer after the output of block5.
seq = Sequential()
vgg_base = VGG16(weights='imagenet', input_shape=(224, 224, 3),include_top=False, pooling='avg')
seq.add(vgg_base)
for layer in vgg_base.layers:
layer.trainable = False
for layer in vgg_base.layers[:14]:
layer.trainable = False
for layer in vgg_base.layers[15:]:
layer.trainable = True
regularizer = tf.keras.regularizers.l2(1e-2)
for layer in vgg_base.layers[15:]:
for attr in ['kernel_regularizer']:
if hasattr(layer, attr):
setattr(layer, attr, regularizer)
vgg_base.summary()
#seq.add(Flatten())
input_a = Input(shape=(224, 224, 3))
out_a = seq(input_a)
hidden1 = Dense(128, activation='relu', name='dense_1', kernel_regularizer=tf.keras.regularizers.l2(0.001),
activity_regularizer=tf.keras.regularizers.l2(0.001))(out_a)
hidden_drp1 = Dropout(0.5)(hidden1)
hidden2 = Dense(32, activation='relu', name='dense_2',kernel_regularizer=tf.keras.regularizers.l2(0.001),
activity_regularizer=tf.keras.regularizers.l2(0.001))(hidden_drp1)
hidden_drp2 = Dropout(0.2)(hidden2)
out = Dense(1, activation='sigmoid', name='dense_3')(hidden_drp2)
model = Model(input_a, out)
Training settings:
if os.path.exists(filepath):
print('weights found... loading...')
model.load_weights(filepath)
train_datagen = ImageDataGenerator(rescale=1.0/255)
#,validation_split=0.4)
# validation_split=0.3) # set validation split
train_path = 'C:/fromtrainv4/'
train_set = train_datagen.flow_from_directory(
train_path,
target_size=(224,224),
shuffle=True,
color_mode='rgb',
batch_size=100,
class_mode='binary')
optim = Adam(lr=0.001)
loss = 'binary_crossentropy'
metrics = ['binary_accuracy', 'acc']
model.compile(loss=loss,
optimizer=optim,
metrics=metrics)
checkpoint = ModelCheckpoint(filepath,
monitor='loss',
verbose=1,
save_weights_only=False,
save_best_only=True,
mode='min')
my_callback = [tf.keras.callbacks.EarlyStopping(monitor='loss', patience=10),
checkpoint,tf.keras.callbacks.TensorBoard(log_dir='C:\\traning_logs\\')]
step_size_train = train_set.n//train_set.batch_size
history = model.fit_generator(train_set,
steps_per_epoch=step_size_train,
shuffle=False,
epochs=100,
callbacks=my_callback,
class_weight='balanced')
I have made sure that the data has same scaling as image_generator and everything else.
I load the model using code below:
from tensorflow.keras.models import load_model
model = load_model('C:/models/model.h5')
System Specs:
Processor: Intell Xeon
Ram: 16GB
GPU: NVIDIA RTX 2080 Super
Memory: M2. 256GB
Interpreter: Spyder Python 3.7.1
tensorflow version: 1.14.0
keras version: 2.3.1
Can someone guide me what I might be doing wrong. Please let me know if more information regarding this is required.
I would guess that your model is doing what is called "overfitting".
In short, by doing 100 train iterations of the same dataset, it kind of memorized by heart your train dataset and thus have a good result when tested on it. However it doesn't yet find a general answer to your problem which lead to lower results when testing with a new dataset with different inputs.
I am wondering if it is an issue (possible data leakage?) when implementing leave one out cross validation by hand if the model is fit each iteration after testing on each fold? It seems like if the model is trained on all data except for "X" and after testing on "X" the model is trained on all data other than "Y" and tested on "Y" it has seen "Y" on the first iteration. Is this actually a problem, and does my implementation of LOOCV by hand appear to be correct?
Thanks for your time!
i = 0
j = 0
for i in range(0, 41):
X_copy = X_orig[(i):(i+1)] #Slice the ith element from the numpy array
y_copy = y_orig[(i):(i+1)]
X_model = X_orig
y_model = y_orig
X_model = np.delete(X_model, i, axis = 0)
y_model = np.delete(y_model, i, axis = 0)
model.fit(X_model, y_model, epochs=115, batch_size=28, verbose = 0) #verbose = 0 removes learning info
prediction = model.predict(X_copy)
prediction[prediction>=0.5] = 1
prediction[prediction<0.5] = 0
print(prediction, y_copy)
if np.array_equal(y_copy, prediction):
j = j + 1
#print(y_copy, prediction)
if np.not_equal:
#print(y_copy, prediction)
pass
print(j/41) #For 41 samples in dataset
Why don't you use this?
from sklearn.model_selection import LeaveOneOut
loo = LeaveOneOut()
model =...
test_fold_predictions = []
for train_index, test_index in loo.split(X):
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = y[train_index], y[test_index]
model.fit(X_train, y_train)
test_fold_predictions.append(model.predict(X_test))
EDIT
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
model = Sequential()
model.add(Dense(5000, activation='relu', input_dim=X_train.shape[1]))
model.add(Dropout(0.1))
model.add(Dense(600, activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(y_train.shape[1], activation='sigmoid'))
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy',
optimizer=sgd)
from sklearn.model_selection import LeaveOneOut
loo = LeaveOneOut()
test_fold_predictions = []
for train_index, test_index in loo.split(X):
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = y[train_index], y[test_index]
model.fit(X_train, y_train, epochs=5, batch_size=2000)
test_fold_predictions.append(model.predict(X_test))
How do I set training to False for the validation_data used in Keras fit_generator? I have Dropout layers in my model and I want training to be True during training, and False during validation and testing.
Keras automatically sets learning_phase to False when doing validation. There is nothing extra that you need to do.
Dropout nodes automatically check whether they are in training mode.
https://github.com/keras-team/keras/blob/master/keras/layers/core.py#L126
If you want to verify that Keras automatically changes the learning mode flag you can execute the code bellow. It adds a Lambda layer that adds a print tensor to the graph that outputs a different message in each case.
from tensorflow import keras
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
def inspect(x):
xp = K.in_train_phase(K.print_tensor(x, message='train x:'),
K.print_tensor(x, message='test x:'))
return xp
def make_model():
inp = Input(shape=(4,))
h1 = Dense(2)(inp)
h1p = Lambda(inspect)(h1)
out = Dense(1)(h1p)
model = Model(inp, out)
model.compile('adam', 'mse')
return model
model = make_model()
model.summary()
import numpy as np
X_train = np.random.rand(1, 4)
Y_train = np.random.rand(1, 1)
X_test = np.random.rand(1, 4)
Y_test = np.random.rand(1, 1)
model.fit(X_train, Y_train, validation_data=(X_test, Y_test))
I added an is_training argument to my data generator function. If it's True, I set the Keras learning phase to 1, else to 0 (see docs at keras.io/backend/):
if is_training:
K.set_learning_phase( 1 )
else:
K.set_learning_phase( 0 )
So, for my training data generator, I use is_training = True, and for my validation data generator, I use is_training = False.