I have trained a model on colab using artifical neural network (keras) and in the end I want to save it but getting error. I have tried pydrive method also. I have all the required library previously when I run it on local computer it worked.
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.models import model_from_json
model = Sequential() # create model
model.add(Dense(6, input_dim = 8, activation = 'relu'))
model.add(Dense(6, activation = 'relu')) # hidden layer
model.add(Dense(1, activation = 'sigmoid')) # output layer
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.fit(X_train, y_train, epochs=100, batch_size=10)
def build_classifier():
model = Sequential() # create model
model.add(Dense(6, input_dim = 8, activation = 'relu'))
model.add(Dense(6, activation = 'relu')) # hidden layer
model.add(Dense(1, activation = 'sigmoid')) # output layer
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
return model
model = KerasClassifier(build_fn = build_classifier, epochs=100, batch_size=32)
accuracies = cross_val_score(estimator = model,X = X_train, y = y_train, cv = 10, n_jobs = -1)
model.save("model.h5")
but getting error 'KerasClassifier' object has no attribute 'save'. does google colab required different method to save model?
'KerasClassifier' object has no attribute 'save'
Error occur because I tried to save KerasClassifier(Cross-validation function) Since the The purpose of cross-validation is model checking, not model building. So after changing parameter model will be saved.Thanks #MatiasValdenegro for insight.
Better explanation available at
https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation
Related
This question already has answers here:
How to save/restore a model after training?
(29 answers)
Closed 6 months ago.
I am new to Neural networks and I have successfully trained an RNN but it takes a while to train the data. It would not be feasible for me to train the data every time I want to make a prediction. So the question is, how do I make the training data persistent in that the RNN does not have to train every time every time I make a prediction?
This is the code I am using...
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense
from keras.utils.data_utils import pad_sequences
class Predict:
TrainingData = None
XScale = None
def __init__(self,PredData,TrainingData):
PreparedData = self.PrepareData(PredData,TrainingData)
#set the training data
self.Train(PreparedData)
#split all of the training data into their respective variables
X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(self.XScale, self.Y, test_size = 0.6)
X_val, X_test, Y_val, Y_test = train_test_split(X_val_and_test, Y_val_and_test, test_size = 0.6)
#define the model
self.DefineModel(X_train, Y_train, X_val, Y_val)
#do the prediction
return self.predict(PredData)
def PrepareData(self, PredData,TrainingData):
LNumber = 0
for I in TrainingData:
if(len(I) > LNumber):
LNumber = len(I)
PadData = pad_sequences(TrainingData, maxlen = LNumber, padding = 'post', truncating = 'post')
return PadData
def Train(self,TrainData):
min_max_scaler = preprocessing.MinMaxScaler()
self.X = TrainData[0:10]
self.Y = TrainData[-10:]
self.XScale = min_max_scaler.fit_transform(self.X)
def DefineModel(self,X_train, T_train, X_val, Y_val):
self.model = Sequential([
Dense(32, activation = 'relu', input_shape = (10,)),
Dense(32, activation = 'relu'),
Dense(1, activation = 'sigmoid'),
])
self.model.compile( optimizer = 'sgd',
loss = 'binary_crossentropy',
metrics = ['accuracy']
)
self.model.fit( X_train, Y_train,
batch_size = 32, epochs = 100,
validation_data = (X_val, Y_val))
def predict(self,PredData):
Self.Prediction = model.predict(PredData)
As suggested in the comments you can use SavedModel to save your entire architecture + weights. I suggest you having a look at this page on how to Save and load Keras models.
Basically you just need to save like this:
self.model.save('path/to/location')
And to restore later on:
from tensorflow import keras
model = keras.models.load_model('path/to/location')
Also if your training is quite long, you can also think about saving checkpoints of the best model so far, using the callback tf.keras.callbacks.ModelCheckpoint. You create the callback and add it to your fit:
checkpoint_filepath = '/tmp/checkpoint'
model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_filepath,
save_weights_only=True,
monitor='val_accuracy',
mode='max',
save_best_only=True)
# Model weights are saved at the end of every epoch, if it's the best seen so far.
model.fit(X_train, Y_train, batch_size=32, epochs=100,
validation_data=(X_val, Y_val), callbacks=[model_checkpoint_callback])
If your train crashes or got interrupted you can re-load the best weights like this:
# The model weights (that are considered the best) are loaded into the model.
model.load_weights(checkpoint_filepath)
Note that if you are re-starting your script you will have to re-create the model object: load_weights does not re-create the architecture for you like it does load_model. But it is just a matter, in your case, of doing first:
self.model = Sequential([
Dense(32, activation = 'relu', input_shape = (10,)),
Dense(32, activation = 'relu'),
Dense(1, activation = 'sigmoid'),
])
I want to try to test some hyperparameters, thats i want to use the GridSearchCV, because it seems like thats the way to do it.
But i also want to use the validation split. To use Callsbacks like EarlyStopping or/and ReduceLROnPlateau. So my question is:
How do i implement GridSearchCV + validation_split correctly that none of the data in validation split is using for training and the whole training set is used to train my model?
Afaik GridSearchCV split again my remaining train data (which is 1-validation_split) and split it again? I get kinda high accuracy and im thinking that i dont split the data correctly
model = KerasClassifier(build_fn=create_model,verbose=2, validation_split=0.1)
optimizers = ['rmsprop', 'adam']
init = ['glorot_uniform',
#'normal',
'uniform',
'he_normal',
#'lecun_normal',
#'he_uniform'
]
epochs = [3] #5,8,10,30
batches = [64] #32,64
param_grid = dict(optimizer=optimizers, epochs=epochs, batch_size=batches, init=init)
grid = GridSearchCV(estimator=model, param_grid=param_grid)
grid_result = grid.fit(X_train, Y_train)
You can use your self-defined validation data by passing an extra argument to the grid.fit() function that is validation_data=(X_test, Y_test). The documentation, states that grid.fit() function accepts all valid arguments that can be passed to the actual model.fit() function of the default Keras model. Therefore, you can pass the validation data through the grid.fit() function. You may also pass the callback functions there.
I am adding a working code below (applied on MNIST-digit dataset). Notice how I added the validation data on grid.fit() and removed the 'validation_split':
import tensorflow as tf
import numpy as np
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
Y_train = to_categorical(Y_train, 10)
Y_test = to_categorical(Y_test, 10)
X_train = np.expand_dims(X_train, 3)
X_test = np.expand_dims(X_test, 3)
def create_model(optimizer, init):
model = tf.keras.Sequential([
tf.keras.layers.Convolution2D(32, 3, input_shape=(28, 28, 1),
activation='relu', kernel_initializer=init),
tf.keras.layers.Convolution2D(32, 3, activation='relu',
kernel_initializer=init),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(12, activation='relu',
kernel_initializer=init),
tf.keras.layers.Dense(10, activation='softmax',
kernel_initializer=init),
])
model.compile(loss='categorical_crossentropy',
optimizer=optimizer, metrics=['accuracy'])
return model
model = KerasClassifier(build_fn=create_model, verbose=2,)
optimizers = ['rmsprop', 'adam']
init = ['glorot_uniform',
#'normal',
'uniform',
'he_normal',
#'lecun_normal',
#'he_uniform'
]
epochs = [4,]
batches = [32, 64]
param_grid = dict(optimizer=optimizers, nb_epoch=epochs,
batch_size=batches, init=init)
grid = GridSearchCV(estimator=model, param_grid=param_grid)
grid_result = grid.fit(X_train, Y_train, validation_data=(X_test, Y_test))
Hope this helps. Thanks.
I am new in Keras. I want to implement a layer where not all the weights will update. For example, in the following code, I want the dilation layer will update in a way that some center weights are never updated. For say, the shape of each feature matrix (out of 1024) in the dilation layer is 448, 448 and a block of 8x8 at the center of all feature matrices will never be updated, i.e. the 8x8 block is a (non-trainable) mask to the feature matrices.
input_layer=Input(shape=(896,896,3))
new_layer = Conv2D(32, kernel_size=(3,3), padding="same", activation='relu', kernel_initializer='he_normal')(input_layer)
new_layer = MaxPooling2D(pool_size=(2, 2), strides=(2,2), padding='same', data_format=None)(new_layer)
new_layer = Conv2D(64, kernel_size=(3,3), padding='same', activation='relu', kernel_initializer='he_normal')(new_layer)
new_layer = Conv2D(1024, kernel_size=(7,7), dilation_rate=8, padding="same", activation='relu', kernel_initializer='he_normal', name='dialation')(new_layer)
new_layer = Conv2D(32, kernel_size=(1,1), padding="same", activation='relu', kernel_initializer='he_normal')(new_layer)
new_layer = Conv2D(32, kernel_size=(1,1), padding="same", activation='relu', kernel_initializer='he_normal')(new_layer)
model = Model(input_layer, new_layer)
I was trying with the Keras's custom layer [link], but it was difficult for me to understand. Anyone would please help.
UPDATE:
I added the following figure for a better understanding. The dilation layer contains 1024 features. I want the middle region of each feature to be non-trainable (static).
Use this mask for both cases:
mask = np.zeros((1,448,448,1))
mask[:,220:228,220:228] = 1
Replacing part of the feature
If you replace part of the feature with constant values, this means the feature will be static, but it will still participate in backpropagation (because weights will still be multiplied and summed for this part of the image and there is a connection)
constant = 0 (will annulate kernel, but not bias)
def replace(x):
return x*(1-mask) + constant*mask
#before the dilation layer
new_layer=Lambda(replace)(new_layer)
Keeping the feature value, but stopping backpropagation
Here, the weights of the dilation layer and further will be updated normally, but the weights before the dilation layer will not receive the influence of the central region.
def stopBackprop(x):
stopped=K.stop_gradients(x)
return x*(1-mask) + stopped*mask
#before the dilation layer
new_layer=Lambda(stopBackprop)(new_layer)
I am working on clustering weights and then freezing specific clusters and train the network.
I am trying to freeze the specific weights in this network using the above example. But I'm not sure how to set the shape of the mask and custom layer in run_certain_weights().
This is the code I'm using:
from keras.layers import Dense, Flatten, Lambda
from keras.utils import to_categorical
from keras.models import Sequential, load_model
from keras.datasets import mnist
from keras.losses import categorical_crossentropy
from keras.backend import stop_gradient
import numpy as np
def load_data():
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
y_train = to_categorical(y_train, num_classes=10)
y_test = to_categorical(y_test, num_classes=10)
return x_train, y_train, x_test, y_test
def run():
x_train, y_train, x_test, y_test = load_data()
model=Sequential(Flatten(input_shape=(28, 28)))
layer = Dense(300, name='dense1', activation='relu')
layer.trainable=True
model.add(layer)
layer2 = Dense(100, name='dense2', activation='relu')
layer2.trainable=False
model.add(layer2)
layer3 = Dense(10, name='dense3', activation='softmax')
model.add(layer3)
model.compile(loss=categorical_crossentropy, optimizer='Adam',metrics ['accuracy'])
print(model.summary())
print("x_train.shape():",x_train.shape)
print("y_train.shape()",y_train.shape)
model.fit(x_train, y_train, epochs=5, verbose=2)
print(model.evaluate(x_test, y_test))
return model
def stopBackprop(x):
stopped=stop_gradient(x)
return x*(1-mask) + stopped*mask
def run_certain_weights():
x_train, y_train, x_test, y_test = load_data()
model=Sequential(Flatten(input_shape=(28, 28)))
mask = np.zeros((300,))
print(mask.shape)
mask[220:228,] = 1
layer = Dense(300, name='dense1', activation='relu')
layer.trainable=False
model.add(layer)
#before the dense2 layer
new_layer=Lambda(stopBackprop)(layer)
model.add(new_layer)
layer2 = Dense(300, name='dense2', activation='relu')
layer2.trainable=True
model.add(layer2)
layer3 = Dense(10, name='dense3', activation='softmax')
model.add(layer3)
model.compile(loss=categorical_crossentropy, optimizer='Adam',metrics = ['accuracy'])
print(model.summary())
print("x_train.shape():",x_train.shape)
print("y_train.shape()",y_train.shape)
model.fit(x_train, y_train, epochs=5, verbose=2)
print(model.evaluate(x_test, y_test))
return model
def freeze(model):
x_train, y_train, x_test, y_test = load_data()
name = 'dense2'
weightsAndBias = model.get_layer(name=name).get_weights()
# freeze the weights of this layer
model.get_layer(name=name).trainable = False
# record the weights before retrain
weights_before = weightsAndBias[0]
# retrain
print("x_train.shape():",x_train.shape)
print("y_train.shape()",y_train.shape)
model.fit(x_train, y_train, verbose=2, epochs=1)
weights_after = model.get_layer(name=name).get_weights()[0]
if (weights_before == weights_after).all():
print('the weights did not change!!!')
else:
print('the weights changed!!!!')
if __name__ == '__main__':
model = run()
freeze(model)
model = run_certain_weights()
freeze(model)
I am building a model based on FFNN (Feed Forward Neural Network) using Keras.
I built a first version:
def mlp0(input_dim, loss):
model = Sequential()
model.add(Dropout(0.5, input_shape=(input_dim,)))
model.add(Dense(512, activation='sigmoid'))
model.add(Dense(1, activation='relu'))
model.compile(loss=loss, optimizer=Adagrad())
return model
This gives me very good results in k-fold cross-validation, but when I predict on validation set, the performance is bad.
So I tried another version.
def mlp1(input_dim, loss):
inputs = keras.Input(shape=(input_dim,))
x = keras.layers.Dropout(0.5)(inputs)
x = keras.layers.Dense(512, activation='sigmoid')(x)
outputs = keras.layers.Dense(1, activation='relu')(x)
model = keras.Model(inputs, outputs)
model.compile(loss=loss, optimizer=Adagrad())
return model
This second model gives worse results on cross-validation but the results are compatible with the results on the validation set.
To my eyes, they are identical models built in different ways, but for some reason they give me different answers. What am I doing wrong?
Edit:
These models behave the same way:
def mlp0(input_dim, loss):
model = Sequential()
model.add(Dense(512, activation='sigmoid', input_shape=(input_dim,), kernel_regularizer=regularizers.l2(0.01)))
model.add(Dense(1, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
model.compile(loss=loss, optimizer=Adam())
return model
import keras
from keras import regularizers
def mlp1(input_dim, loss):
inputs = keras.Input(shape=(input_dim,))
x = keras.layers.Dense(512, activation='sigmoid', kernel_regularizer=regularizers.l2(0.01))(inputs)
outputs = keras.layers.Dense(1, activation='relu', kernel_regularizer=regularizers.l2(0.01))(x)
model = keras.Model(inputs, outputs)
model.compile(loss=loss, optimizer=Adam())
return model
These make me think there is a catch in prediction phase with the dropout
I am using ANN for Multiclass Classification(12 classes) in Python. However i am getting errors. Here is the code snippet:
import keras
from keras.models import Sequential
from keras.layers import Dense
# Initialising the ANN
# Initialising the ANN
classifier = Sequential()
# Adding the input layer and the first hidden layer
classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu', input_dim = 4))
# Adding the second hidden layer
classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu'))
# Adding the output layer
classifier.add(Dense(units = 13, kernel_initializer = 'uniform', activation = 'softmax'))
# Compiling the ANN
classifier.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])
# Fitting the ANN to the Training set
classifier.fit(X_train, y_train, batch_size =200 , epochs = 100)
# Predicting the Test set results
y_pred = classifier.predict(X_test)
# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
The program runs all the way until running the neural code and also finds the y_pred. After that i get this error i.e the confusion matrix is not formed.
The error:
ValueError: Classification metrics can't handle a mix of multiclass and continuous-multioutput targets
from sklearn.metrics import confusion_matrix
y_pred = classifier.predict(X_test)
predictions = np.argmax(y_pred, axis=-1)
cm = confusion_matrix(y_test, y_pred)
I hope it will resolve your problem
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import LabelEncoder
y_pred = classifier.predict(X_test)
predictions = np.argmax(y_pred, axis=-1)
label_encoder = LabelEncoder().fit(y_test)
label_y = label_encoder.transform(y_test)
cm = confusion_matrix(label_y, predictions)