Related
I have generated a tf.data.Dataset which can be accessed as a class attribute using window.train. This dataset is not batched and its dimensions can be displayed as follows:
for inputs, labels in window.train.take(1):
print(inputs.shape, labels.shape)
which yields:
(46, 27)
(46, 1)
Alternatively, if we were to batch the dataset and take the first batch:
for inputs, labels in window.train.batch(3).take(1):
print(inputs.shape, labels.shape)
which yields:
(3, 46, 27)
(3, 46, 1)
I then created an elementary model that takes a linear combination of the inputs to predict the labels. Since I intend to gradually make this model more articulated, I set it up as a Keras Tuner Hypermodel.
class LinearModel(kt.HyperModel):
def build(self, hp):
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(units=1, activation=None)
model.compile(tf.keras.optimizers.Adam(), loss='mse')
return model
While I understand how to set up the build method, I am not sure how I should set up the fit method, which I need to find the optimal batch size. I tried something as follows:
def fit(self, hp, model, x, validation_data, **kwargs):
hp_batch_size = hp.Choice('batch_size', [100, 200])
x = window.train.batch(hp_batch_size)
validation_data = window.valid.batch(hp_batch_size)
return model.fit(x, validation_data, **kwargs)
This doesn't work. In particular I do not understand what arguments I should pass to fit and why. Can someone help me understand how to define fit? I am familiar with the examples provided by keras but they were not helpful. Importantly, my model is then tuned as follows:
model_tuner = kt.Hyperband(hypermodel=LinearModel, objective='val_loss')
model_tuner.search(x=window.train, validation_data=window.valid, epochs=500)
I want custom Keras layer, realize the output of the two models assign different weights and the weight can be trained as the following
prediction1=model1.output
prediction2=model2.output
class WeightedSum(Layer):
def __init__(self,**kwargs):
super(WeightedSum, self).__init__(**kwargs)
def build(self, input_shape):
self.weights =K.variable(np.random.random(1))
self.trainable_weights=[self.weights]
def call(self, two_model_outputs):
return self.weights * two_model_outputs[0] + (1 - self.weights) * two_model_outputs[1]
def compute_output_shape(self, input_shape):
return input_shape[0]
final_pred=WeightedSum()([prediction1,prediction2])
but I made a mistake in writing and don't know how to do it.
Traceback (most recent call last):
File "test-paper3.py", line 182, in <module>
final_pred=WeightedSum()([prediction1,prediction2])
File "/root/anaconda3/lib/python3.7/site-packages/keras/engine/base_layer.py", line 431, in __call__
self.build(unpack_singleton(input_shapes))
File "test-paper3.py", line 162, in build
self.weights =K.variable(np.random.random(1))
AttributeError: can't set attribute
Maybe Keras is protecting itself from letting you use a word it considers reserved somehow?
Try to add the weights in a standard way and use another variable name:
def build(self, input_shape):
self.kernel = self.add_weight(name='kernel',
shape=(1,),
initializer='uniform',
#I suggest a constraint here, see below
trainable=True)
#this works as an initializer for the weights
K.set_value(self.kernel, np.array([0.5]))
#you can use np.random here, but it seems safer to go with 0.5
#this tells keras that the layer is build in fact
super(WeightedSum, self).build(shapes)
Of course you will need to replace weights with kernel in the call method.
Non related:
I suggest you also use a constraint to keep the kernel within 0 and 1.
from keras.constraints import MinMaxNorm
........
self.kernel = self.add_weight(name='kernel',
shape=(1,),
initializer='uniform',
constraint = MinMaxNorm(0,1)
trainable=True)
........
I'm building this model:
inputs = model.inputs[:2]
layer_output = model.get_layer('Encoder-12-FeedForward-Norm').output
input_layer= keras.layers.Input(shape=(SEQ_LEN,768))(layer_output)
conv_layer= keras.layers.Conv1D(100, kernel_size=3, activation='relu', data_format='channels_first')(input_layer)
maxpool_layer = keras.layers.MaxPooling1D(pool_size=4)(conv_layer)
flat_layer= keras.layers.Flatten()(maxpool_layer)
outputs = keras.layers.Dense(units=3, activation='softmax')(flat_layer)
model = keras.models.Model(inputs, outputs)
model.compile(RAdam(learning_rate =LR),loss='sparse_categorical_crossentropy',metrics=['sparse_categorical_accuracy'])
and I keep getting this error TypeError: 'Tensor' object is not callable I know layer_output is a tensor and not a layer and Keras works with layers. But I'm finding it difficult to figure out the right thing to do. I have previously build a biLSTM model with similar inputs and it works fine. Can someone point me to something that will help me understand the issue better? I have tried passing the input_layer to the conv_layer but I get this error TypeError: Layer conv1d_1 does not support masking, but was passed an input_mask: Tensor("Encoder-12-FeedForward-Add/All:0", shape=(?, 35), dtype=bool)
input_layer= keras.layers.Input(shape=(SEQ_LEN,768))(layer_output)
You're trying to pass an input to an input tensor???
Either you have a tensor: layer_output; or you have an input tensor: Input(shape...). There is no point in trying to mix both things.
In your code, everything on the left side are Tensor, and that's correct!
Everything in the middle are Layer, and all layers are called with the right side, which are Tensor.
tensor_instance = Layer(...)(tensor_instance)
But Input is not a layer, Input is a tensor. You cannot Input(...)(tensor_instance) because Input is not a layer.
There is no need to do anything with layer_output (tensor). You already have it, so just go ahead:
conv_layer_output_tensor = Conv1D(...)(layer_output)
Suggestion:
inputs = model.inputs[:2] #what is this model??
layer_output = model.get_layer('Encoder-12-FeedForward-Norm').output
#this may not work
#unless this output can be fully gotten with the two inputs you selected
#(and there is a chance that Keras code is not prepared for this)
conv_output = keras.layers.Conv1D(100, kernel_size=3, activation='relu',
data_format='channels_first')(layer_output)
maxpool_output = keras.layers.MaxPooling1D(pool_size=4)(conv_output)
flat_output= keras.layers.Flatten()(maxpool_output)
outputs = keras.layers.Dense(units=3, activation='softmax')(flat_output)
another_model = keras.models.Model(inputs, outputs)
another_model.compile(RAdam(learning_rate = LR),
loss='sparse_categorical_crossentropy',
metrics=['sparse_categorical_accuracy'])
Try to add this:
output = Lambda(lambda x: x, output_shape=lambda s: s)(output)
before Conv1D layer.
After saved weights and json configuration of a KerasClassifier model https://github.com/keras-team/keras/blob/master/keras/wrappers/scikit_learn.py I need to restore it and verify results.
But if I restore weight and model then I have a Sequential object, how can I rebuild original KerasClassifier from that??
I'm not sure I understood you correcly, but propose following solution. KerasClassifier inherits from BaseWrapper which has the following __init__ signature:
def __init__(self, build_fn=None, **sk_params):
self.build_fn = build_fn
self.sk_params = sk_params
self.check_params(sk_params)
okay, what's the build_fn and sk_params?
The build_fn should construct, compile and return a Keras model, which
will then be used to fit/predict. One of the following
three values could be passed to build_fn:
1. A function
2. An instance of a class that implements the __call__ method
3. None. This means you implement a class that inherits from either
KerasClassifier or KerasRegressor. The __call__ method of the
present class will then be treated as the default build_fn.
...
sk_params takes both model parameters and fitting parameters. Legal model
parameters are the arguments of build_fn. Note that like all other
estimators in scikit-learn, build_fn should provide default values for
its arguments, so that you could create the estimator without passing any
values to sk_params.
...
some commints are omitted
you can read full comment at this and this links.
As the build_fn expects the function which returns compiled keras model (no matter what it is - Sequential or just Model) - you can pass as value function which returns loaded model.
Edit also you should call fit with some params to restore model using that approach.
load model as build_fn
fit method invokes a build_fn, hence each time you try to train such classifier you will load and then fit loaded clssifier.
For example:
from keras.models import load_model # or another method - but this one is simpliest
from keras.wrappers.scikit_learn import KerasClassifier
def load_model(*args, **kwargs):
"""probably this function expects sk_params, so you can use it in theory"""
path="my_model.hd5"
model = load_model(path)
return model
keras_classifier = KerasClassifier(load_model, sk_params) # use your sk_params
keras_classifier.fit(X_tr, y_tr) # I use slice (1, input_shape) to train
- it will work, as the loaded model almost trained & compiled. But it gives a small shift for your model even if you'll call it with a batch of size 1 and for 1 epoch.
load via build_fn closure
Also you can load the model first (if you wish to provide path easily and it's unnacceptable to hardcode path), then return a function which is "build_fn - acceptable":
def load_model_return_build_fn(path):
model = load_model(path)
def build_fn(*args, **kwars):
"""probably this function expects sk_params"""
return model # defined above
return build_fn
build_fn = load_model_return_build_fn("model.hd5")
keras_classifier = KerasClassifier(build_fn, sk_params) # use your sk_params
keras_classifier.fit(X_tr, y_tr) # I use slice (1, input_shape) to train
assign a model to it's attribute
If you plan just load and use pre-trained model, you can use any to load it, assign to the model attribute and don't call fit.
build_fn = load_model_return_build_fn("model.hd5")
# or the function which realy builds and fits a model
keras_classifier = KerasClassifier(build_fn, sk_params) # use your sk_params
keras_classifier.model = model # assign model here, don't call fit
- that case you set model explicitly to it's attribute. Note that build_fn should be a coorrect one build_fn - otherwise it doesn't pass the self.check_params(sk_params) test.
Inherit from KerasClassifier (not so easy as I've thought)
After all, the best solution I know is inherit from KerasClassifier and add a load and/or from_file method.
class KerasClassifierLoadable(KerasClassifier):
#classmethod
def from_file(cls, path, *args, **kwargs):
keras_classifier = cls(*args, **kwargs)
keras_classifier.model = load_model(path)
outp_shape = keras_classifier.model.layers[-1].output_shape[-1]
if outp_shape > 1:
keras_classifier.classes_ = np.arange(outp_shape, dtype='int32')
else:
raise ValueError("Inconsistent output shape: outp_shape={}".format(outp_shape))
keras_classifier.n_classes_ = len(keras_classifier.classes_)
return keras_classifier
def load(self, path):
self.model = load_model(path)
outp_shape = keras_classifier.model.layers[-1].output_shape[-1]
if outp_shape > 1:
keras_classifier.classes_ = np.arange(outp_shape, dtype='int32')
else:
raise ValueError("Inconsistent output shape: outp_shape={}".format(outp_shape))
self.n_classes_ = len(self.classes_)
here we shoul set self.classes_ to the correct class labels - but I use just an integer values from `range(0, n_classes).
Usage (the build_fn can be any appropiate build_fn):
keras_classifier = KerasClassifierLoadable.from_file("model.hd5", build_fn=build_fn)
keras_classifier = KerasClassifierLoadable(build_fn=build_fn)
keras_classifier.load("model.hd5")
If you have two files model.json and weights.h5, then you can easily load the model and use it as you want.
from keras.models import model_from_json
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
loaded_model.load_weights("model.h5")
# evaluate loaded model on test data
loaded_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
score = loaded_model.evaluate(X, Y, verbose=0)
print("%s: %.2f%%" % (loaded_model.metrics_names[1], score[1]*100))
Inspired by tf.keras.Model subclassing I created custom model.
I can train it and get successfull results, but I can't save it.
I use python3.6 with tensorflow v1.10 (or v1.9)
Minimal complete code example here:
import tensorflow as tf
from tensorflow.keras.datasets import mnist
class Classifier(tf.keras.Model):
def __init__(self):
super().__init__(name="custom_model")
self.batch_norm1 = tf.layers.BatchNormalization()
self.conv1 = tf.layers.Conv2D(32, (7, 7))
self.pool1 = tf.layers.MaxPooling2D((2, 2), (2, 2))
self.batch_norm2 = tf.layers.BatchNormalization()
self.conv2 = tf.layers.Conv2D(64, (5, 5))
self.pool2 = tf.layers.MaxPooling2D((2, 2), (2, 2))
def call(self, inputs, training=None, mask=None):
x = self.batch_norm1(inputs)
x = self.conv1(x)
x = tf.nn.relu(x)
x = self.pool1(x)
x = self.batch_norm2(x)
x = self.conv2(x)
x = tf.nn.relu(x)
x = self.pool2(x)
return x
if __name__ == '__main__':
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(*x_train.shape, 1)[:1000]
y_train = y_train.reshape(*y_train.shape, 1)[:1000]
x_test = x_test.reshape(*x_test.shape, 1)
y_test = y_test.reshape(*y_test.shape, 1)
y_train = tf.keras.utils.to_categorical(y_train)
y_test = tf.keras.utils.to_categorical(y_test)
model = Classifier()
inputs = tf.keras.Input((28, 28, 1))
x = model(inputs)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Dense(10, activation="sigmoid")(x)
model = tf.keras.Model(inputs=inputs, outputs=x)
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(x_train, y_train, epochs=1, shuffle=True)
model.save("./my_model")
Error message:
1000/1000 [==============================] - 1s 1ms/step - loss: 4.6037 - acc: 0.7025
Traceback (most recent call last):
File "/home/user/Data/test/python/mnist/mnist_run.py", line 62, in <module>
model.save("./my_model")
File "/home/user/miniconda3/envs/ml3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1278, in save
save_model(self, filepath, overwrite, include_optimizer)
File "/home/user/miniconda3/envs/ml3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/saving.py", line 101, in save_model
'config': model.get_config()
File "/home/user/miniconda3/envs/ml3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1049, in get_config
layer_config = layer.get_config()
File "/home/user/miniconda3/envs/ml3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1028, in get_config
raise NotImplementedError
NotImplementedError
Process finished with exit code 1
I looked into the error line and found out that get_config method checks self._is_graph_network
Do anybody deal with this problem?
Thanks!
Update 1:
On the keras 2.2.2 (not tf.keras)
Found comment (for model saving)
file: keras/engine/network.py
Function: get_config
# Subclassed networks are not serializable
# (unless serialization is implemented by
# the author of the subclassed network).
So, obviously it won't work...
I wonder, why don't they point it out in the documentation (Like: "Use subclassing without ability to save!")
Update 2:
Found in keras documentation:
In subclassed models, the model's topology is defined as Python code
(rather than as a static graph of layers). That means the model's
topology cannot be inspected or serialized. As a result, the following
methods and attributes are not available for subclassed models:
model.inputs and model.outputs.
model.to_yaml() and model.to_json()
model.get_config() and model.save().
So, there is no way to save model by using subclassing.
It's possible to only use Model.save_weights()
TensorFlow 2.2
Thanks for #cal for noticing me that the new TensorFlow has supported saving the custom models!
By using model.save to save the whole model and by using load_model to restore previously stored subclassed model. The following code snippets describe how to implement them.
class ThreeLayerMLP(keras.Model):
def __init__(self, name=None):
super(ThreeLayerMLP, self).__init__(name=name)
self.dense_1 = layers.Dense(64, activation='relu', name='dense_1')
self.dense_2 = layers.Dense(64, activation='relu', name='dense_2')
self.pred_layer = layers.Dense(10, name='predictions')
def call(self, inputs):
x = self.dense_1(inputs)
x = self.dense_2(x)
return self.pred_layer(x)
def get_model():
return ThreeLayerMLP(name='3_layer_mlp')
model = get_model()
# Save the model
model.save('path_to_my_model',save_format='tf')
# Recreate the exact same model purely from the file
new_model = keras.models.load_model('path_to_my_model')
See: Save and serialize models with Keras - Part II: Saving and Loading of Subclassed Models
TensorFlow 2.0
TL;DR:
do not use model.save() for custom subclass keras model;
use save_weights() and load_weights() instead.
With the help of the Tensorflow Team, it turns out the best practice of saving a Custom Sub-Class Keras Model is to save its weights and load it back when needed.
The reason that we can not simply save a Keras custom subclass model is that it contains custom codes, which can not be serialized safely. However, the weights can be saved/loaded when we have the same model structure and custom codes without any problem.
There has a great tutorial written by Francois Chollet who is the author of Keras, for how to save/load Sequential/Functional/Keras/Custom Sub-Class Models in Tensorflow 2.0 in Colab at here. In Saving Subclassed Models section, it said that:
Sequential models and Functional models are datastructures that represent a DAG of layers. As such, they can be safely serialized and deserialized.
A subclassed model differs in that it's not a datastructure, it's a
piece of code. The architecture of the model is defined via the body
of the call method. This means that the architecture of the model
cannot be safely serialized. To load a model, you'll need to have
access to the code that created it (the code of the model subclass).
Alternatively, you could be serializing this code as bytecode (e.g.
via pickling), but that's unsafe and generally not portable.
This will be fixed in an upcoming release according to the 1.13 pre-release patch notes:
Keras & Python API:
Subclassed Keras models can now be saved through tf.contrib.saved_model.save_keras_model.
EDIT:
It seems this is not quite as finished as the notes suggest. The docs for that function for v1.13 state:
Model limitations: - Sequential and functional models can always be saved. - Subclassed models can only be saved when serving_only=True. This is due to the current implementation copying the model in order to export the training and evaluation graphs. Because the topology of subclassed models cannot be determined, the subclassed models cannot be cloned. Subclassed models will be entirely exportable in the future.
Tensorflow 2.1 allows to save subclassed models with SavedModel format
From my beginning using Tensorflow, i was always a fan of Model Subclass, i feel this way of build models more pythonic and collaborative friendly. But saving the model was always a point of pain with this approach.
Recently i started to update my knowledge and reach to the following information that seems to be True for Tensorflow 2.1:
Subclassed Models
I found this
Second approach is by using model.save to save whole model and by
using load_model to restore previously stored subclassed model.
This last saves the model, the weight and other stuff into a SavedModel file
And by last the confirmation:
Saving custom objects:
If you are using the SavedModel format, you can
skip this section. The key difference between HDF5 and SavedModel is
that HDF5 uses object configs to save the model architecture, while
SavedModel saves the execution graph. Thus, SavedModels are able to
save custom objects like subclassed models and custom layers without
requiring the orginal code.
I tested this personally, and efectively, model.save() for subclassed models generate a SavedModel save. There is no more need for use model.save_weights() or related functions, they now are more for specific usecases.
This is suposed to be the end of this painful path for all of us interested in Model Subclassing.
I found a way to solve it. Create a new model and load the weights from the saved .h5 model. This way is not preferred, but it works with keras 2.2.4 and tensorflow 1.12.
class MyModel(keras.Model):
def __init__(self, inputs, *args, **kwargs):
outputs = func(inputs)
super(MyModel, self).__init__( inputs=inputs, outputs=outputs, *args, **kwargs)
def get_model():
return MyModel(inputs, *args, **kwargs)
model = get_model()
model.save(‘file_path.h5’)
model_new = get_model()
model_new.compile(optimizer=optimizer, loss=loss, metrics=metrics)
model_new.load_weights(‘file_path.h5’)
model_new.evaluate(x_test, y_test, **kwargs)
UPDATE: Jul 20
Recently I also tried to create my subclassed layers and model. Write your own get_config() function might be difficult. So I used model.save_weights(path_to_model_weights) and model.load_weights(path_to_model_weights). When you want to load the weights, remember to create the model with the same architecture than do model.load_weights(). See the tensorflow guide for more details.
Old Answer (Still correct)
Actually, tensorflow document said:
In order to save/load a model with custom-defined layers, or a subclassed model, you should overwrite the get_config and optionally from_config methods. Additionally, you should use register the custom object so that Keras is aware of it.
For example:
class Linear(keras.layers.Layer):
def __init__(self, units=32, **kwargs):
super(Linear, self).__init__(**kwargs)
self.units = units
def build(self, input_shape):
self.w = self.add_weight(
shape=(input_shape[-1], self.units),
initializer="random_normal",
trainable=True,
)
self.b = self.add_weight(
shape=(self.units,), initializer="random_normal", trainable=True
)
def call(self, inputs):
return tf.matmul(inputs, self.w) + self.b
def get_config(self):
config = super(Linear, self).get_config()
config.update({"units": self.units})
return config
layer = Linear(64)
config = layer.get_config()
print(config)
new_layer = Linear.from_config(config)
The output is:
{'name': 'linear_8', 'trainable': True, 'dtype': 'float32', 'units': 64}
You can play with this simple code. For example, in function "get_config()", remove config.update(), see what's going on. See this and this for more details. These are the Keras guide on tensorflow website.
use model.predict before tf.saved_model.save
Actually recreating the model with
keras.models.load_model('path_to_my_model')
didn't work for me
First we have to save_weights from the built model
model.save_weights('model_weights', save_format='tf')
Then
we have to initiate a new instance for the subclass Model then compile and train_on_batch with one record and load_weights of built model
loaded_model = ThreeLayerMLP(name='3_layer_mlp')
loaded_model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
loaded_model.train_on_batch(x_train[:1], y_train[:1])
loaded_model.load_weights('model_weights')
This work perfectly in TensorFlow==2.2.0