I am working on a project in which I need to edit individual weights and biases.
Is there any way to actually get access to the layers weights and biases so I can edit them manually?
from tf.layers.dense()
So far I have created my own model and stored the biases outside like so:
for _ in range(population_size):
hidden_layer.append(tf.Variable(tf.truncated_normal([11, 20])))
output_layer.append(tf.Variable(tf.truncated_normal([20, 9])))
population.append([hidden_layer, output_layer])
I am then trying to feed population into the model using feed dict. It's turning out to be a real hell because I cannot feed them into the models because of the shapes of the Variables are not the same.
Is there any native support for getting the weights from the dense layer?
From the keras doc:
All Keras layers have a number of methods in common:
layer.get_weights(): returns the weights of the layer as a list of Numpy arrays.
layer.set_weights(weights): sets the weights of the layer from a list of Numpy arrays (with the same shapes as the output of get_weights).
You can easily access all layers inside your model with yourmodel.layers.
Related
How can I use the weights of a pre-trained network in my tensorflow project?
I know some theory information about this but no information about coding in tensorflow.
As been pointed out by #Matias Valdenegro in the comments, your first question does not make sense. For your second question however, there are multiple ways to do so. The term that you're searching for is Transfer Learning (TL). TL means transferring the "knowledge" (basically it's just the weights) from a pre-trained model into your model. Now there are several types of TL.
1) You transfer the entire weights from a pre-trained model into your model and use that as a starting point to train your network.
This is done in a situation where you now have extra data to train your model but you don't want to start over the training again. Therefore you just load the weights from your previous model and resume the training.
2) You transfer only some of the weights from a pre-trained model into your new model.
This is done in a situation where you have a model trained to classify between, say, 5 classes of objects. Now, you want to add/remove a class. You don't have to re-train the whole network from the start if the new class that you're adding has somewhat similar features with (an) existing class(es). Therefore, you build another model with the same exact architecture as your previous model except the fully-connected layers where now you have different output size. In this case, you'll want to load the weights of the convolutional layers from the previous model and freeze them while only re-train the fully-connected layers.
To perform these in Tensorflow,
1) The first type of TL can be performed by creating a model with the same exact architecture as the previous model and simply loading the model using tf.train.Saver().restore() module and continue the training.
2) The second type of TL can be performed by creating a model with the same exact architecture for the parts where you want to retain the weights and then specify the name of the weights in which you want to load from the previous pre-trained weights. You can use the parameter "trainable=False" to prevent Tensorflow from updating them.
I hope this helps.
I trained a LeNet architecture on a first dataset. I want to train a VGG architecture on an other dataset by initializing the weights of VGG with weights obtained from LeNet.
All initialization functions in keras are predefined and I do not find how to customize them. For example :
keras.initializers.Zeros()
Any idea how I can set the weights?
https://keras.io/layers/about-keras-layers/
According to the Keras documentation above:
layer.set_weights(weights) sets the weights of the layer from a list of Numpy arrays
layer.get_weights() returns the weights of the layer as a list of Numpy arrays
So, you can do this as follows:
model = Sequential()
model.add(Dense(32))
... building the model's layers ...
# access any nth layer by calling model.layers[n]
model.layers[0].set_weights( your_weights_here )
Of course, you'll need to make sure you are setting the weights of each layer to the appropriate shape they should be.
I am using model.train_on_batch in keras in order to be able to handle different batches of input data differently. Essentially I cannot use model.fit
But I need to store histogram and images of activations and weights in Tensorboard. Is there a way to do this?
You can do it manually by calling summary.histogram and passing the weights of each layer as below
with summary_writer.as_default():
for layer in self.model.layers:
for weight in layer.weights:
tf.summary.histogram('weights/{}'.format(layer.name), weight, step=your_step)
I haven't used Keras and I'm thinking whether to use it or not.
I want to save a trained layer to use later. For example:
I train a model.
Then, I gain a trained layer t_layer.
I have another model to train which consists of layer1, layer2, layer3 .
I want to use t_layer as layer2 and not to update this layer(i.e. t_layer does not learn any more).
This may be an odd attempt, but I want to try this. Is this possible on Keras?
Yes, it is.
You will probably have to save the layer's weights and biases instead of saving the layer itself, but it's possible.
Keras also allows you to save entire models.
Suppose you have a model in the var model:
weightsAndBiases = model.layers[i].get_weights()
This is a list of numpy arrays, very probably with two arrays: weighs and biases. You can simply use numpy.save() to save these two arrays and later you can create a similar layer and give it the weights:
from keras.layers import *
from keras.models import Model
inp = Input(....)
out1 = SomeKerasLayer(...)(inp)
out2 = AnotherKerasLayer(....)(out1)
....
model = Model(inp,out2)
#above is the usual process of creating a model
#supposing layer 2 is the layer you want (you can also use names)
weights = numpy.load(...path to your saved weights)
biases = numpy.load(... path to your saved biases)
model.layers[2].set_weights([weights,biases])
You can make layers untrainable (must be done before the model compilation):
model.layers[2].trainable = False
Then you compile the model:
model.compile(.....)
And there you go, a model, whose one layer is untrainable and has weights and biases defined by you, taken from somewhere else.
Yes, it is a common practice in transfer learning, see here.
Thjs piece_to_share below can be one or more layers.
piece_to_share = tf.keras.Model(...)
full_model = tf.keras.Sequential([piece_to_share, ...])
full_model.fit(...)
piece_to_share.save(...)
I'm confusing about the mechanism of the merge layer in Keras.
For example, branch_left is a Sequential()model that contains two dense() layers while branch_right is another Sequential() model that has only one dense(). Then there is a merge() layer connect them in a concatenate way and I put the merge layer into a Sequential() model merged_model. Since I wanna get the name or weights saved in layers from those two branch model, I tried to call merged_model.layers and found the first layer is merge layer.
Considering len(erge.get_weights())=0, is there any way to get the layers existing in branch_left and branch_right? If that is possible, then I can modify the properties which are belonging to those layers.
I'm not sure it's a good idea to put the merge layer into a sequential...
(Sequential models do not support branches)
The best to do is to create a functional API model:
from keras.models import Model
merged_model = Model([branch_left.input,branch_right.input],concatLayer.output)