I defined a custom Keras layer custom_layer with two outputs: output_1 and output_2. Next, I want two independent layers A and B to connect to output_1 and output_2 respectively. How to implement this kind of network?
Using the keras api mode you can create any network architecture.
In your case a possible solution is
input_layer = Input(shape=(100,1))
custom_layer = Dense(10)(input_layer)
# layer A model
layer_a = Dense(10, activation='relu')(custom_layer)
output1 = Dense(1, activation='sigmoid')(layer_a)
# layer B model
layer_b = Dense(10, activation='relu')(custom_layer)
output1 = Dense(1, activation='sigmoid')(layer_b)
# define model input and output
model = Model(inputs=input_layer, outputs=[output1, output2])
If the custom layer has two output tensors (i.e. it returns a list of output tensors) when applied on one input, then:
custom_layer_output = CustomLayer(...)(input_tensor)
layer_a_output = LayerA(...)(custom_layer_output[0])
layer_b_output = LayerB(...)(custom_layer_output[1])
But if it is applied on two different input tensors, then:
custom_layer = CustomLayer(...)
out1 = custom_layer(input1)
out2 = custom_layer(input2)
layer_a_output = LayerA(...)(out1)
layer_b_output = LayerB(...)(out2)
# alternative way
layer_a_output = LayerA(...)(custom_layer.get_output_at(0))
layer_b_output = LayerB(...)(custom_layer.get_output_at(1))
Keras supports having multiple output layers in your custom layer. There is a merge, which will update the documentation soon.
The basic idea is to work with lists. Everithing you have to reutrn in your custom layer (like layers and shape) you have to return as lists of them.
If you implement your custom layer in the right way the rest is simple:
output_1, output_2 = custom_layer()(input_layer)
layer_a_output = layer_a()(output_1)
layer_b_output = layer_b()(output_2)
Related
Is it possible to simply share one embedding layer with one input with multiple features ?
Is it possible to avoid to create multiple inputs layers one by feature.
I would like to avoid to create 34 input layers (one by feature).
The goal is to pass throw one embedding layer 34 feature sequence, get 34 embedded vector sequences. Concatenate them to obtain one super feature vector sequence. And then feed a LSTM.
input shape (None,100,34) -> Embedding_layer_size_64 -> (None,100, 34*64) -> LSTM -> softmax
hope it's clear
The Solution:
# Shared embedding
embedding_layer = Embedding(input_dim = vocab_size+1, output_dim = emb_dim, input_length = nb_timesteps, mask_zero = True)
# For every features we have it's own input
feature_inputs = [Input(shape=(nb_timesteps, ), name='feature_' + str(i + 1)) for i in range(nb_features)]
# Repeat this for every feature
feature_embeddings = [embedding_layer(f) for f in feature_inputs]
# Concatenate the embedding outputs
concatenated_embeddings = concatenate(feature_embeddings, axis=-1)
lstm_1 = LSTM(output_dim)(concatenated_embeddings)
output_layer = Dense(nb_classes, activation='softmax')(lstm_1)
model = Model(inputs=feature_inputs, outputs=output_layer, name="Multi_feature_Embedding_LSTM")
I would like to know how to stack many layers of RNN but every layer are the same RNN. I want every layer share the same weight. I have read stack LSTM and RNN, but I found that each layer was not the same.
1 layer code:
inputs = keras.Input(shape=(maxlen,), batch_size = batch_size)
Emb_layer = layers.Embedding(max_features,word_dim)
Emb_output = Emb_layer(inputs)
first_layer = layers.SimpleRNN(n_hidden,use_bias=True,return_sequences=False,stateful =False)
first_layer_output = first_layer(Emb_output)
dense_layer = layers.Dense(1, activation='sigmoid')
dense_output = dense_layer(first_layer_output )
model = keras.Model(inputs=inputs, outputs=dense_output)
model.summary()
enter image description here
RNN 1 layer
inputs = keras.Input(shape=(maxlen,), batch_size = batch_size)
Emb_layer = layers.Embedding(max_features,word_dim)
Emb_output = Emb_layer(inputs)
first_layer = layers.SimpleRNN(n_hidden,use_bias=True,return_sequences=True,stateful =True)
first_layer_output = first_layer(Emb_output)
first_layer_state = first_layer.states
second_layer = layers.SimpleRNN(n_hidden,use_bias=True,return_sequences=False,stateful =False)
second_layer_set_state = second_layer(first_layer_output, initial_state=first_layer_state)
dense_layer = layers.Dense(1, activation='sigmoid')
dense_output = dense_layer(second_layer_set_state )
model = keras.Model(inputs=inputs, outputs=dense_output)
model.summary()
enter image description here
Stack RNN 2 layer.
For example, I want to build two layers RNN, but the first layer and the second must have the same weight, such that when I update the weight in the first layer the second layer must be updated and share the same value. As far as I know, TF has RNN.state. It returns the value from the previous layer. However, when I use this, it seems that each layer is treated independently. The 2-layer RNN that I want should have trainable parameters equal to the 1-layer since they shared the same weight, but this did not work.
You can view the layer object as a container for the weights that knows how to apply the weights. You can use the layer object as many times as you want. Assuming the embedding and the RNN dimension are the same, you can do:
states = Emb_layer(inputs)
first_layer = layers.SimpleRNN(n_hidden, use_bias=True, return_sequences=True)
for _ in range(10):
states = first_layer(states)
There is no reason to set stateful to true. This is used when you split long sequences into multiple batches and what the RNN to remember the state between batches, so you do not have yo manually set initial states. You can get the final state of the RNN (that you wany you want to use for classification) by simply indexing the last position from states.
Megre doesn't work anymore. I tried the new functional API (concatenate, add, multiply) but it doesn't work for models. How to implement it?
lower_model = [self.build_network(self.model_config['critic_lower'], input_shape=(self.history_length, self.n_stock, 1))
for _ in range(1 + self.n_smooth + self.n_down)]
merged = Merge(lower_model, mode='concat')
# upper layer
upper_model = self.build_network(self.model_config['critic_upper'], model=merged)
# action layer
action = self.build_network(self.model_config['critic_action'], input_shape=(self.n_stock,), is_conv=False)
# output layer
merged = Merge([upper_model, action], mode='mul')
model = Sequential()
model.add(merged)
model.add(Dense(1))
return model
I cannot really give you the exact answer, because your question is not detailed enough, but I can provide you an example, where layers are concatenated. Common problem is to import Concatenate and use it as in previous versions.
nlp_input = Input(shape=(seq_length,), name='nlp_input')
meta_input = Input(shape=(10,), name='meta_input')
emb = Embedding(output_dim=embedding_size, input_dim=100, input_length=seq_length)(nlp_input)
nlp_out = Bidirectional(LSTM(128, dropout=0.3, recurrent_dropout=0.3, kernel_regularizer=regularizers.l2(0.01)))(emb)
x = concatenate([nlp_out, meta_input])
x = Dense(classifier_neurons, activation='relu')(x)
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[nlp_input , meta_input], outputs=[x])
This is a dirty workaround to show how to get input and output tensors from models and use concatenate layers with them. Also to learn how to use Dense and other layers with tensors and create functional API models.
Ideally, you should rewrite everything that's inside build_network for clean and optimized code. (Perhaps this doesn't even work depending on the content of this function, but this is the idea)
lower_model = [self.build_network(
self.model_config['critic_lower'],
input_shape=(self.history_length, self.n_stock, 1))
for _ in range(1 + self.n_smooth + self.n_down)]
#for building models you need input and output tensors
lower_inputs = [model.input for model in lower_model]
lower_outputs = [model.output for model in lower_model]
#these lines assume each model in the list has only one input and output
#using a concatenate layer on a list of tensors
merged_tensor = Concatenate()(lower_outputs) #or Concatenate(axis=...)(lower_outputs)
#this is a workaround for compatibility.
#ideally you should work just with tensors, not create unnecessary intermediate models
merged_model = Model(lower_inputs, merged_tensor) #make model from input tensors to outputs
# upper layer
upper_model = self.build_network(self.model_config['critic_upper'], model=merged_model)
# action layer
action = self.build_network(self.model_config['critic_action'], input_shape=(self.n_stock,), is_conv=False)
# output layer - get the output tensors from the models
upper_out = upper_model.output
action_out = action.output
#apply the Multiply layer on the list of tensors
merged_tensor = Multiply()([upper_out, action_out])
#apply the Dense layer on the merged tensor
out = Dense(1)(merged_tensor)
#get input tensors to create a model
upper_iputs = upper_model.inputs #should be a list
action_inputs = action.inputs #if not a list, append to the previous list
inputs = upper_inputs + action_inputs
model = Model(inputs, out)
return model
Using the functional API, I specified a complex model. I reuse this model several times like specified:
def build_model():
# build a model
model = Model(..., name="complex_model")
return model
complex_model = build_model() # return a Keras model
input_a = Input(...)
input_b = Input(...)
inst_a = complex_model(input_a)
inst_b = complex_model(input_b)
merged = ... # merge inst_a, inst_b
output = Dense(..., name="last_dense")(merged)
model = Model(inputs=[input_a, input_b], outputs=output)
model.compile(...)
model.fit(..., callbacks=...) # see below for callbacks
I want to visualize the weights of some layers using TensorBoard. This can easily be done for the last_dense layer like
ks.callbacks.TensorBoard(..., embeddings_freq=1, embeddings_layer_names=["last_dense"])
But how can I access a layer that is inside the complex model?
Say I want to access a layer called first_dense, neither of the following works:
ks.callbacks.TensorBoard(..., embeddings_freq=1, embeddings_layer_names=["first_dense"]
ks.callbacks.TensorBoard(..., embeddings_freq=1, embeddings_layer_names=["complex_model/first_dense"]
Instead, it throws an
ValueError: No variables to save
Is it ever possible to access this layer and if yes, how?
I need to feed variable length sequences into my model.
My model is Embedding + LSTM + Conv1d + Maxpooling + softmax.
When I set mask_zero = True in Embedding, I fail to compile at Conv1d.
How can I input mask value in Conv1d or is there another solution?
The Masking layer expects every downstream layer to support masking, which is not the case of the Conv1D layer. Fortunately, there is another way to apply masking, using the Functional API:
inputs = Input(...)
mask = Masking().compute_mask(inputs) # <= Compute the mask
embed = Embedding(...)(inputs)
lstm = LSTM(...)(embed, mask=mask) # <= Apply the mask
conv = Conv1D(...)(lstm)
...
model = Model(inputs=[inputs], outputs=[...])
Conv1D layer does not support masking at this time. Here is an open issue on the keras repo.
Depending on the task you might be able to get away with embedding the mask_value just like the other values in the sequence and apply global pooling (as you're doing now).