I'm trying to obtain a structure of Keras layer. Within a simple network, this is possible by just iterating through the model.layers. However, if the network is complex (e.g. concatenates different layer) this approach is not sufficient.
This is an example:
FEATURES = ['A','B','C','D']
IMPORTANT_FEATURES = [0, 3]
NORMAL_FEATURES = [1, 2]
inputLayer = [Input(shape=(1, )) for i,f in enumerate(FEATURES)]
importantInput = keras.layers.Concatenate(axis=-1)([inputLayer[i] for i in IMPORTANT_FEATURES])
layer1 = Dense()(importantInput)
normalInput = keras.layers.Concatenate(axis=-1)(layer1 + [inputLayer[i] for i in NORMAL_FEATURES])
layer2 = Dense()(normalInput)
model = Model([
inputLayer[i]
for i in range(len(FEATURES))
], layer2)
This produces in model.layer a list of Keras layers composed like this:
[Input1, Input2, Concatenate1, Dense1, Input3, Input4, Concatenate2, Dense2]
The only way to get which level is in input to the concatenate is to access to Concatenate1.input. However, this returns a list of Tensorflow layers.
Is it possible to obtain a tree-structure of layers just using Keras layers?
The Keras list of input layers can be obtained by Concatenate1._inbound_nodes[0].inbound_layers
Related
I have 'n' no. of vectors of 'm' size each. I need to send them to a SimpleRNN of keras. The vectors should be send such that each neuron of the RNN takes a vector(eg: vector1 to neuron1, vector2 t neuron2 etc) along with hidden state of previous input vector.
I have tried concatenating them, but this distorts the nature of the input.
input1 = Dense(20, activation = "relu")(input1)
input2 = Dense(20, activation = "relu")(input2)
I need to send these vectors(input1 and input2) to the RNN.
You can use tf.stack in Tensorflow or keras.backend.stack in Keras. This operator:
Stacks a list of rank-R tensors into one rank-(R+1) tensor
Based on your code, Dense layers can be stacked in the following way:
import tensorflow as tf
inps1 = tf.keras.layers.Input(shape=(30,))
inps2 = tf.keras.layers.Input(shape=(30,))
dense1 = tf.keras.layers.Dense(20, activation='relu')(inps1)
dense2 = tf.keras.layers.Dense(20, activation='relu')(inps2)
dense = tf.keras.layers.Lambda(lambda x: tf.stack([x[0], x[1]], axis=1), output_shape=(None, 2, 20))([dense1, dense2])
rnn = tf.keras.layers.SimpleRNN(100)(dense)
I defined a custom Keras layer custom_layer with two outputs: output_1 and output_2. Next, I want two independent layers A and B to connect to output_1 and output_2 respectively. How to implement this kind of network?
Using the keras api mode you can create any network architecture.
In your case a possible solution is
input_layer = Input(shape=(100,1))
custom_layer = Dense(10)(input_layer)
# layer A model
layer_a = Dense(10, activation='relu')(custom_layer)
output1 = Dense(1, activation='sigmoid')(layer_a)
# layer B model
layer_b = Dense(10, activation='relu')(custom_layer)
output1 = Dense(1, activation='sigmoid')(layer_b)
# define model input and output
model = Model(inputs=input_layer, outputs=[output1, output2])
If the custom layer has two output tensors (i.e. it returns a list of output tensors) when applied on one input, then:
custom_layer_output = CustomLayer(...)(input_tensor)
layer_a_output = LayerA(...)(custom_layer_output[0])
layer_b_output = LayerB(...)(custom_layer_output[1])
But if it is applied on two different input tensors, then:
custom_layer = CustomLayer(...)
out1 = custom_layer(input1)
out2 = custom_layer(input2)
layer_a_output = LayerA(...)(out1)
layer_b_output = LayerB(...)(out2)
# alternative way
layer_a_output = LayerA(...)(custom_layer.get_output_at(0))
layer_b_output = LayerB(...)(custom_layer.get_output_at(1))
Keras supports having multiple output layers in your custom layer. There is a merge, which will update the documentation soon.
The basic idea is to work with lists. Everithing you have to reutrn in your custom layer (like layers and shape) you have to return as lists of them.
If you implement your custom layer in the right way the rest is simple:
output_1, output_2 = custom_layer()(input_layer)
layer_a_output = layer_a()(output_1)
layer_b_output = layer_b()(output_2)
In keras, I want to calculate the mean of nonzero embedding output.
I wonder what is the difference between mask_zero=True or False in Embedding Layer.
I tried the code below :
input_data = Input(shape=(5,), dtype='int32', name='input')
embedding_layer = Embedding(1000, 24, input_length=5,mask_zero=True,name='embedding')
out = word_embedding_layer(input_data)
def antirectifier(x):
x = K.mean(x, axis=1, keepdims=True)
return x
def antirectifier_output_shape(input_shape):
shape = list(input_shape)
return tuple(shape)
out = Lambda(antirectifier, output_shape=antirectifier_output_shape,name='lambda')(out)
But it seems that the result is the mean of all the elements, how can i just calculate the mean of all nonzero inputs?
From the function's doc :
If this is True then all subsequent layers in the model need to
support masking
Your lambda function doesn't support masking. For example Recurrent layers in Keras support masking. If you set mask_zero=True in your embeddings, then all the 0 indices that you feed to the embedding layer will be propagated as "masked" and the following layers that are able to understand the "masked" information will use them.
Basically, if you build a "mean" layer that grabs the mask and computes the average only for non-masked values, then you will get the desired results.
You can find here a way to build your lambda layers that support masking
I hope it helps.
I'm currently trying to implement a Recurrent Neural Network in Keras. The data consists of a collection of 45.000 whereby each entry is a collection (of variable length) of MFCC vectors with each 13 coefficients:
spoken = numpy.load('spoken.npy')
print(spoken[0]) # Gives:
example_row = [
[
5.67170000e-01 -1.79430000e-01 -7.27360000e+00 -9.59300000e-02
-9.30140000e-02 -1.62960000e-01 4.11620000e-01 3.00590000e-01
6.86360000e-02 1.07130000e+00 1.07090000e-01 5.00890000e-01
7.51750000e-01],
[.....]
]
print(spoken.shape) # Gives: (45000,0)
print(spoken[0].shape) # Gives (N, 13) --> N amount of MFCC vectors
I'm struggling to understand how I need to reshape this Numpy array in order to feed it to the SimpleRNN of Keras:
model = Sequential()
model_spoken.add(SimpleRNN(units=10, activation='relu', input_shape=?))
.....
Therefore, my question is how do I need to reshape a collection of variable length MFCC vectors so that I can feed it to the SimpleRNN object of Keras?
It was actually quite simple since Keras has built in function for reformatting the array and padding zeros to get a static length:
spoken_train = pad_sequences(spoken_train, maxlen=100)
See github issue
I would like to merge a forward LSTM and a backward LSTM in Keras. The input array of the backward LSTM is different from that of a forward LSTM. Thus, I cannot use keras.layers.Bidirectional.
The forward input is (10, 4).
The backward input is (12, 4) and it is reversed before put into the model. I would like to reverse it again after LSTM and merge it with the forward.
The simplified model is as follows.
from lambdawithmask import Lambda as MaskLambda
def reverse_func(x, mask=None):
return tf.reverse(x, [False, True, False])
forward = Sequential()
backward = Sequential()
model = Sequential()
forward.add(LSTM(input_shape = (10, 4), output_dim = 4, return_sequences = True))
backward.add(LSTM(input_shape = (12, 4), output_dim = 4, return_sequences = True))
backward.add(MaskLambda(function=reverse_func, mask_function=reverse_func))
model.add(Merge([forward, backward], mode = "concat", concat_axis = 1))
When I run this, the error message is:
Tensors in list passed to 'values' of 'ConcatV2' Op have types [bool, float32] that don't all match.
Could anyone help me? I coded in Python 3.5.2 with Keras (2.0.5) and the backend is tensorflow (1.2.1).
First of all, if you have two different inputs, you cannot use a Sequential model. You must use the functional API Model:
from keras.models import Model
The two first models can be sequential, no problem, but the junction must be a regular model. When it's about concatenating, I also use the functional approach (create the layer, then pass the input):
junction = Concatenate(axis=1)([forward.output,backward.output])
Why axis=1? You can only concatenate things with the same shape. Since you have 10 and 12, they're not compatible unless you use this exact axis for the merge, which is the second axis, considering you have (BatchSize, TimeSteps, Units)
For creating the final model, use the Model, specify the inputs and outputs:
model = Model([forward.input,backward.input], junction)
In the model to be reversed, use simply a Lambda layer. A MaskLambda does more than just the function you want. I also suggest you use the keras backend insted of tensorflow functions:
import keras.backend as K
#instead of the MaskLambda:
backward.add(Lambda(lambda x: K.reverse(x,axes=[1]), output_shape=(12,?))
Here, the ? is the amount of units your LSTM layers have. See PS at the end.
PS: I'm not sure output_dim is useful in the LSTM layer. It's necessary in Lambda layers, but I never use it anywhere else. Shapes are natural consequences of the amount of "units" you put in your layers. Strangely, you didn't specify the amount of units.
PS2: How exactly do you want to concatenate two sequences with different sizes?
As said in above answer, using a Functional API offers you much flexibility in case of multi input/output models. You can simply set the go_backwards argument as True to reverse the traversal of the input vector by the LSTM layer.
I have defined the smart_merge function below which merges the forward and backward LSTM layers together along with handling the single traversal case.
from keras.models import Model
from keras.layers import Input, merge
def smart_merge(vectors, **kwargs):
return vectors[0] if len(vectors)==1 else merge(vectors, **kwargs)
input1 = Input(shape=(10,4), dtype='int32')
input2 = Input(shape=(12,4), dtype='int32')
LtoR_LSTM = LSTM(56, return_sequences=False)
LtoR_LSTM_vector = LtoR_LSTM(input1)
RtoL_LSTM = LSTM(56, return_sequences=False, go_backwards=True)
RtoL_LSTM_vector = RtoL_LSTM(input2)
BidireLSTM_vector = [LtoR_LSTM_vector]
BidireLSTM_vector.append(RtoL_LSTM_vector)
BidireLSTM_vector= smart_merge(BidireLSTM_vector, mode='concat')