I'm trying to add an extra branch to the original model to improve its capability.
To be more specific, the original model looks like this:
input1 = Input()
x1 = branch1(x1)
x1 = residualnet(x1)
model = Model(inputs = [input1], outputs = x1)
My improvements would be like this:
input1 = Input()
input2 = Input()
x1 = branch1(x1) #1
x2 = branch2(x2) #2
x1 = concat([x1, x2])
x1 = residualnet(x1) #3
model = Model(inputs = [input1, input2], outputs = x1)
In this condition, I already have trained weights to #1 and #3. What is the best way to apply trained weights to those layers?
Related
For a concrete example here is some code:
input1 = Input(shape = (3,2))
x1 = LSTM(8)(input1)
x1 = Flatten()(x1)
x1 = Dense(10)(x1)
input2 = Input(shape = (3,2))
x2 = LSTM(8)(input2)
x2 = Flatten()(x2)
x2 = Dense(10)(x2)
x = concatenate([x1, x2])
y = Dense(1)(x)
model = Model([input1, input2], y)
model.compile(optimizer='Adam',
loss='mean_squared_error')
Is it possible to use a Keras generator to fit this model with two numpy arrays?
E.g.
X1 = np.random.randn((100, 2))
X2 = np.random.randn((100, 2))
I am working on a Bert + MLP model for text classification problem. Essentially, I am trying to replace the MLP model with a basic LSTM model.
Is it possible to create a LSTM with embedding? Or, is best to create a LSTM with embedded layer?
More specifically, I am having a hard time trying to create embedded matrix so I can create embedding layer using Bert embedding.
def get_bert_embeddings(dataset='gap_corrected_train',
dataset_path=TRAIN_PATH,
bert_path=BERT_UNCASED_LARGE_PATH,
bert_layers=BERT_LAYERS):
"""Get BERT embeddings for all files in dataset_path and specified BERT layers and write them to file."""
df = None
for file in os.listdir(dataset_path):
if df is None:
df = pd.read_csv(dataset_path+'/'+file, sep='\t')
else:
next_df = pd.read_csv(dataset_path+'/'+file, sep='\t')
df = pd.concat([df, next_df], axis=0)
df.reset_index(inplace=True, drop=True)
for i, layer in enumerate(bert_layers):
embeddings_file = INTERIM_PATH + 'emb_bert' + str(layer) + '_' + dataset + '.h5'
if not os.path.exists(embeddings_file):
print('Embeddings file: ', embeddings_file)
print('Extracting BERT Layer {0} embeddings for {1}...'.format(layer, dataset))
print("Started at ", time.ctime())
emb = get_bert_token_embeddings(df, bert_path, layer)
emb.to_hdf(embeddings_file, 'table')
print("Finished at ", time.ctime())
def build_mlp_model(input_shape):
input_layer = layers.Input(input_shape)
input_features = layers.Input((len(FEATURES),))
x = layers.Concatenate(axis=1, name="concate_layer")([input_layer, input_features])
x = layers.Dense(HIDDEN_SIZE, name='dense1')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.Dropout(DROPOUT, seed=RANDOM)(x)
x = layers.Dense(HIDDEN_SIZE//2, name='dense2')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.Dropout(DROPOUT//2, seed=RANDOM)(x)
x = layers.Dense(HIDDEN_SIZE//4, name='dense3')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.Dropout(DROPOUT//2, seed=RANDOM)(x)
output_layer = layers.Dense(3, name='output', kernel_regularizer = regularizers.l2(LAMBDA))(x)
output_layer = layers.Activation('softmax')(output_layer)
model = models.Model(input=[input_layer, input_features], output=output_layer, name="mlp")
return model
You can create model that uses first the Embedding layer which is followed by LSTM and then Dense.
Such as here:
deep_inputs = Input(shape=(length_of_your_data,))
embedding_layer = Embedding(vocab_size, output_dim = 3000, trainable=True)(deep_inputs)
LSTM_Layer_1 = LSTM(512)(embedding_layer)
dense_layer_1 = Dense(number_of_classes, activation='softmax')(LSTM_Layer_1)
model_AdGroups = Model(inputs=deep_inputs, outputs=dense_layer_1)
I need to create a neural network that approximates a function given its parameters. I give four parameters to my neural network (A, x0, phi, omega) and I want to obtain, as output,
A sin(omega x + phi) + x0
(I need this net as a part of another network)
However, I am not able to train the network as I obtain a very poor convergence. Why is that?
I use a fully connected network with three layers. This is the code
def get_batches(N_batches):
A = tan( random.uniform(low=0.,high=2*pi,size=[N_batches,1]))
x0 = random.randn(N_batches,1)*10
omega = random.uniform(low=0.,high=10*pi, size=[N_batches,1])
phi = random.uniform(low=0.,high=2*pi, size=[N_batches,1])
x = linspace(0,t_max, n_max)
x = tile(x,N_batches).reshape(N_batches,n_max)
return (A*sin(omega*x+phi) + x0, hstack([A,x0,phi,omega]) )
N_batches = 80
N_epochs = 50
t_max = 5.0
n_max = 100
n_par = 4
net_layers = []
net_inp = Input(shape=(n_par,))
net_layers.append(Dense(25, input_shape=(n_par,), activation="relu"))
net_layers.append(Dense(25, activation="relu"))
net_layers.append(Dense(25, activation="relu"))
net_layers.append(Dense(n_max, activation="linear"))
net_l = net_inp
for i in range(len(net_layers)):
net_l = net_layers[i](net_l)
net = Model(net_inp, net_l)
net.compile(loss="mean_squared_error", optimizer="adam")
costs = zeros(N_epochs)
for i in range(N_epochs):
y_true, y_in = get_batches(N_batches)
costs[i]=net.train_on_batch(y_in,y_true)
Even if I train more, I don't get better results than this
picture (approximated function and real function plot for a test sample):
The plot of the cost function is quite strange:
What mistakes did I do? Thank you!
Most tutorials I've followed shows how I can give a single input into the first layer of a Dense network with something like this using Keras:
Inp = Input(shape=(1,))
x = Dense(100, activation='relu', name = "Dense_1")(Inp)
x = Dense(100, activation='relu', name = "Dense_2")(x)
output = Dense(50, activation='softmax', name = "outputL")(x)
However, if I want to provide 2 or more inputs into the first layer of a Dense network, how can I do so with Keras? The idea is just simply to have 2 inputs of x1 and x2, like this:
I've tried something like this which I've modified from snippets found on one of the pages in the Keras documentation:
Inp1 = Input(shape=(1,))
Inp2 = Input(shape=(1,))
Inp = keras.layers.concatenate([Inp1, Inp2])
x = Dense(100, activation='relu', name = "Dense_1")(Inp)
x = Dense(100, activation='relu', name = "Dense_2")(x)
output = Dense(50, activation='softmax', name = "outputL")(x)
res = model.fit([x1_train, x2_train], y_train,
validation_data=([x1_test, x2_test], y_test))
But so the far, the results that I'm getting from the model training appears to have ridiculously low accuracy. Is what I've done what I've actually intended?
I am using a Siamese architecture in my model for a classification task of whether both the inputs are similar.
in1 = Input(shape=(None,), dtype='int32', name='in1')
x1 = Embedding(output_dim=dim, input_dim=n_symbols, input_length=None,
weights=[embedding_weights], name='x1')(in1)
in2 = Input(shape=(None,), dtype='int32', name='in2')
x2 = Embedding(output_dim=dim, input_dim=n_symbols, input_length=None,
weights=[embedding_weights], name='x2')(in2)
l = Bidirectional(LSTM(units=100, return_sequences=False))
y1 = l(x1)
y2 = l(x2)
y = concatenate([y1, y2])
out = Dense(1, activation='sigmoid')(y)
model = Model(inputs=[in1, in2], outputs=[out])
It works correctly as the number of weights to be trained remain the same even when I use a single input. The thing that confused my though, was the tensorboard vizualization of the model.
tensorboard graph
Shouldn't both x1 and x2 map to the same bidirectional node?
Also, what do the 18 and 32 tensors signify?