keras - embedding layer, can I alter values of a trained embedding layer in the pipeline of a model? - keras

I am following examples on this page: https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/
which trains a word embedding on the data using an Embedding layer, like below:
model = Sequential()
model.add(Embedding(100, 8, input_length=max_length))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
# summarize the model
print(model.summary())
the model starts with learning a word embedding from data, for each word, creates a 8-dimension vector.
What I would like to do, is that after this embedding is learned, I want to alter the matrix (or vectors of each word), by adding two more dimensions appended to the end of each vector. I will have another process that computes the values for this two dimensions.
Is there anyway I can do this?
Many thanks in advance


Yes - it's possible. Try to do this using following procedure:
Extract weight matrix:
weight_matrix = model.layers[0].get_weights()[0] # Matrix shape (100, 8).
Append your vectors:
new_weight_matrix = your_append(weight_matrix)
# Be sure that new_weight_matrix has shape of (100, 10)
Build an adjusted copy of your model:
new_model = Sequential()
new_model.add(Embedding(100, 10, input_length=max_length)) # Notice a change
new_model.add(Flatten())
new_model.add(Dense(1, activation='sigmoid'))
(Optional) freeze layers: In case you want to freeze embedding set:
new_model = Sequential()
new_model.add(Embedding(100, 10, input_length=max_length
trainable=False)) # Notice a change
new_model.add(Flatten())
new_model.add(Dense(1, activation='sigmoid'))
Compile a new model:
new_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])


After compile and fit, you need to replace the weights with the new ones:
new_model.layers[0].set_weights(new_weight_matrix)

Related

Saving and applying a keras model to unseen data

I am building a model for my multiclass text classification problem which looks like this:
model = tf.keras.Sequential()
model.add(hub_layer)
for units in [128, 128, 64 , 32]: # automatically adding layers through a for loop
model.add(tf.keras.layers.Dense(units, activation='relu'))
model.add(tf.keras.layers.Dropout(0.3))
model.add(tf.keras.layers.Dense(18, activation='softmax'))
model.summary()
# Optimiser Function
model.compile(optimizer='adam',
loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_data_f,
epochs=20,
validation_data=test_data_f,
verbose=1,
class_weight=class_weights
)
Once the build is done, I would like to do the following:
Save the model
Load the model and apply it to an unseen text (For Example: the column on which I would want to apply this model is df['Unseen_Text]
how can I achieve this?

tensorflow sequential model outputting nan

Why is my code outputting nan? I'm using a sequential model with a 30x1 input vector and a single value output. I'm using tensorflow and python. This is one of my firs
While True:
# Define a simple sequential model
def create_model():
model = tf.keras.Sequential([
keras.layers.Dense(30, activation='relu',input_shape=(30,)),
keras.layers.Dense(12, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(7, activation='relu'),
keras.layers.Dense(1, activation = 'sigmoid')
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
return model
# Create a basic model instance
model = create_model()
# Display the model's architecture
model.summary()
train_labels=[1]
test_labels=[1]
train_images= [[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]]
test_images=[[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]]
model.fit(train_images,
train_labels,
epochs=10,
validation_data=(test_images, test_labels),
verbose=1)
print('predicted:',model.predict(train_images))

You are using SparseCategoricalCrossentropy. It expects labels to be integers starting from 0. So, you have only one label 1, but it means you have at least two categories - 0 and 1. So you need at least two neurons in the last layer:
keras.layers.Dense(2, activation = 'sigmoid')
( If your goal is classification, you should maybe consider to use softmax instead of sigmoid, without from_logits=True )

You're using the wrong loss function for those labels. You need to use BinaryCrossentropy.
Change:
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
To:
model.compile(optimizer='adam',
loss=tf.keras.losses.BinaryCrossentropy,
metrics=[tf.keras.metrics.BinaryAccuracy()])

What to expect from model.predict in Keras?

I am new to Keras and trying to write my first code. I want to understand what 'model.predict' should return. Consider a simple model below.
model = keras.Sequential()
model.add(keras.layers.Dense(12, input_dim=232, activation='relu'))
model.add(keras.layers.Dense(232, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))
# compile the keras model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# fit the keras model on the dataset
model.fit(vSignal, vLabels, epochs=15, batch_size=100 )
# evaluate the keras model
_, accuracy = model.evaluate(vSignal, vLabels)
print('Accuracy: %.2f' % (accuracy*100))
pred=model.predict(vSignalT)
Consider we train the "model" with "vSignal" and "vLabels" as shown above. Now consider that the accuracy of the model as given by model.evaluate is 100%. Now if we give same data 'vSignal' to 'model.predict' should we get the 'vLabels' return?

pred=model.predict(vSignalT) returns a numpy arrays of predictions.
each row consists of one of the vlabels that the model predicted.
for more information refer to here
save return value of fit function:
hist = model.fit(vSignal, vLabels, epochs=15, batch_size=100 );
then check the
hist.history["accuracy"]

Can model.summarize() in Keras (Sequential) - Multi Input (Numerical + n Embeddings) work?

I'm having difficulty printing the model.summary() after using the Sequential class in keras to build a structure like so:
embedding_inputs* numerical_input
\ /
\ /
-- CONCATENATE--
|
DENSE (50) #1
DENSE (50) #2
DENSE (50) #3
DENSE (50) #4
DENSE (1) #output
* embedding_inputs are a bunch of concatenated sequential models from
categorical variables. For the sake of simplicity,
let's pretend there is only one.
I know without the embedding layer(s), my model works and looks fine. But following my addition of an embedding layer and a concatenate layer, I'm told I need to build the model or that my Output tensors "must be the output of a Keras Layer."
I'm just utterly confused at this point. (I'm used to using the functional api but embarrassingly am having so much trouble with the Sequential one and would like to learn).
categorical = Sequential()
categorical.add(Embedding(
input_dim=len(df_train['mon'].astype('category').cat.categories),
output_dim=2,
input_length=1))
categorical.add(Flatten())
numeric = Sequential()
numeric.add(InputLayer(input_shape(1,len(numeric_column_names)),dtype='float32',name='numerical_in'))
model = Sequential()
model.add(Concatenate([numeric,categoric]))
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal')) #output layer (1 number)
If I attempt to use model.summary() without a build:
ValueError: This model has not yet been built. Build the model first by calling build() or calling fit() with some data. Or specify input_shape or batch_input_shape in the first layer for automatic build.
If I attempt to use model.build() first, I get a message like:
ValueError: Output tensors to a Sequential must be the output of a Keras `Layer` (thus holding past layer metadata). Found: None

deep learning data preparation

I have a text dataset, that contains 6 classes. for each sample, I have the percent value and sum of the 6 percent values is 100% (features are related to each other). For example :
{A:16, B:35, C:7, D:0, E:3, F:40}
how can I feed a deep learning algorithm with this dataset?
I actually want the prediction to be exactly in the shape of training data.

Here is what you can do:
First of all, normalize all of your labels and scale them between 0-1.
Use a softmax layer for prediction.
Here is some code in Keras for intuition:
model = Sequential()
model.add(Dense(100, input_dim = x.shape[1], activation='relu'))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')

Resources