In an attempt to apply 3 different activation functions to the 3 columns of the last Dense layer I'm using Lambda layers. First I split and then reassemble the output. No actual activation is applied as of yet, because I found that the following gives me an error at the model fit step:
def createModel():
in_put = Input(shape=(3300,))
layer1 = Dense(512, activation='relu')(in_put)
layer2 = Dense(1024, activation='relu')(layer1)
layer3 = Dense(32, activation='relu')(layer2)
out_put = Dense(3)(layer3)
#splitting output into 3 columns for further activation
theta = Lambda(lambda x: x[:,0], output_shape=(1,))(out_put)
phi = Lambda(lambda x: x[:,1], output_shape=(1,))(out_put)
r = Lambda(lambda x: x[:,2], output_shape=(1,))(out_put)
#combining the activated layers (activation has not been implemented yet)
out_put_a = Lambda(lambda x: K.stack([x[0], x[1], x[2]]),output_shape=(3,), name="output")([theta, phi, r])
model = Model(inputs=in_put, outputs=out_put_a)
return model
my_network=createModel()
batch_size = 1000
epochs = 1
my_network.compile(optimizer='rmsprop', loss='mean_squared_error')
history = my_network.fit_generator(generator=training_generator, epochs=epochs,
validation_data=testing_generator)
It crashes with the following error:
InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Incompatible shapes: [3] vs. [1000]
[[{{node training/RMSprop/gradients/loss/output_loss/mul_grad/BroadcastGradientArgs}}]]
[[loss/mul/_55]]
(1) Invalid argument: Incompatible shapes: [3] vs. [1000]
[[{{node training/RMSprop/gradients/loss/output_loss/mul_grad/BroadcastGradientArgs}}]]
0 successful operations.
0 derived errors ignored.>
At the same time the code below (with no Lambda layers works just fine):
def createModel():
in_put = Input(shape=(3300,))
layer1 = Dense(512, activation='relu')(in_put)
layer2 = Dense(1024, activation='relu')(layer1)
layer3 = Dense(32, activation='relu')(layer2)
out_put = Dense(3)(layer3)
model = Model(inputs=in_put, outputs=out_put)
return model
my_network=createModel()
batch_size = 1000
epochs = 1
my_network.compile(optimizer='rmsprop', loss='mean_squared_error')
history = my_network.fit_generator(generator=training_generator, epochs=epochs,
validation_data=testing_generator)
Could anyone point out what's wrong with the lambda layer?
Related
This is my model
engine1 = tf.keras.applications.Xception(
# Freezing the weights of the top layer in the InceptionResNetV2 pre-traiined model
include_top = False,
# Use Imagenet weights
weights = 'imagenet',
# Define input shape to 224x224x3
input_shape = (256, 256 , 3)
)
x1 = tf.keras.layers.GlobalAveragePooling2D(name = 'avg_pool')(engine1.output)
x1 =tf.keras.layers.Dropout(0.75)(x1)
x1 = tf.keras.layers.BatchNormalization(
axis=-1,
momentum=0.99,
epsilon=0.01,
center=True,
scale=True,
beta_initializer="zeros",
gamma_initializer="ones",
moving_mean_initializer="zeros",
moving_variance_initializer="ones",
)(x1)
out1 = tf.keras.layers.Dense(3, activation = 'softmax', name = 'dense_output')(x1)
# Build the Keras model
model1 = tf.keras.models.Model(inputs = engine1.input, outputs = out1)
# Compile the model
model1.compile(
# Set optimizer to Adam(0.0001)
optimizer = tf.keras.optimizers.Adam(learning_rate= 3e-4),
#optimizer= SGD(lr=0.001, decay=1e-6, momentum=0.99, nesterov=True),
# Set loss to binary crossentropy
#loss = tf.keras.losses.SparseCategoricalCrossentropy(),
loss = 'categorical_crossentropy',
# Set metrics to accuracy
metrics = ['accuracy']
)
I want logits so I wrote this
logits = model1(X_test)
probs = tf.nn.softmax(logits)
Getting error as
ResourceExhaustedError: OOM when allocating tensor with shape[1288,64,125,125] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Conv2D]
How to fix this and get the logits? I want to apply the distillation method after getting the logits. My test set consists of 3 classes and 60 samples.
so logit matrix should be a matrix of 60 * 3.
Edit
To get the logits(1288 * 3) I made a change in the output layer of my model
out1 = tf.keras.layers.Dense(3, activation = 'linear', name = 'dense_output')(x1)
Now I am getting logits,
y_pred_logits = model1.predict(X_test)
I want to apply softmax on this, My softmax function looks like this,
def softmax(x):
"""Compute softmax values for each sets of scores in x."""
e_x = np.exp(x)
return e_x / e_x.sum(axis=1)
But when I am doing this
y_pred_logits_activated = softmax(y_pred_logits)
Getting errors as
How to fix this and is this method correct? Further, I want to apply this on logits
When trying to connect LSTM with Dense, it gives an error (when trying to train):
input = Input(shape=(x_train.shape[1], None))
X = Embedding(num_words, max_article_len)(input)
X = LSTM(128, return_sequences=True, dropout = 0.5)(X)
X = LSTM(128)(X)
X = Dense(32, activation='softmax')(X)
model = Model(inputs=[input], outputs=[X])
...
>>> ValueError: Error when checking target: expected dense to have shape (32,) but got array with shape (1,)
I tried different connection options, but the error repeats:
X, h, c = LSTM(128, return_sequences=False, return_state=True, dropout = 0.5)(X)
X = Dense(32, activation='softmax')(X)
>>> ValueError: Error when checking target: expected dense to have shape (32,) but got array with shape (1,)
Any solution options on the functional API / Sequential?
Data conversion code:
train = pd.read_csv('train.csv')
articles = train['text']
y_train = train['lang']
num_words = 50000
max_article_len = 20
tokenizer = Tokenizer(num_words=num_words)
tokenizer.fit_on_texts(articles)
sequences = tokenizer.texts_to_sequences(articles)
x_train = pad_sequences(sequences, maxlen=max_article_len, padding='post')
x_train.shape
>>> (18974, 100)
y_train.shape
>>> (18974,)
The last parameter must be set to False;
X = LSTM(128, return_sequences=True, dropout = 0.5)(X)
X = LSTM(128, return_sequences=False)(X)
If you still have issues, then the problem must be with your input shape.
I am trying to train a 2D neural network using keras. I have a weird error message, "ValueError: setting an array element with a sequence." when I try to use model.fit function in keras. Specifically, the error says that my "tensor_train_labels" is a sequence instead of an array. But my labels are indeed numpy arrays (not a sequence). I am not sure why does keras complain about it ?
I am following this tutorial for building my network
tensor_train_data.shape
#TensorShape([Dimension(209), Dimension(64), Dimension(64), Dimension(3)])
tensor_test_data.shape
#TensorShape([Dimension(50), Dimension(64), Dimension(64), Dimension(3)])
tensor_train_labels = tf.reshape(tensor_train_labels, [209,1])
tensor_test_labels = tf.reshape(tensor_test_labels, [50,1])
batch_size = 10
epochs = 8
model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv2D(32, kernel_size=(3,3), activation='relu',
input_shape=(64, 64, 3)))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2,2)))
model.add(tf.keras.layers.Dropout(0.25))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation = 'relu'))
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(2, activation = 'softmax'))
model.compile(loss='categorical_crossentropy', optimizer =
tf.keras.optimizers.Adam(lr=0.0001, decay=1e-6), metrics=['accuracy'])
model.fit(tensor_train_data/255.0,
tf.keras.utils.to_categorical(tensor_train_labels),
batch_size = batch_size,
shuffle = True,
epochs = epochs,
validation_data = (tensor_test_data/ 255.0,
tf.keras.utils.to_categorical(tensor_test_labels)))
scores = model.evaluate(tensor_test_labels/ 255.0,
tf.keras.utils.to_categorical(tensor_test_labels))
print('Loss: %.3f' % scores[0])
print('Accuracy: %.3f' % scores[1])
The Error :
ValueError Traceback (most recent call last)
<ipython-input-224-80431a1b3e79> in <module>
1 model.compile(loss='categorical_crossentropy', optimizer = tf.keras.optimizers.Adam(lr=0.0001, decay=1e-6), metrics=['accuracy'])
----> 2 model.fit(tensor_train_data/255.0, tf.keras.utils.to_categorical(tensor_train_labels),
3 batch_size = batch_size,
4 shuffle = True,
5 epochs = epochs,
~\AppData\Local\conda\conda\envs\deeplearning\lib\site-packages\tensorflow\python\keras\utils\np_utils.py in to_categorical(y,
num_classes)
37 last.
38 """
---> 39 y = np.array(y, dtype='int')
40 input_shape = y.shape
41 if input_shape and input_shape[-1] == 1 and len(input_shape) > 1:
ValueError: setting an array element with a sequence.
The possible error is that you have arrays of different sizes when you are trying to convert it into the numpy array. Possible solution : https://stackoverflow.com/a/49617425/8185479
I am attempting to wrap my keras models in scikit learn GridSearchCV and Pipeline structures for hyperparameter tuning.
It works absolutely fine when the build_fn function takes 0 arguments for use in KerasClassifier. However it fails whenever I use a function which takes arguments.
Example code below
def prepare_classifier(x, y):
shape_of_input = x.shape
shape_of_target = y.shape
classifier = Sequential()
## number of neurons = 30
## kernel_initializer determines how the weights are initialized
## activation is the activation function at this particular hidden layer
## input_shape is the number of features in a single row.. in this case it is shape_of_input[1]
## shape_of_input[0] is the total number of such rows
classifier.add(Dense(units = 30, activation = 'relu', kernel_initializer = 'uniform', input_dim = shape_of_input[1]))
classifier.add(Dense(units = 30, activation = 'relu', kernel_initializer = 'uniform'))
## we are predicting 10 digits for each row of x.
## in total there are shape_of_input[0] rows in total
classifier.add(Dense(10, activation = 'softmax'))
## categorical_crossentropy is the loss function for multi output loss function
classifier.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
return classifier
def fit(classifier, x_train, y_train, epoch_size, batch_size = 10):
pipeline = Pipeline([
('keras_classifier', classifier)
])
param_grid = {
'keras_classifier__batch_size' : [10,20,30,50],
'keras_classifier__epochs' : [100, 200, 300],
'keras_classifier__x' : [x_train],
'keras_classifier__y' : [y_train],
}
grid = GridSearchCV(estimator = pipeline, param_grid = param_grid, n_jobs = -1)
grid.fit(x_train, y_train)
print("Best parameters are : ", grid.best_params_, '\n grid best score :', grid.best_score_)
classifier = KerasClassifier(build_fn = prepare_classifier, x = x_train[0:100], y = y_train )
fit(classifier, x_train[:100], y_train, epoch_size )
This is for some x, and some y data (p.s. I have used mnist data)
The error I get is :
RuntimeError: Cannot clone object , as the constructor either does not set or modifies parameter x
However if my prepare_classifier function takes no arguments code works absolutely fine.
What am I doing incorrectly?
solved it. essentially the below line was the issue
classifier = KerasClassifier(build_fn = prepare_classifier, x = x_train[0:100], y = y_train )
needed to be changed to
classifier = KerasClassifier(build_fn = prepare_classifier)
and the parameters for the prepare_classifier needs to be sent using param_grid
model = Sequential()
model.add(Embedding(630, 210))
model.add(LSTM(1024, dropout = 0.2, return_sequences = True))
model.add(LSTM(1024, dropout = 0.2, return_sequences = True))
model.add(Dense(210, activation = 'softmax'))
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
filepath = 'ner_2-{epoch:02d}-{loss:.5f}.hdf5'
checkpoint = ModelCheckpoint(filepath, monitor = 'loss', verbose = 1, save_best_only = True, mode = 'min')
callback_list = [checkpoint]
model.fit(X, y , epochs = 20, batch_size = 1024, callbacks = callback_list)
X: the input vector is of the shape (204564, 630, 1)
y: the target vector is of the shape (204564, 210, 1)
i.e. for every 630 inputs 210 outputs have to be predicted but the code throws the following error on compilation
ValueError Traceback (most recent call last)
<ipython-input-57-05a6affb6217> in <module>()
50 callback_list = [checkpoint]
51
---> 52 model.fit(X, y , epochs = 20, batch_size = 1024, callbacks = callback_list)
53 print('successful')
ValueError: Error when checking model input: expected embedding_8_input to have 2 dimensions, but got array with shape (204564, 630, 1)
Please someone explain why this error is occurring and how to solve this
The message says:
Your first layer expects an input with 2 dimensions: (BatchSize, SomeOtherDimension). But your input has 3 dimensions (BatchSize=204564,SomeOtherDimension=630, 1).
Well... remove the 1 from your input, or reshape it inside the model:
Solution 1 - Removing it from the input:
X = X.reshape((204564,630))
Solution 2 - Adding a reshape layer:
model = Sequential()
model.add(Reshape((630,),input_shape=(630,1)))
model.add(Embedding.....)