I created three Convolutional Autoencoders with the same architecture to extract features from some images related to different types of trees.
My code is something like:
model1 = myAutoencoder()
model2 = myAutoencoder()
model3 = myAutoencoder()
opt = keras.optimizers.Adam(learning_rate=0.001)
loss = keras.losses.MeanSquaredError()
model1.compile(opt=opt, loss=loss)
model2.compile(opt=opt, loss=loss)
model3.compile(opt=opt, loss=loss)
Then I train:
#X1, X2, X3 are tensors of 64x64 RGB images: for example(100, 64,64,3)
model1.fit(X1, X1)
model2.fit(X2, X2)
model3.fit(X3, X3)
However, only the first model is learning, while the second and third are stuck with the same loss as in the figure:
enter image description here
Interestingly, if I swap the positions of let's say model1 and model2, like this:
model2.fit(X2, X2)
model1.fit(X1, X1)
model3.fit(X3, X3)
then only model 2 is learning and models 1 and 3 are stuck. I cannot figure out why...
edit: The actual training that I am doing is this:
def scheduler(epoch, lr):
if epoch < 50:
return lr
else:
return lr * np.math.exp(-0.1)
model2.fit(X2, X2, epochs=100, callbacks=[LearningRateScheduler(scheduler)])
model1.fit(X1, X1, epochs=100, callbacks=[LearningRateScheduler(scheduler)])
model3.fit(X3, X3, epochs=100, callbacks=[LearningRateScheduler(scheduler)])
I figured out that if I delete the callbacks the learning process is "normal", is there a reason why the callbacks are interfering between models?
Related
I am working on a classification problem using (1D-CNN) the model I built and the data loaders are working perfectly. However, during the training and the validation loops I encounter an issue.
I have a multilabel classification problem and before training the model on all 4 labels I have, I did a splitting and encoding for the label data so that each label will trained individually. the code below shows the splitting and the encoding steps are shown in the first 12 lines in the for loop below.
executing the code above within the training loop did the job for me. However, when I move onto the training functions, the model will train the outputs to match the first label group only, and when I observe the values of the output for the second, third and fourth run, the values are the same. the code below shows the training step: (the code below is included in the for loop mentioned above):
for epoch in tqdm(range(50)):
for batch, (features, labels) in enumerate(train_loader):
#encoding the labels
Ordinal_Labels = encoder.fit_transform(labels)
y1 = torch.from_numpy(Ordinal_Labels)
y2 = y1.to(torch.int64)
y_train = functional.one_hot(y2)
x1 = y_train[:, 0:1, :]
x2 = y_train[:, 1:2, :]
x3 = y_train[:, 2:3, :]
x4 = y_train[:, 3:4, :]
label1 = torch.squeeze(x1, 1).to(torch.float64)
label2 = torch.squeeze(x2, 1).to(torch.float64)
label3 = torch.squeeze(x3, 1).to(torch.float64)
label4 = torch.squeeze(x4, 1).to(torch.float64)
optimizer.zero_grad()
#training the model
output1= model(features)
output2= model(features)
output3= model(features)
output4= model(features)
#obtaining the loss for each label
loss1= criterion(output1, label1)
loss2= criterion(output2, label2)
loss3= criterion(output3, label3)
loss4= criterion(output4, label4)
#backward pass tp update the model parameters
loss1.backward()
loss2.backward()
loss3.backward()
loss4.backward()
#updating the grad parameters after the backward propagation
optimizer.step()
#obtaining the total loss value
loss= loss4 #+ loss2 + loss3 + loss4
epoch_loss.append(loss.item())
I tried building separate functions for each label on which the model will be executed on a one label group, however, running all 4 function together in the same for loop gives the same error.
Any help in such a problem is much appreciated
I found in official doc that CAddTable should be done as
x = x1 + x2 # instead of CAddTable(x1, x2) in older version
and PyTorch would do the rest of things like autograd
But how about if I have multiple tensors, aka. changing the input above from two tensors to a list of tensors. Could PyTorch still do the similar things?
Just for a clean display of the code snip in the comment:
x = torch.stack((x1, x2, x3, x4), dim=0)
y = torch.sum(x, dim=0, keepdim=False) # same shape as x1, x2...
I have data of the form
y1, x1
y2, x2
...
I am trying to train a neural network function say f such that over a set of pairs xi, xj, f(xi) - f(xj) is trained to be similar to (yi - yj) i.e. pairwise loss
You need to reconstruct the data, for each pair Xi, Xj the label will be Yi-Yj. since the problem is a regression problem use MSE as a loss function and that hopefully will lead to what you want.
I am using a Siamese architecture in my model for a classification task of whether both the inputs are similar.
in1 = Input(shape=(None,), dtype='int32', name='in1')
x1 = Embedding(output_dim=dim, input_dim=n_symbols, input_length=None,
weights=[embedding_weights], name='x1')(in1)
in2 = Input(shape=(None,), dtype='int32', name='in2')
x2 = Embedding(output_dim=dim, input_dim=n_symbols, input_length=None,
weights=[embedding_weights], name='x2')(in2)
l = Bidirectional(LSTM(units=100, return_sequences=False))
y1 = l(x1)
y2 = l(x2)
y = concatenate([y1, y2])
out = Dense(1, activation='sigmoid')(y)
model = Model(inputs=[in1, in2], outputs=[out])
It works correctly as the number of weights to be trained remain the same even when I use a single input. The thing that confused my though, was the tensorboard vizualization of the model.
tensorboard graph
Shouldn't both x1 and x2 map to the same bidirectional node?
Also, what do the 18 and 32 tensors signify?
I have a dataset with features and their labels.
it looks like this:
X1, X2, X3, X4, X5 .. Xn L1, L2, L3
Y1, Y2, Y3, Y4, Y5 .. Yn L5, L2
..
I want to train a KNeighborsClassifier on this dataset. It seems like sklearn does not take multilabels. I have been trying this:
mlb = MultiLabelBinarizer()
Y = mlb.fit_transform(Y)
# parameters: n_neighbors=[5,15], weights = 'uniform', 'distance'
bagging = BaggingClassifier(KNeighborsClassifier(n_neighbors =5,weights ='uniform'), max_samples = 0.6, max_features= 0.7, verbose =1, oob_score =True)
scores = cross_val_score(bagging, X, Y, verbose =1, cv=3, n_jobs=3, scoring='f1_macro')
It is giving me ValueError: bad input shape
Is there a way that I can run multilabel classifier in sklearn?
According to sklearn documentation the classifiers that support multioutput-multiclass classification tasks are:
Decision Trees, Random Forests, Nearest Neighbors
Since you have a binary matrix for your labels, you can use OneVsRestClassifier to make your BaggingClassifier handle multilabel predictions. Code should now look like:
bagging = BaggingClassifier(KNeighborsClassifier(n_neighbors=5, weights='uniform'), max_samples=0.6, max_features=0.7, verbose=1, oob_score=True)
clf = OneVsRestClassifier(bagging)
scores = cross_val_score(clf, X, Y, verbose=1, cv=3, n_jobs=3, scoring='f1_macro')
You can use the OneVsRestClassifier with any of the sklearn models to do multilabel classification.
Here's an explanation:
http://scikit-learn.org/stable/modules/multiclass.html#one-vs-the-rest
And here are the docs:
http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html
For anybody who finds this looking for multi-label KNN (MLKNN) options, I would recommend using skmultilearn, which is built on top of sklearn, so easy to use if you are familiar with the latter package.
Documentation here. This example is from the documentation:
from skmultilearn.adapt import MLkNN
classifier = MLkNN(k=3)
# train
classifier.fit(X_train, y_train)
# predict
predictions = classifier.predict(X_test)