I have a tensor which size is (batch_size, seqLength, seqLength, label_number), and a mask tensor with size (batch_size, seqLength), when I calculate loss, I need to mask out the word that I padding in. My question is how to use this 2d mask to masking out the padding samples in the 4d tensor so that I can calculate the right loss?
Related
I am using a GPT2 model that outputs logits (before softmax) in the shape (batch_size, num_input_ids, vocab_size) and I need to compare it with the labels that are of shape (batch_size, num_input_ids) to calculate BCELoss. How do I calculate it?
logits = output.logits #--of shape (32, 56, 592)
logits = torch.nn.Softmax()(logits)
labels = labels #---------of shape (32, 56)
torch.nn.BCELoss()(logits, labels)
but the dimensions do not match, so how do I contract logits to labels shape or expand labels to logits shape?
Binary cross-entropy is used when the final classification layer is a sigmoid layer, i.e., for each output dimension, only a true/false output is possible. You can imagine it as assigning some tags to the input. This also means that the labels need to have the same dimension as the logits, having 0/1 for each logit. Statistically speaking, for 592 output dimensions, you predict 592 Bernoulli (= binary) distributions. The expected shape is 32 × 56 × 592.
When using the softmax layer, you assume only one target class is possible; you predict a single categorical distribution over 592 possible output classes. However, in this case, the correct loss function is not binary cross-entropy but categorical cross-entropy, implemented by the CrossEntropyLoss class in PyTorch. Note that it takes the logits directly before the softmax normalization and does the normalization internally. The expected shape is 32 × 56, as in the code snippet.
I have an input tensor of shape: data (x,y,z).
I have a binary mask tensor of shape: mask (x,y).
When I do data[mask > 0] I obtain a new tensor of shape (q,z) where q is the number of ones in the mask tensor.
I would instead want to get a tensor of original shape (x,y,z) but for the values for which we have zeros in mask being eliminated and instead being padded at the end with 0 in data tensor so we keep the original shape ( the reason it doesn't do that now is because we would have variable length across second dimension).
Of course this can be easily done in python with basic matrix operations, but is there an efficient tensor-way to do it in pytorch?
Example (imagine a,b,c...are 1D tesnors):
data
[[a,b,c],
[d,e,f]]
mask
[[0,1,0],
[1,0,1]]
Ideal output:
[[b,junk,junk],
[d,f, junk]]
Here, the missing stuff is padded with some "junk" to keep the original shape.
I have an y_train (4D) matrix which content is (33, 224, 224, 1) where (n_images, n_pixels_y, n_pixels_x, n_bands) that needed to be transformed into a 2D vector using Flatten
now i need to do the inverse process and turn the 2D vector back to the 4D matrix with the same dimensions, what is the best way to do it?
I've been trying to use reshape but it's not being very effective
Ex:
y_pred = modelo.predict(x_test)
print(y_pred.shape) #this results in the current shape that is (33, 50176)
img = np.reshape(y_pred[1], (33,224,224,1))
plt.imshow(img, cmap='gray')
plt.show() #this results in a error 'cannot reshape array of size 50176 into shape (33,224,224,1)'
I had difficulty finding information on reshaping in PyTorch. Tensorflow is quite easy.
My tensor has shape torch.Size([3, 480, 480]).
I want to convert it to a 4D tensor with shape [1,3,480,480].
How do I do that?
You can use unsqueeze()
For example:
x = torch.zeros((4,4,4)) # Create 3D tensor
x = x.unsqueeze(0) # Add dimension as the first axis (1,4,4,4)
I've seen a few people use indexing with None to add a singular dimension as well. For example:
x = torch.zeros((4,4,4)) # Create 3D tensor
print(x[None].shape) # (1,4,4,4)
print(x[:,None,:,:].shape) # (4,1,4,4)
print(x[:,:,None,:].shape) # (4,4,1,4)
print(x[:,:,:,None].shape) # (4,4,4,1)
Personally, I prefer unsqueeze(), but it's good to be familiar with both.
I was just modifying some an LSTM network I had written to print out the test error. The issues, I realized, is that the model I had defined depends on the batch size.
Specifically, the input is a tensor of shape [batch_size, time_steps, features]. The input enters the LSTM cell and the output, which I turn into a list of time_steps 2D tensors, with each 2D tensor having shape [batch_size, hidden_units]. Each 2D tensor is then multiplied by a weight vector of shape [hidden_units] to yield a vector of shape [batch_size] which has added to it a bias vector of shape [batch_size].
In words, I give the model N sequences, and I expect it to output a scalar for each time step for each sequence. That is, the output is a list of N vectors, one for each time step.
For training, I give the model batches of size 13. For the test data, I feed the entire data set, which consists of over 400 examples. Thus, an error is raised, since the bias has fixed shape batch_size.
I haven't found a way to make it's shape variable without raising an error.
I can add complete code if requested. Added code anyways.
Thanks.
def basic_lstm(inputs, number_steps, number_features, number_hidden_units, batch_size):
weights = {
'out': tf.Variable(tf.random_normal([number_hidden_units, 1]))
}
biases = {
'out': tf.Variable(tf.constant(0.1, shape=[batch_size, 1]))
}
lstm_cell = rnn.BasicLSTMCell(number_hidden_units)
init_state = lstm_cell.zero_state(batch_size, dtype=tf.float32)
hidden_layer_outputs, states = tf.nn.dynamic_rnn(lstm_cell, inputs,
initial_state=init_state, dtype=tf.float32)
results = tf.squeeze(tf.stack([tf.matmul(output, weights['out'])
+ biases['out'] for output
in tf.unstack(tf.transpose(hidden_layer_outputs, (1, 0, 2)))], axis=1))
return results
You want the biases to be a shape of (batch_size, )
For example (using zeros instead of tf.constant but similar problem), I was able to specify the shape as a single integer:
biases = tf.Variable(tf.zeros(10,dtype=tf.float32))
print(biases.shape)
prints:
(10,)