I'm a student who just started study pytorch.
I hope to generate pairwise cross product matrix.
Example is below.
x=torch.Tensor([[1,0,0],[0,1,0]]) # 2*3
y=torch.Tensor([[2,0,0],[0,2,0],[0,0,2]]) # 3*3
#what I want to get is below using torch.cross
z=torch.Tensor([[[0,0,0],[0,0,2],[0,-2,0]],\
[0,0,-2],[0,0,0],[2,0,0]]]) # 2*3*3
#So I write code
tmp_x=0*y+x.unsqueeze(1)# 2*3*3 , horrible coding.. please modify this.
tmp_y=y+0*x.unsqueeze(1)
z=torch.cross(tmp_x,tmp_y,dim=-1)
I think my code is not good. May I have advice for more good code?
And there is precision problem, when I use it for float type. Cross product between a vector and itself is not zero-vector.
When I print each matrix in above code like below, in this case x=y,
print x
print y
print '====='
print tmp_x
print tmp_y
print '====='
print z
then I got result.
tensor([[ 0.2957, 1.6024, -0.1010],
[-1.4755, 0.1937, -0.6091],
[ 1.3540, 0.7203, 0.4037]])
tensor([[ 0.2957, 1.6024, -0.1010],
[-1.4755, 0.1937, -0.6091],
[ 1.3540, 0.7203, 0.4037]])
=======
tensor([[[ 0.2957, 1.6024, -0.1010],
[ 0.2957, 1.6024, -0.1010],
[ 0.2957, 1.6024, -0.1010]],
[[-1.4755, 0.1937, -0.6091],
[-1.4755, 0.1937, -0.6091],
[-1.4755, 0.1937, -0.6091]],
[[ 1.3540, 0.7203, 0.4037],
[ 1.3540, 0.7203, 0.4037],
[ 1.3540, 0.7203, 0.4037]]])
tensor([[[ 0.2957, 1.6024, -0.1010],
[-1.4755, 0.1937, -0.6091],
[ 1.3540, 0.7203, 0.4037]],
[[ 0.2957, 1.6024, -0.1010],
[-1.4755, 0.1937, -0.6091],
[ 1.3540, 0.7203, 0.4037]],
[[ 0.2957, 1.6024, -0.1010],
[-1.4755, 0.1937, -0.6091],
[ 1.3540, 0.7203, 0.4037]]])
=======
tensor([[[ 5.0209e-09, -2.8436e-10, -1.0191e-08],
[-9.5639e-01, 3.2915e-01, 2.4217e+00],
[ 7.1970e-01, -2.5616e-01, -1.9567e+00]],
[[ 9.5639e-01, -3.2915e-01, -2.4217e+00],
[ 6.0023e-10, -2.4257e-08, 1.3432e-08],
[ 5.1692e-01, -2.2897e-01, -1.3251e+00]],
[[-7.1970e-01, 2.5616e-01, 1.9567e+00],
[-5.1692e-01, 2.2897e-01, 1.3251e+00],
[ 9.2649e-09, 5.7705e-09, -4.0787e-09]]])
z[0,0,:] must be zero vector, but it is not. What did I wrong?
Related
I would like to implement a point-convolution to my input tensor.
Let's say we have:
input_ = torch.randn(1,1,8,1)
input_.shape
Out[70]: torch.Size([1, 1, 8, 1])
input_
Out[71]:
tensor([[[[ 0.7656],
[-0.3400],
[-0.2487],
[ 0.6246],
[ 2.0450],
[-0.9588],
[ 1.2221],
[-1.3164]]]])
where the dimensions represent respectively (batch_size, n_channels, height, width).
Then, (keeping fixed batch_size and channel), I would like to apply to each a nn.Conv1d layer basically to expand the number of channels. What I've tried has been:
list(torch.unbind(input_,dim=2))
Out[72]:
[tensor([[[0.7656]]]),
tensor([[[-0.3400]]]),
tensor([[[-0.2487]]]),
tensor([[[0.6246]]]),
tensor([[[2.0450]]]),
tensor([[[-0.9588]]]),
tensor([[[1.2221]]]),
tensor([[[-1.3164]]])]
and then applying nn.Conv1d entrywise to these elements? Would that be a correct way?
EDIT:
Should I use nn.Conv1d directly on my original input_ by doing
conv = nn.Conv1d(in_channels=1, out_channels=3, kernel_size=(1,1)
conv(input_)
conv(input_)
Out[89]:
tensor([[[[-0.1481],
[ 1.5345],
[ 0.3082],
[ 1.8677],
[ 0.7515],
[ 1.2916],
[ 0.0218],
[ 0.5606]],
[[-1.2975],
[-0.3080],
[-1.0292],
[-0.1121],
[-0.7685],
[-0.4509],
[-1.1976],
[-0.8808]],
[[-0.7169],
[ 0.6493],
[-0.3464],
[ 0.9199],
[ 0.0135],
[ 0.4521],
[-0.5790],
[-0.1415]]]], grad_fn=<ThnnConv2DBackward0>)
conv(input_).shape
Out[90]: torch.Size([1, 3, 8, 1])
I'm not sure though if doing this is the same as my original purpose.
I have something like this model=BertModel.from_pretrained('bert-base-uncased',return_dict=True)
What exactly is this "return_dict" used for? What happens when True and what when False?
When instantiating a BertModel, the default output of the model when evaluating or predicting will be a tuple consisting of loss, logits, hidden_states and attentions.
predictions = model(ids_tensor)
print(predictions)
# MaskedLMOutput(loss=None, logits=tensor([[
# [ -0.2506, -5.6671, -5.1753, ..., -5.3228, -7.9154, -4.5786],
# [ -4.1528, -8.2391, -8.5691, ..., -8.4557, -8.2903, -10.1395],
# [-15.5995, -17.0001, -16.9896, ..., -14.1423, -15.6004, -15.8228],
# ...,
# [ 3.0180, -2.9339, -3.3522, ..., -4.1684, -4.9487, -1.7176],
# [-12.7654, -12.9510, -12.9151, ..., -10.5786, -11.1695, -9.6117],
# [ -4.0356, -9.7091, -9.5329, ..., -9.3969, -10.5371, -9.2839]]]),
# hidden_states=None, attentions=None)
If the argument return_dict is defined as True, the output changes into a 'ModelOutput', called like this in the documentation of HuggingFace. This output consist on the elements last_hidden_state, hidden_states, pooler_output, past_key_values, attentionsand cross_attentions. Hope I could be of help.
model = BertModel.from_pretrained('bert-base-uncased',return_dict=True)
predictions = model(ids_tensor)
print(predictions)
# BaseModelOutputWithPoolingAndCrossAttentions(last_hidden_state=tensor([[
# [ 0.0769, -0.0024, 0.0389, ..., -0.0489, 0.0484, 0.4760],
# [-0.1383, -0.3266, 0.2738, ..., -0.0745, 0.0224, 0.8426],
# [-0.4573, -0.0621, 0.4206, ..., 0.0188, 0.1578, 0.4477],
# ...,
# [ 0.7070, -0.1623, 0.4451, ..., -0.1530, 0.0902, 0.8289],
# [ 0.7154, 0.0767, -0.2292, ..., 0.2946, -0.5152, -0.2444],
# [ 0.3558, 0.1660, 0.0459, ..., 0.5960, -0.7525, -0.0851]]]),
# pooler_output=tensor([[-7.4716e-01, -1.4339e-01, …, 7.6550e-01]]),
# hidden_states=None,
# past_key_values=None,
# attentions=None,
# cross_attentions=None)
Source: Bert Documentation in Transformers - HuggingFace
Code is taking too much time for computation and I want to reduce number of iterations. I tried to use method suggested by #Albuquerque How to optimise the following for loop code?, but for this code I have 3d array. Please suggest how to optimize following code.
K=2
D=3
N=3
sigma=np.asarray([[1,2,3],[4,5,6]])
F=np.asarray([[1,2,3],[4,5,6]])
X=np.asarray([[1,2,3],[4,5,6],[7,8,9]])
W=
[
[
[1,2,3],
[3,2,1]
],
[
[1,1,19],
[1,2,1]
],
[
[2,2,2],
[1,3,5]
]
]
result2= np.ones([N, D])
for i in range(N):
for l in range(D):
result2[i][l]=np.sum(W[i][k][l]*(F[k][l]+sigma[k][l]*X[i][l]) for k in range(K))
output-
array([[ 26., 42., 60.],
[ 25., 72., 441.],
[ 48., 171., 360.]])
For this tensor is PyTorch,
tensor([[ 0.7646, 0.5573, 0.4000, 0.2188, 0.7646, 0.5052, 0.2042, 0.0896,
0.7667, 0.5938, 0.3167, 0.0917],
[ 0.4271, 0.1354, 0.5000, 0.1292, 0.4260, 0.1354, 0.4646, 0.0917,
-1.0000, -1.0000, -1.0000, -1.0000],
[ 0.7208, 0.5656, 0.3000, 0.1688, 0.7177, 0.5271, 0.1521, 0.0667,
0.7198, 0.5948, 0.2438, 0.0729],
[ 0.6292, 0.8250, 0.4000, 0.2292, 0.6271, 0.7698, 0.2083, 0.0812,
0.6281, 0.8604, 0.3604, 0.0917]], device='cuda:0')
How can I extract to new Tensor for those values
0.7646, 0.5573, 0.4000, 0.2188
0.4271, 0.1354, 0.5000, 0.1292
How to get the first 4 of two rows into a new tensor?
Actually the question was answered from #zihaozhihao in the Comments but in case you are wondering where that comes from it would be helpful if you structured your Tensor like this:
x = torch.Tensor([
[ 0.7646, 0.5573, 0.4000, 0.2188, 0.7646, 0.5052, 0.2042, 0.0896, 0.7667, 0.5938, 0.3167, 0.0917],
[ 0.4271, 0.1354, 0.5000, 0.1292, 0.4260, 0.1354, 0.4646, 0.0917, -1.0000, -1.0000, -1.0000, -1.0000],
[ 0.7208, 0.5656, 0.3000, 0.1688, 0.7177, 0.5271, 0.1521, 0.0667, 0.7198, 0.5948, 0.2438, 0.0729],
[ 0.6292, 0.8250, 0.4000, 0.2292, 0.6271, 0.7698, 0.2083, 0.0812, 0.6281, 0.8604, 0.3604, 0.0917]
])
so now it is more clear that you have a shape (4, 12) you can think about it like an excel file, you have 4 rows and 12 columns. Now what you want is to extract from the two first rows the 4 first columns and that's why your solution would be:
x[:2, :4] # 2 means you want to take all the rows until the second row and then you set that you want all the columns until the fourth column, this Code will also give the same result x[0:2, 0:4]
I want to know how do I select channels by the mask in Pytorch.
[channel1 channel2 channel3 channel4] x [1,0,0,1] --> [channel1,channel4]
I tried torch.masked_select() and it did't work.
if the input has a shape like [B,C,H,W] the output's shape should be [B,masked_C,H,W],
import torch
from torch import nn
input = torch.randn((1,5,3,3))
pool = nn.AdaptiveAvgPool2d(1)
w = torch.sigmoid(pool(input)).view(1,-1)
mask = torch.gt(w,0.5)
print(input)
print(w)
print(mask)
the output is as following:
tensor([[[[ 0.9129, -0.9763, 1.4460],
[ 0.3608, 0.5561, -1.4612],
[ 1.4953, -1.2474, 0.4069]],
[[-0.9121, 0.1261, 0.4661],
[-1.1624, -1.0266, -1.5419],
[ 1.0644, 1.0039, -0.4022]],
[[-1.8454, -0.2150, 2.3703],
[ 0.5224, 0.3366, 1.7545],
[-0.4624, 1.2639, 1.8032]],
[[-1.1558, -1.9985, -1.1336],
[-0.4400, -0.2092, 0.0677],
[-0.4172, -0.3614, -1.3193]],
[[-0.9441, -0.2944, 0.3381],
[ 1.6562, -0.5623, 0.0599],
[ 0.7229, 0.0472, -0.5122]]]])
tensor([[0.5414, 0.4341, 0.6489, 0.3156, 0.5142]])
tensor([[1, 0, 1, 0, 1]], dtype=torch.uint8)
the result I want is like this:
tensor([[[[ 0.9129, -0.9763, 1.4460],
[ 0.3608, 0.5561, -1.4612],
[ 1.4953, -1.2474, 0.4069]],
[[-1.8454, -0.2150, 2.3703],
[ 0.5224, 0.3366, 1.7545],
[-0.4624, 1.2639, 1.8032]],
[[-0.9441, -0.2944, 0.3381],
[ 1.6562, -0.5623, 0.0599],
[ 0.7229, 0.0472, -0.5122]]]])
I believe you can simply do:
input[mask]
Btw. you don't need to call sigmoid and then .gt(0.5). You can directly do .gt(0.0) without calling the sigmoid.