Using broadcasting in normal tensors, the following works
torch.ones(3, 1)*torch.ones(3, 10)
I need to extend this behavior to sparse vectors, but I can't:
i = torch.tensor([[0, 1, 1],
[2, 0, 2]])
a = torch.sparse_coo_tensor(i, torch.ones(3, 1), [2, 4, 1])
b = torch.sparse_coo_tensor(i, torch.ones(3, 10), [2, 4, 10])
a*b
# gives RuntimeError: mul operands have incompatible sizes
Why shouldn't this work?
Is there an pytorch function to do this?
If not what is the best alternative algorithm?
Related
In PyTorch, we have nn.linear that applies a linear transformation to the incoming data:
y = WA+b
In this formula, W and b are our learnable parameters and A is my input data matrix. The matrix 'A' for my case is too large for RAM to complete loading, so I use it sparsely. Is it possible to perform such an operation on sparse matrices using PyTorch?
This is possible with PyTorch using sparse matrix multiply. In your case, I think you want something like:
>> i = [[0, 1, 1],
[2, 0, 2]]
>> v = [3, 4, 5]
>> A = torch.sparse_coo_tensor(i, v, (2, 3))
>> A.to_dense()
tensor([[0, 0, 3],
[4, 0, 5]])
# compute W#A by computing ((A.T)#(W.T)).T because...
# at time of writing, the sparse matrix must be first in the matmul
>> (A.t() # W.t()).t()
Say I have tensor A, and indexes Tensor: A = [1, 2, 3, 4], indexes = [1, 0, 3, 2]
I want to create a new Tensor from these two with the following result : [2, 1, 4, 3]
Each element of the result is element from A and the order is defined by the indexes Tensor.
Is there a way to do it with PyTorch tensor ops without loops?
My goal is to do it for 2D Tensor, but I don't think there is a way to do it without loops, so I thought to project it to 1D, do the work and project it back to the 2D.
You can use scatter:
A = torch.tensor([1, 2, 3, 4])
indices = torch.tensor([1, 0, 3, 2])
result = torch.tensor([0, 0, 0, 0])
print(result.scatter_(0, indices, A))
In 1D you can simply perform A[indexes].
In 2D it is still doable in this way:
A = torch.arange(5, 10).repeat(3, 1) # shape: (3, 5)
indexes = torch.stack([torch.randperm(5) for _ in range(3)]) # shape (3, 5)
A_sort = A[torch.arange(3).unsqueeze(1), indexes]
print(A_sort)
I want to do an operation similar to matrix multiplication, except instead of multiplying I want to check equality. The effect that I want to achieve is similar to the following:
a = torch.Tensor([[1, 2, 3], [4, 5, 6]]).to(torch.uint8)
b = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]).to(torch.uint8)
result = [[sum(a[i] == b [j]) for j in range(len(b))] for i in range(len(a))]
Is there a way that I can use einsum, or any other function in pytorch to achieve the above efficiently?
You can use torch.repeat and torch.repeat_interleave:
a = torch.Tensor([[1, 2, 3], [4, 5, 6]])
b = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
mask = a.repeat_interleave(3, dim=0) == b.repeat((2, 1))
torch.sum(mask, axis=1).reshape(a.shape)
# output
tensor([[3, 0, 0],
[0, 3, 0]])
You can make use of the broadcasting to do the same, for instance with
result = (a[:, None, :] == b[None, :, :]).sum(dim=2)
Here None just introduces a dummy dimensions - alternatively you can use the less visual .unsqueeze() instead.
matrix multiplication is ij,jk->ik in einsum notation, all of these operations are equivalent with varying levels of verbosity:
a # b
torch.einsum("ij,jk", a, b)
torch.einsum("ij,jk->ik", a, b)
(a[:,:,None] * b[None,:,:]).sum(1)
"multiply i and k dimensions and reduce j dimension"
i, j, k i, j, k
a: (2, 3) => (2, 3, None)
b: (3, 3) (None, 3, 3)
It should now be clear from this function decomposition that multiplication can be replaced with any binary operation, e.g. the equality operation.
Unfortunately, there is no generalized form of einsum (AFAIK) in pytorch that swaps the multiplication "out-of-the-box". There is however the einops library which is basically a wrapper around deep learning frameworks such as PyTorch.
I'm working with linear transformation in the form of Y=Q(X+A), where X is the input tensor and Y is the output, Q and A are two tensors to be learned. Q is an arbitrary tensor, therefore I can use nn.Linear. But A is a (differentiable) tensor that has some specific pattern, as a short example,
A = [[a0,a1,a2,a2,a2],
[a1,a0,a1,a2,a2],
[a2,a1,a0,a1,a2],
[a2,a2,a1,a0,a1],
[a2,a2,a2,a1,a0]].
So I cannot define such a pattern in nn.Linear. Is there any way to define such a tensor in Pytorch?
This looks like a Toeplitz matrix. A possible implementation in PyTorch is:
def toeplitz(c, r):
vals = torch.cat((r, c[1:].flip(0)))
shape = len(c), len(r)
i, j = torch.ones(*shape).nonzero().T
return vals[j-i].reshape(*shape)
In your case with a0 as 0, a1 as 1 and a2 as 2:
>>> toeplitz(torch.tensor([0,1,2,2,2]), torch.tensor([0,1,2,2,2]))
tensor([[0, 1, 2, 2, 2],
[1, 0, 1, 2, 2],
[2, 1, 0, 1, 2],
[2, 2, 1, 0, 1],
[2, 2, 2, 1, 0]])
For a more detailed explanation refer to my other answer here.
I am trying to do index-assignment in Theano using set_subtensor(), but it is giving different results to Numpy's index-assignment. Am I doing something wrong, or is this a difference in how set_subtensor and Numpy's index-assignment work?
What I want to do:
X = np.zeros((2, 2))
X[[[0, 1], [0, 1]]] = np.array([1, 2])
X is now:
[[ 1. 0.]
[ 0. 2.]]
Trying to do the same thing in Theano:
X = theano.shared(value=np.zeros((2, 2)))
X = T.set_subtensor(X[[[0, 1], [0, 1]]], np.array([1, 2]))
X.eval()
Raises this error
ValueError: array is not broadcastable to correct shape
This highlights a subtle difference between numpy and Theano but it can be worked around easily.
Advanced indexing can be enabled in numpy by using a list of positions or a tuple of positions. In Theano, one can only use a tuple of positions.
So changing
X = T.set_subtensor(X[[[0, 1], [0, 1]]], np.array([1, 2]))
to
X = T.set_subtensor(X[([0, 1], [0, 1])], np.array([1, 2]))
solves the problem in Theano.
One continues to get the same result in numpy if one changes
X[[[0, 1], [0, 1]]] = np.array([1, 2])
to
X[([0, 1], [0, 1])] = np.array([1, 2])