I am trying to write a function that reduces a numpy ndarray to a given shape, which effectively "unbroadcasts" the array. For example, using add as the universal function, i want the following results:
A = [1, 2, 3, 4, 5], reduced_shape = (1,) -> [1+2+3+4+5] = [15] (sum over axis=0)
A = [[1,2], [1,2]], reduced_shape = (2,) -> [1+1, 2+2] = [2, 4] (sum over axis=0)
A = [[1,2], [1,2]], reduced_shape = (1,) -> [1+2+1+2] = [6] (sum over axis=(0,1))
A = [[[1,2], [1,2]], [[1,2], [1,2]]], reduced_shape = (2,2) -> [[1+1, 2+2], [1+1, 2+2]] = [[2,4], [2,4]] (sum over axis=0)
This is the solution i came up with:
def unbroadcast(A, reduced_shape):
fill = reduced_shape[-1] if reduced_shape else None
reduced_axes = tuple(i for i, (a,b) in enumerate(itertools.zip_longest(A, shape, fillvalue=fill)) if a!=b)
return np.add.reduce(A, axis=reduced_axes).reshape(shape)
But it feels unnecessarily complex, is there way to implement this that relies on Numpy's public API?
It's not clear how this is an 'un-broadcasting'.
The straight forward way of doing your calculations is to use the axis parameter of sum:
In [124]: np.array([1,2,3,4,5]).sum()
Out[124]: 15
In [125]: np.array([[1,2],[1,2]]).sum(axis=0)
Out[125]: array([2, 4])
In [126]: np.array([[1,2],[1,2]]).sum(axis=(0,1))
Out[126]: 6
In [128]: np.array([[[1,2], [1,2]], [[1,2], [1,2]]]).sum(axis=0)
Out[128]:
array([[2, 4],
[2, 4]])
I don't use reduce as much, but looks like axis does the same:
In [130]: np.add.reduce(np.array([1,2,3,4,5]))
Out[130]: 15
In [132]: np.add.reduce(np.array([[[1,2], [1,2]], [[1,2], [1,2]]]),axis=0)
Out[132]:
array([[2, 4],
[2, 4]])
But I haven't worked out the logic of going from your reduced_shape to the necessary axis values. With shapes like (2,) and (2,2,2), there's potential ambiguity when you say reduce the shape to (2,2). It might be clearer if you worked with samples arrays like np.arange(24).reshape(2,3,4)
Related
I want to do an operation similar to matrix multiplication, except instead of multiplying I want to check equality. The effect that I want to achieve is similar to the following:
a = torch.Tensor([[1, 2, 3], [4, 5, 6]]).to(torch.uint8)
b = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]).to(torch.uint8)
result = [[sum(a[i] == b [j]) for j in range(len(b))] for i in range(len(a))]
Is there a way that I can use einsum, or any other function in pytorch to achieve the above efficiently?
You can use torch.repeat and torch.repeat_interleave:
a = torch.Tensor([[1, 2, 3], [4, 5, 6]])
b = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
mask = a.repeat_interleave(3, dim=0) == b.repeat((2, 1))
torch.sum(mask, axis=1).reshape(a.shape)
# output
tensor([[3, 0, 0],
[0, 3, 0]])
You can make use of the broadcasting to do the same, for instance with
result = (a[:, None, :] == b[None, :, :]).sum(dim=2)
Here None just introduces a dummy dimensions - alternatively you can use the less visual .unsqueeze() instead.
matrix multiplication is ij,jk->ik in einsum notation, all of these operations are equivalent with varying levels of verbosity:
a # b
torch.einsum("ij,jk", a, b)
torch.einsum("ij,jk->ik", a, b)
(a[:,:,None] * b[None,:,:]).sum(1)
"multiply i and k dimensions and reduce j dimension"
i, j, k i, j, k
a: (2, 3) => (2, 3, None)
b: (3, 3) (None, 3, 3)
It should now be clear from this function decomposition that multiplication can be replaced with any binary operation, e.g. the equality operation.
Unfortunately, there is no generalized form of einsum (AFAIK) in pytorch that swaps the multiplication "out-of-the-box". There is however the einops library which is basically a wrapper around deep learning frameworks such as PyTorch.
Supposing I have two square matrices A, B of the same size
A = torch.tensor([[1, 2], [3, 4]])
B = torch.tensor([[1, 1], [1, 1]])
And I want a resulting tensor that consists of the row-wise dot product, say
tensor([3, 7]) # i.e. (1*1 + 2*1, 3*1 + 4*1)
What is an efficient means of achieving this in PyTorch?
As you said you can use torch.bmm but you first need to broadcast your inputs:
>>> torch.bmm(A[..., None, :], B[..., None])
tensor([[[3]],
[[7]]])
Alternatively you can use torch.einsum:
>>> torch.einsum('ij,ij->i', A, B)
tensor([3, 7])
import torch
import numpy as np
def row_wise_product(A, B):
num_rows, num_cols = A.shape[0], A.shape[1]
prod = torch.bmm(A.view(num_rows, 1, num_cols), B.view(num_rows, num_cols, 1))
return prod
A = torch.tensor(np.array([[1, 2], [3, 4]]))
B = torch.tensor(np.array([[1, 1], [1, 1]]))
C = row_wise_product(A, B)
Suppose I have a matrix such as P = [[0,1],[1,0]] and a vector v = [a,b]. If I multiply them I have:
Pv = [b,a]
The matrix P is simply a permutation matrix, which changes the order of each element.
Now suppose that I have the same P, but I have the matrices M1 = [[1,2],[3,4]] and M2=[[5,6],[7,8]]. Now let me combine them as the 3D Tensor T= [[[1,2],[3,4]], [[5,6],[7,8]]] with dimensions (2,2,2) - (C,W,H). Suppose I multiply P by T such that:
PT = [[[5,6],[7,8]], [[1,2],[3,4]]]
Note that now M1 now equals [[5,6],[7,8]] and M2 equals [[1,2],[3,4]] as the values have been permuted across the C dimension in T (C,W,H).
How can I multiply PT (P=2D tensor,T=3D tensor) in pytorch using matmul? The following does not work:
torch.matmul(P, T)
An alternative solution to #mlucy's answer, is to use torch.einsum. This has the benefit of defining the operation yourself, without worrying about torch.matmul's requirements:
>>> torch.einsum('ij,jkl->ikl', P, T)
tensor([[[5, 6],
[7, 8]],
[[1, 2],
[3, 4]]])
Or with torch.matmul:
>>> (P # T.flatten(1)).reshape_as(T)
tensor([[[5, 6],
[7, 8]],
[[1, 2],
[3, 4]]])
You could do something like:
torch.matmul(P, X.flatten(1)).reshape(X.shape)
I am trying to roll only the first n elements from my numpy axis instead of all. However, I am at a loss on how to accomplish this.
import numpy as np
foo = np.random.rand(32,3,16,16)
#Foo is a batch of 32 images, with 3 channels and a height, width of 16
print("Foo Shape = ", foo.shape)
#Foo Shape = (32, 3, 16, 16)
I would like to roll each first element of the second axis by 1 step. Basically roll the first channel of each image by 1.
np.roll(foo, 1, 1)
The code above rolls all the elements of the second axis (channel dimension) by 1, instead of just rolling the first element. I couldn't find any numpy functionality that helps with this issue.
Select only the elements you want using a 2D slice:
>>> import numpy as np
>>> arr = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
>>> print(arr)
[[[1 2]
[3 4]]
[[5 6]
[7 8]]]
>>> arr[:, 0] = np.roll(arr[:, 0], shift=1, axis=1)
>>> print(arr)
[[[2 1]
[3 4]]
[[6 5]
[7 8]]]
>>>
I have a theano tensor and I would like to clip its values, but each index to a different range.
For example, if I have a vector [a,b,c] , I want to clip a to [0,1] , clip b to [2,3] and c to [3,5].
How can I do that efficiently?
Thanks!
The theano.tensor.clip operation supports symbolic minimum and maximum values so you can pass three tensors, all of the same shape, and it will perform an element-wise clip of the first with respect to the second (minimum) and third (maximum).
This code shows two variations on this theme. v1 requires the minimum and maximum values to be passed as separate vectors while v2 allows the minimum and maximum values to be passed more like a list of pairs, represented as a two column matrix.
import theano
import theano.tensor as tt
def v1():
x = tt.vector()
min_x = tt.vector()
max_x = tt.vector()
y = tt.clip(x, min_x, max_x)
f = theano.function([x, min_x, max_x], outputs=y)
print f([2, 1, 4], [0, 2, 3], [1, 3, 5])
def v2():
x = tt.vector()
min_max = tt.matrix()
y = tt.clip(x, min_max[:, 0], min_max[:, 1])
f = theano.function([x, min_max], outputs=y)
print f([2, 1, 4], [[0, 1], [2, 3], [3, 5]])
def main():
v1()
v2()
main()