Given tensors x and y, each with shape (num_batches, d), how can I use PyTorch to compute the sum of every combination of x and y within a batch?
This is similar to outer product, except we don't want to multiply, but sum. (This implies that I could solve this by exponentiating, outer product, and taking the log, but of course that has numerical and performance disadvantages).
It could be done via cartesian product and then summing each of the combinations.
Essentially, I'd like osum[b, i, j] == x[b, i] + y[b, j]. Can PyTorch do this in tensors, without loops?
This can easily be done, by introducing singleton dimensions into x and y and broadcasting along these singleton dimensions:
osum = x[..., None] + y[:, None, :]
For example:
x = torch.arange(6).view(2,3)
y = x * 10
osum = x[..., None] + y[:, None, :]
Results with:
tensor([[[ 0, 10, 20],
[ 1, 11, 21],
[ 2, 12, 22]],
[[33, 43, 53],
[34, 44, 54],
[35, 45, 55]]])
Update (July, 14th): How it works?
You have two tensors, x and y of shape bxn, and you want to compute:
osum[b,i,j] = x[b, i] + y[b, j]
We can, conceptually, create new variables xx and yy by repeating each element of x and y along a third dimension, such that:
xx[b, i, j] == x[b, i] # for all j
yy[b, i, j] == y[b, j] # for all i
With these new variables, it is easy to see that:
osum = xx + yy
since, by deinition
osum[b, i, j] == xx[b, i, j] + yy[b, i, j] == x[b, i] + y[b, j]
Now, you can use commands such as torch.expand or torch.repeat to explicitly create xx and yy - but why bother? since their elements are just trivial repetitions of the elements along specific dimensions, broadcasting does this implicitly for you.
You can perform such operation using broadcasting:
>>> x = torch.randint(0,10,(2,4))
tensor([[0, 6, 5, 8],
[3, 0, 7, 5]])
>>> y = torch.randint(0,10,(2,5))
tensor([[6, 9, 9, 8, 7],
[0, 4, 6, 2, 5]])
>>> x[:,:,None].shape
(2, 4, 1)
>>> y[:,None].shape
(2, 1, 5])
Adding a singleton to dimensions which differ ensures the 'outer' operation
is performed.
>>> osum = x[:,:,None] + y[:,None]
tensor([[[ 6, 9, 9, 8, 7],
[12, 15, 15, 14, 13],
[11, 14, 14, 13, 12],
[14, 17, 17, 16, 15]],
[[ 3, 7, 9, 5, 8],
[ 0, 4, 6, 2, 5],
[ 7, 11, 13, 9, 12],
[ 5, 9, 11, 7, 10]]])
Related
I'd like to take a vector and get an array of vectors in which the i-th element of each vector are the k neighbors of the i-th element of the original vector. Also, I'm looking for the fastest way to do so.
I've already done that in MATLAB:
a=zeros(k, length(v));
I=cell(1,k);
a(1,:) = v;
for j=2:k
a(k,:)=[a(k-1,2:end),a(k-1,1)];
end
aux1=[a(:,(end-r+1):end),a(:,1:(end-r))];
for j=1:k
I{k}=aux1(k,:);
end
For example, v = [1, 2, 3, 4, 5] and k = 1; and I want to get:
M = [[5, 1, 2, 3, 4], [1, 2, 3, 4, 5], [2, 3, 4, 5, 1]]
so that, for the 1st element of each vector, I get [5; 1; 2], which are the element 1 and its neighbors.
Hope it makes sense. Thanks for reading :)
You could use the numpy roll function:
import numpy as np
def get_neighbors(v, k):
N = len(v)
M = np.zeros((k*2+1, N), dtype=int)
for i in range(-k, k+1):
M[i+k, :] = np.roll(v, -i)
return M
v = np.array([1, 2, 3, 4, 5])
k = 1
M = get_neighbors(v, k)
print(M)
Output:
[[5 1 2 3 4]
[1 2 3 4 5]
[2 3 4 5 1]]
Using sliding_window_view on a repetition of your array can do it "vectorized" way
# Example array
a = np.arange(1,16)
k = 2 # Window of neighbors
# My solution
np.lib.stride_tricks.sliding_window_view(np.hstack([a,a,a]), (len(a),))[len(a)-k:len(a)+k+1]
Returns
array([[14, 15, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13],
[15, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
[ 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1],
[ 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1, 2]])
Note that sliding_window_view creates just a view. It doesn't create new data. Hence the reason why I do not hesitate creating (in this example) 31 lines (3*15-15+1), and then subset only 5 of them: I do not really create them.
So only real cost of that solution is in hstack, both cpu-wise and memory-wise.
That subset, btw, was done to abide strictly by what you asked. But, depending on what you intend to do, you may drop the subset. Important point is that if
T=np.lib.stride_tricks.sliding_window_view(np.hstack([a,a,a]), (len(a),))
Then T[len(a)+k] is a row made of the kth neighbor, whether k is positive, negative or 0 (the original row)
See timings, since it matters for you
sizes
This method
Roll method
len=15/k=2
51 μs
132 μs
len=15/k=7
51 μs
383 μs
len=1000/k=7
52 μs
422 μs
len=1M/k=7
6 ms
160 ms
len=1M/k=100
6 ms
2.2 s
Roll method is obviously proportional to the size of the window (O(k) — it has one roll to perform per row of output), when sliding_window_view is just a view, and does not really create rows, so is O(1) as far as k is concerned. Both method are equally impacted by len of data (O(n) really, but it shows only for n big enough).
So, all together, this method is O(n) while roll method is O(kn)
How can I slice a 3D tensor using a 1D tensor? For instance, consider the following 2 tensors: t of size [Batch, Sequence, Dim]; and idx of size [Batch]. The values of idx are restricted to be integers between 0 and Sequence-1.
I need tensor idx to select the corresponding slices in the second dimension of tensor t. For example:
t = torch.arange(24).view(2,3,4)
>>> tensor([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
idx = torch.tensor([0,2])
>>> tensor([0, 2])
Then the desired output is: tensor([[ 0, 1, 2, 3], [20, 21, 22, 23]]).
The following code solves the problem, however it's inefficient, as it involves one_hot, multiplication and sum operations.
one_hot_idx = nn.functional.one_hot(idx.long(), num_classes=t.shape[1]).unsqueeze(-1)
(t*one_hot_idx).sum(1)
You can do it like this:
import torch
t = torch.arange(24).view(2, 3, 4)
idx = torch.tensor([0, 2])
print(t[range(len(idx)), idx])
Output:
tensor([[ 0, 1, 2, 3],
[20, 21, 22, 23]])
I would like to calculate all cross-terms of each vector in a matrix.
For example, consider the following matrix:
X = tensor([[1, 2, 3],
[4, 5, 6]]),
and I would like to obtain all cross-terms of each vector in this matrix as:
Y = [[1*1, 1*2, 1*3, 2*2, 2*3, 3*3],
[4*4, 4*5, 4*6, 5*5, 5*6, 6*6]].
= [[1, 2, 3, 4, 6, 9],
[16, 20, 24, 25, 30, 36]].
That is, this is the all combination values of the vector elements
and I believe that this can be calculated using torch.combinations;
however, torch.combinations does not provide the batch implementation
and I couldn't produce the above result in pytorch.
How can I calculate all cross-terms in pytorch?
You can stack the product of combinations with replacement for each of the rows in that matrix
>>> torch.stack(tuple(torch.prod(torch.combinations(data[i],with_replacement=True),1) for i in range(data.shape[0])),0)
>>> tensor([[ 1, 2, 3, 4, 6, 9],
[16, 20, 24, 25, 30, 36]])
I am trying to do cyclic rotation of numbers in List in Python.
For example,
An array A consisting of N integers is given. Rotation of the array means that each element is shifted right by one index, and the last element of the array is moved to the first place. For example, the rotation of array A = [3, 8, 9, 7, 6] is [6, 3, 8, 9, 7] (elements are shifted right by one index and 6 is moved to the first place).
The goal is to rotate array A K times; that is, each element of A will be shifted to the right K times.
For example, given
A = [3, 8, 9, 7, 6]
K = 3
the function should return [9, 7, 6, 3, 8]. Three rotations were made:
[3, 8, 9, 7, 6] -> [6, 3, 8, 9, 7]
[6, 3, 8, 9, 7] -> [7, 6, 3, 8, 9]
[7, 6, 3, 8, 9] -> [9, 7, 6, 3, 8]
I have written a function that is supposed to do the above task. Here is my code:
def sol(A, K):
for i in range(len(A)):
if (i+K) < len(A):
A[i+K] = A[i]
else:
A[i+K - len(A)] = A[i]
return A
A = [3, 8, 9, 7, 6]
K = 3
# call the function
sol(A,K)
[9, 3, 8, 3, 8]
I am getting [9, 3, 8, 3, 8] instead of [9, 7, 6, 3, 8].
Can anyone help me with the above code ?
Thanks.
Let's take a look at what happens if K = 1, on just the first iteration:
def sol(A, K):
for i in range(len(A)): # i = 0
if (i+K) < len(A): # i + 1 < 5
A[i+K] = A[i] # A[1] = A[0]
else:
A[i+K - len(A)] = A[i]
# A is now equal to [3, 3, 9, 7, 6] - the element at A[1] got overwritten
return A
The problem is that you don't have anywhere to store the elements you'd be overwriting, and you're overwriting them before rotating them. The ideal solution is to create a new list, populating it with rotated elements from the previous list, and return that. If you need to modify the old list, you can copy elements over from the new list:
def sol(A, K):
ret = []
for i in range(len(A)):
if (i + K) < len(A):
ret.append(A[i + K])
else:
ret.append(A[i + K - len(A)])
return ret
Or, more concisely (and probably how your instructor would prefer you solve it), using the modulo operator:
def sol(A, K):
return [
A[(i + K) % len(A)]
for i in range(len(A))
]
Arguably the most pythonic solution, though, is to concatenate two list slices, moving the part of the list after index K to the front:
def sol(A, K):
return A[K % len(A):] + A[:K % len(A)]
I need to do some calculations with a NetCDF file.
So I have two variables with following dimensions and sizes:
A [time | 1] x [lev | 12] x [lat | 84] x [lon | 228]
B [lev | 12]
What I need is to produce a new array, C, that is shaped as (1,12,84,228) where B contents are propagated to all dimensions of A.
Usually, this is easily done in NCL with the conform function. I am not sure what is the equivalent of this in Python.
Thank you.
The numpy.broadcast_to function can do something like this, although in this case it does require B to have a couple of extra trailing size 1 dimension added to it to satisfy the numpy broadcasting rules
>>> import numpy
>>> B = numpy.arange(12).reshape(12, 1, 1)
>>> B
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
>>> B = B.reshape(12, 1, 1)
>>> B.shape
(12, 1, 1)
>>> C = numpy.broadcast_to(b, (1, 12, 84, 228))
>>> C.shape
(1, 12, 84, 228)
>>> C[0, :, 0, 0]
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
>>> C[-1, :, -1, -1]
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])