If I have n sized mean vector, n sized variance vector, then how do I do this?
z ∼ N (μ, σ)
import torch
x = torch.randn(3, 3)
mu = x.mean()
sigma = x.var()
What do I do to get z?
If you want to sample from a normal distribution with mean mu and std sigma then you can simply
z = torch.randn_like(mu) * sigma + mu
If you sample many such z their mean and std will converge to sigma and mu:
mu = torch.arange(10.)
Out[]: tensor([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
sigma = 5. - 0.5 * torch.arange(10.)
Out[]: tensor([5.0000, 4.5000, 4.0000, 3.5000, 3.0000, 2.5000, 2.0000, 1.5000, 1.0000, 0.5000])
z = torch.randn(10, 1000000) * sigma[:, None] + mu[:, None]
z.mean(dim=1)
Out[]:
tensor([-5.4823e-03, 1.0011e+00, 1.9982e+00, 2.9985e+00, 4.0017e+00,
4.9972e+00, 6.0010e+00, 7.0004e+00, 7.9996e+00, 9.0006e+00])
z.std(dim=1)
Out[]:
tensor([4.9930, 4.4945, 4.0021, 3.5013, 3.0005, 2.4986, 1.9997, 1.4998, 0.9990,
0.5001])
As you can see when you sample 1,000,000 elements from the distribution the sample mean and std are close to the original mu and sigma you started with.
Related
I am having trouble understanding the conceptual meaning of the grad_outputs option in torch.autograd.grad.
The documentation says:
grad_outputs should be a sequence of length matching output containing the “vector” in Jacobian-vector product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can be None).
I find this description quite cryptic. What exactly do they mean by Jacobian-vector product? I know what the Jacobian is, but not sure about what product they mean here: element-wise, matrix product, something else? I can't tell from my example below.
And why is "vector" in quotes? Indeed, in the example below I get an error when grad_outputs is a vector, but not when it is a matrix.
>>> x = torch.tensor([1.,2.,3.,4.], requires_grad=True)
>>> y = torch.outer(x, x)
Why do we observe the following output; how was it computed?
>>> y
tensor([[ 1., 2., 3., 4.],
[ 2., 4., 6., 8.],
[ 3., 6., 9., 12.],
[ 4., 8., 12., 16.]], grad_fn=<MulBackward0>)
>>> torch.autograd.grad(y, x, grad_outputs=torch.ones_like(y))
(tensor([20., 20., 20., 20.]),)
However, why this error?
>>> torch.autograd.grad(y, x, grad_outputs=torch.ones_like(x))
RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([4]) and output[0] has a shape of torch.Size([4, 4]).
If we take your example we have function f which takes as input x shaped (n,) and outputs y = f(x) shaped (n, n). The input is described as column vector [x_i]_i for i ∈ [1, n], and f(x) is defined as matrix [y_jk]_jk = [x_j*x_k]_jk for j, k ∈ [1, n]².
It is often useful to compute the gradient of the output with respect to the input (or sometimes w.r.t the parameters of f, there are none here). In the more general case though, we are looking to compute dL/dx and not just dy/dx, where dL/dx is the partial derivative of L, computed from y, w.r.t. x.
The computation graph looks like:
x.grad = dL/dx <------- dL/dy y.grad
dy/dx
x -------> y = x*xT
Then, if we look at dL/dx, which is, via the chain rule equal to dL/dy*dy/dx. We have, looking at the interface of torch.autograd.grad, the following correspondences:
outputs <-> y,
inputs <-> x, and
grad_outputs <-> dL/dy.
Looking at the shapes: dL/dx should have the same shape as x (dL/dx can be referred to as the 'gradient' of x), while dy/dx, the Jacobian matrix, would be 3-dimensional. On the other hand dL/dy, which is the incoming gradient, should have the same shape as the output, i.e., y's shape.
We want to compute dL/dx = dL/dy*dy/dx. If we look more closely, we have
dy/dx = [dy_jk/dx_i]_ijk for i, j, k ∈ [1, n]³
Therefore,
dL/dx = [dL/d_x_i]_i, i ∈ [1,n]
= [sum(dL/dy_jk * d(y_jk)/dx_i over j, k ∈ [1, n]²]_i, i ∈ [1,n]
Back to your example, it means for a given i ∈ [1, n]: dL/dx_i = sum(dy_jk/dx_i) over j, k ∈ [1,n]². And dy_jk/dx_i = f(x_j*x_k)/dx_i will equal x_j if i = k, x_k if i = j, and 2*x_i if i = j = k (because of the squared x_i). This being said matrix y is symmetric... So the result comes down to 2*sum(x_i) over i ∈ [1, n]
This means dL/dx is the column vector [2*sum(x)]_i for i ∈ [1, n].
>>> 2*x.sum()*torch.ones_like(x)
tensor([20., 20., 20., 20.])
Stepping back look at this other graph example, here adding an additional operation after y:
x -------> y = x*xT --------> z = y²
If you look at the backward pass on this graph, you have:
dL/dx <------- dL/dy <-------- dL/dz
dy/dx dz/dy
x -------> y = x*xT --------> z = y²
With dL/dx = dL/dy*dy/dx = dL/dz*dz/dy*dy/dx which is in practice computed in two sequential steps: dL/dy = dL/dz*dz/dy, then dL/dx = dL/dy*dy/dx.
I want to compute the gradient in the following scenario:
y = w_0x+w_1 and z = w_2x + (dy/dx)^2
w = torch.tensor([2.,1.,3.], requires_grad=True)
x = torch.tensor([0.5], requires_grad=True)
y = w[0]*x + w[1]
y.backward()
l = x.grad
l.requires_grad=True
w.grad.zero_()
z = w[2]*x + l**2
z.backward()
I expect [4, 0, 0.5] instead I get [0, 0, 0.5]. I know in this case I can replace l by w_0 but, l can be a complex function of x in which case it is important that I compute the gradients numerically instead of changing the expression for z. Please let me know what changes I need to do get the correct gradient w.r.t w
You should print your gradients along the way, it would be easier this way.
I will comment out what's going on in code:
import torch
w = torch.tensor([2.0, 1.0, 3.0], requires_grad=True)
x = torch.tensor([0.5], requires_grad=True)
y = w[0] * x + w[1]
y.backward()
l = x.grad
l.requires_grad = True
print(w.grad) # [0.5000, 1.0000, 0.0000] as expected
w.grad.zero_()
print(w.grad) # [0., 0., 0.] as you cleared the gradient
z = w[2] * x + l ** 2
z.backward()
print(w.grad) # [0., 0., 0.5] - see below
Last print(w.grad) works like that because your are using last element of tensor and it's the only taking part in equation z, it's multiplied by x which is 0.5 hence gradient is 0.5. You cleared the gradient before by issuing w.grad_zero_(). I can't see how could you get [4., 0., 0.5]. If you didn't clear the gradient, you would get: tensor([0.5000, 1.0000, 0.5000]), the first two being from the first y equation, the second one and the last from the z equation.
I wan to compute the product between two sets of feature matrices X and Y
of dimensions (H,W,12) each:
Inefficiently I would do:
H = []
for i in range(12):
for j in range(12):
h = X[:,:,i]*Y[:,:,j]
H.append(h)
which will output H of dimension (H,W,144)
How can this be done in pytorch without iterating in two loops?
I have tried used tensordot solutions but cant replicate the behavior.
I am not sure this is the most efficient, but you can do something like this (warning: ugly code ahead =]):
import torch
# I choose not to use random -- easier to verify, IMO
a = torch.Tensor([[[1,2],[3,4],[5,6]],[[1,2],[3,4],[5,6]]])
b = torch.Tensor([[[1,2],[3,4],[5,6]],[[1,2],[3,4],[5,6]]])
c = torch.bmm(
a.view(-1, a.size(-1), 1),
b.view(-1, 1, b.size(-1))
).view(*(a.shape[:2]), -1)
print(c)
print(a.shape)
print(b.shape)
print(c.shape)
Output:
tensor([[[ 1., 2., 2., 4.],
[ 9., 12., 12., 16.],
[25., 30., 30., 36.]],
[[ 1., 2., 2., 4.],
[ 9., 12., 12., 16.],
[25., 30., 30., 36.]]])
torch.Size([2, 3, 2]) # a
torch.Size([2, 3, 2]) # b
torch.Size([2, 3, 4]) # c
Basically, the outer product. Let me know if you need me to explain.
Timings
While using the torch.bmm, 16 out of 32 cores were being used. I used a GeForce RTX 2080 Ti to run the CUDA version (GPU usage was ~97% during execution). Note that the dimensions used on GPU timings are x10, otherwise it is just too fast.
Script:
import timeit
setup = '''
import torch
a = torch.randn(({H}, {W}, 12))
b = torch.randn(({H}, {W}, 12))
'''
setup_cuda = setup.replace("))", ")).to(torch.device('cuda'))")
bmm = '''
c = torch.bmm(
a.view(-1, a.size(-1), 1),
b.view(-1, 1, b.size(-1))
).view(*(a.shape[:2]), -1)
'''
loop = '''
c = []
for i in range(a.size(-1)):
for j in range(b.size(-1)):
c.append(a[:, :, i] * b[:, :, j])
c = torch.stack(c).permute(1, 2, 0)
'''
min_dim = 10
max_dim = 100
num_repeats = 10
print('BMM')
for d in range(min_dim, max_dim+1, 10):
print(d, min(timeit.Timer(bmm, setup=setup.format(H=d, W=d)).repeat(num_repeats, 1000)))
print('LOOP')
for d in range(min_dim, max_dim+1, 10):
print(d, min(timeit.Timer(loop, setup=setup.format(H=d, W=d)).repeat(num_repeats, 1000)))
print('BMM - CUDA')
for d in range(min_dim*10, (max_dim*10)+1, 100):
print(d, min(timeit.Timer(bmm, setup=setup_cuda.format(H=d, W=d)).repeat(num_repeats, 1000)))
Output:
BMM
10 0.022082214010879397
20 0.034024904016405344
30 0.08957623899914324
40 0.1376199919031933
50 0.20248223491944373
60 0.2657837320584804
70 0.3533527449471876
80 0.42361779196653515
90 0.6103016039123759
100 0.7161333339754492
LOOP
10 1.7369094720343128
20 1.8517447559861466
30 1.9145489090587944
40 2.0530637570191175
50 2.2066439649788663
60 2.394576688995585
70 2.6210166650125757
80 2.9242434420157224
90 3.5709626079769805
100 5.413458575960249
BMM - CUDA
100 0.014253990724682808
200 0.015094103291630745
300 0.12792395427823067
400 0.307440347969532
500 0.541196970269084
600 0.8697826713323593
700 1.2538292426615953
800 1.6859236396849155
900 2.2016236428171396
1000 2.764942280948162
In the Theano Tutorial:
http://deeplearning.net/software/theano/tutorial/gradients.html
or
>>> import theano
>>> import theano.tensor as T
>>> x = T.dvector('x')
>>> y = x ** 2
>>> J, updates = theano.scan(lambda i, y,x : T.grad(y[i], x), sequences=T.arange(y.shape[0]), non_sequences=[y,x])
>>> f = theano.function([x], J, updates=updates)
>>> f([4, 4])
array([[ 8., 0.],
[ 0., 8.]])
"theano.scan" is used to compute the Jacobian. I understand how to compute the Jacobian by hand but I do not quite understand how the function scan is calculating it iteratively; specifically, I don't understand how the "zeroes" are obtained in the Jacobian.
Can someone show me, for each iteration, how this is calculated? Thanks!
I had a chat with an engineer the other day and we both were stumped on a question related to bundle adjustment. For a refresher, here is a good link explaining the problem:
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/ZISSERMAN/bundle/bundle.html
The problem requires optimization over 3n+11m parameters. The camera optimization consists of 5 intrinsic camera parameters, 3 DOF for position (x,y,z), and 3 DOF for rotation (pitch, yaw and roll).
Now, when you actually go about implementing this algorithm, a rotation matrix consists an optimization over 9 numbers. Euler's Axis Theorem says these 9 numbers are related and there are only 3 degrees of freedom overall.
Suppose you represent the rotation using a normalized quaternion. Then you have optimization over 3 numbers. Same DOF.
Is one representation more computationally efficient and better than the other? Will you have less variables to optimize using a rotation quaternion over rotation matrix?
You never optimize over 9 numbers! Of course this would be inefficient. One efficient representation in which you only need 3 parameters is to parametrize your rotation matrix R using the Lie algebra of the groupe SO(3). If you are not familiar with Lie algebra, here's a tutorial that explains everything in an intuitive (but sometimes oversimplified) manner. To explain it in a few short sentences, in this representation, each rotation matrix R is written as expmat(a*G_1+b*G_2+c*G_3) where expmat is the matrix exponential, and the G_i are the "generators" of the lie algebra of SO(3), i.e. the tangent space to SO(3) at the identity. Therefore, to estimate a rotation matrix, you only need to learn the three parameters a,b,c. This is roughly equivalent to decomposing your rotation matrix in three rotations around x,y,z and estimating the three angles of these rotations.
A solution not mentioned yet is to use axis-angle parameterization.
Basically, you represent the rotation as a single 3D vector. The direction v/|v| of the vector is the axis of rotation, and the norm |v| is the angle of rotation around that axis.
This method has 3 DOF directly, unlike quaternions' 4 DOF. So with quaternions, you need to use either constrained optimization or additional parameterization to get down to 3 DOF.
I'm not familiar with #Ash's suggestion, but he does mention in the comment that it only works for small angles. Axis-angle representation doesn't have this limitation.
One option is as relatively_random suggests to optimize over the axis-angle parameterization. The derivative can then, relatively simple, be computed as described in this paper. The only problem might be that some numerical issues might arise for rotations close to the identity.
import numpy as np
def hat(v):
"""
vecotrized version of the hat function, creating for a vector its skew symmetric matrix.
Args:
v (np.array<float>(..., 3, 1)): The input vector.
Returns:
(np.array<float>(..., 3, 3)): The output skew symmetric matrix.
"""
E1 = np.array([[0., 0., 0.], [0., 0., -1.], [0., 1., 0.]])
E2 = np.array([[0., 0., 1.], [0., 0., 0.], [-1., 0., 0.]])
E3 = np.array([[0., -1., 0.], [1., 0., 0.], [0., 0., 0.]])
return v[..., 0:1, :] * E1 + v[..., 1:2, :] * E2 + v[..., 2:3, :] * E3
def exp(v, der=False):
"""
Vectorized version of the exponential map.
Args:
v (np.array<float>(..., 3, 1)): The input axis-angle vector.
der (bool, optional): Wether to output the derivative as well. Defaults to False.
Returns:
R (np.array<float>(..., 3, 3)): The corresponding rotation matrix.
[dR (np.array<float>(3, ..., 3, 3)): The derivative of each rotation matrix.
The matrix dR[i, ..., :, :] corresponds to
the derivative d R[..., :, :] / d v[..., i, :],
so the derivative of the rotation R gained
through the axis-angle vector v with respect
to v_i. Note that this is not a Jacobian of
any form but a vectorized version of derivatives.]
"""
n = np.linalg.norm(v, axis=-2, keepdims=True)
H = hat(v)
with np.errstate(all='ignore'):
R = np.identity(3) + (np.sin(n) / n) * H + ((1 - np.cos(n)) / n**2) * (H # H)
R = np.where(n == 0, np.identity(3), R)
if der:
sh = (3,) + tuple(1 for _ in range(v.ndim - 2)) + (3, 1)
dR = np.swapaxes(np.expand_dims(v, axis=0), 0, -2) * H
dR = dR + hat(np.cross(v, ((np.identity(3) - R) # np.identity(3).reshape(sh)), axis=-2))
dR = dR # R
n = n**2 # redifinition
with np.errstate(all='ignore'):
dR = dR / n
dR = np.where(n == 0, hat(np.identity(3).reshape(sh)), dR)
return R, dR
else:
return R
# generate two sets of points which differ by a rotation
np.random.seed(1001)
n = 100 # number of points
p_1 = np.random.randn(n, 3, 1)
v = np.array([0.3, -0.2, 0.1]).reshape(3, 1) # the axis-angle vector
p_2 = exp(v) # p_1 + np.random.randn(n, 3, 1) * 1e-2
# estimate v with least sqaures, so the objective function becomes:
# minimize v over f(v) = sum_[1<=i<=n] (||p_1_i - exp(v)p_2_i||^2)
# Due to the way least_squres is implemented we have to pass the
# individual residuals ||p_1_i - exp(v)p_2_i||^2 as ||p_1_i - exp(v)p_2_i||.
from scipy.optimize import least_squares
def loss(x):
R = exp(x.reshape(1, 3, 1))
y = p_2 - R # p_1
y = np.linalg.norm(y, axis=-2).squeeze(-1)
return y
def d_loss(x):
R, d_R = exp(x.reshape(1, 3, 1), der=True)
y = p_2 - R # p_1
d_y = -d_R # p_1
d_y = np.sum(y * d_y, axis=-2) / np.linalg.norm(y, axis=-2)
d_y = d_y.squeeze(-1).T
return d_y
x0 = np.zeros((3))
res = least_squares(loss, x0, d_loss)
print('True axis-angle vector: {}'.format(v.reshape(-1)))
print('Estimated axis-angle vector: {}'.format(res.x))