Computing intermediate gradients using backward method in Pytorch - pytorch

Question

I am having trouble in understanding backward method in pytorch
x1 = tensor(2.).requires_grad_()
x2 = tensor(3.).requires_grad_() # or x2 = tensor(3.)
x3 = x1 + x2
l = (x3**2).sum()
l.backward()
print(x1)
print(x3)
print(x1.grad)
print(x3.grad)
Results are
tensor(2., requires_grad=True)
tensor(5., grad_fn=<AddBackward0>)
tensor(10.)
None
Why is x3.grad still None? Shouldn't it be tensor(10.) ?
When I run the following lines of code, x3.grad is evaluated to tensor(10.)
x3 = tensor(5.).requires_grad_()
l = (x3**2).mean()
l.backward()
print(x3.grad)

Answer 1

If you print x3.grad on your first example you might notice torch outputs a warning:
UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See here for more informations.
To save memory the gradients of the non-leaf tensors (non user-created tensors) are not buffered.
If you wish to see those gradients though you can retain the gradient on x3 by calling .retain_grad() before creating the graph (i.e. before calling .backward().
x3.retain_grad()
l.backward()
print(x3.grad)
will indeed output tensor(10.)