I don't understand how the normalization in Pytorch works.
I want to set the mean to 0 and the standard deviation to 1 across all columns in a tensor x of shape (2, 2, 3).
A simple example:
>>> x = torch.tensor([[[ 1., 2., 3.],
[ 4., 5., 6.]],
[[ 7., 8., 9.],
[10., 11., 12.]]])
>>> norm = transforms.Normalize((0, 0), (1, 1))
>>> norm(x)
tensor([[[ 1., 2., 3.],
[ 4., 5., 6.]],
[[ 7., 8., 9.],
[10., 11., 12.]]])
So nothing has changed when applying the normalization transform. Why is that?
To give an answer to your question, you've now realized that torchvision.transforms.Normalize doesn't work as you had anticipated. That's because it's not meant to:
normalize: (making your data range in [0, 1]) nor
standardize: making your data's mean=0 and std=1 (which is what you're looking for.
The operation performed by T.Normalize is merely a shift-scale transform:
output[channel] = (input[channel] - mean[channel]) / std[channel]
The parameters names mean and std which seems rather misleading knowing that it is not meant to refer to the desired output statistics but instead any arbitrary values. That's right, if you input mean=0 and std=1, it will give you output = (input - 0) / 1 = input. Hence the result you received where function norm had no effect on your tensor values when you were expecting to get a tensor of mean and variance 0 and 1, respectively.
However, providing the correct mean and std parameters, i.e. when mean=mean(data) and std=std(data), then you end up calculating the z-score of your data channel by channel, which is what is usually called 'standardization'. So in order to actually get mean=0 and std=1, you first need to compute the mean and standard deviation of your data.
If you do:
>>> mean, std = x.mean(), x.std()
(tensor(6.5000), tensor(3.6056))
It will give you the global average, and global standard deviation respectively.
Instead, what you want is to measure the 1st and 2nd order statistics per-channel. Therefore, we need to apply torch.mean and torch.std on all dimensions expect dim=1. Both of those functions can receive a tuple of dimensions:
>>> mean, std = x.mean((0,2)), x.std((0,2))
(tensor([5., 8.]), tensor([3.4059, 3.4059]))
The above is the correct mean and standard deviation of x measured along each channel. From there you can go ahead and use T.Normalize(mean, std) to correctly transform your data x with the correct shift-scale parameters.
>>> norm(x)
tensor([[[-1.5254, -1.2481, -0.9707],
[-0.6934, -0.4160, -0.1387]],
[[ 0.1387, 0.4160, 0.6934],
[ 0.9707, 1.2481, 1.5254]]])
Follow the explanation on documentation of torchvision.transforms.Normalize:
Normalize a tensor image with mean and standard deviation. Given mean:
(mean[1],...,mean[n]) and std: (std[1],..,std[n]) for n channels, this
transform will normalize each channel of the input torch.*Tensor i.e.,
output[channel] = (input[channel] - mean[channel]) / std[channel]
So if you have mead=0 and std=1 then output=(output - 0) / 1 will not change.
Example to show above explanation:
from torchvision import transforms
import torch
norm = transforms.Normalize((0,0),(1,2))
x = torch.tensor([[[1.0,2,3],[4,5,6]],[[7,8,9],[10,11,12]]])
out = norm(x)
print(x)
print(out)
Outputs:
tensor([[[ 1., 2., 3.],
[ 4., 5., 6.]],
[[ 7., 8., 9.],
[10., 11., 12.]]])
tensor([[[1.0000, 2.0000, 3.0000],
[4.0000, 5.0000, 6.0000]],
[[3.5000, 4.0000, 4.5000],
[5.0000, 5.5000, 6.0000]]])
As you can see, the first channel is not change and second channel is divide by
2.
Related
Using torch.round() is it possible to eventually round specific entries of a tensor? Example:
tensor([ 8.5040e+00, 7.3818e+01, 5.2922e+00, -1.8912e-01, 5.4389e-01,
-3.6032e-03, 4.5763e-01, -2.7471e-02])
Desired output:
tensor([ 9., 74., 5., 0., 5.4389e-01,
-3.6032e-03, 4.5763e-01, -2.7471e-02])
(Only first 4 rounded)
you can do as follow
a[:4]=torch.round(a[:4])
Another (a little bit shorter) option is
t = torch.tensor([ 8.5040e+00, 7.3818e+01, 5.2922e+00, -1.8912e-01, 5.4389e-01, -3.6032e-03, 4.5763e-01, -2.7471e-02])
t[:4].round()
or inplace
t[:4].round_()
Lets see the result that I got first. This is one of a convolution layer of my model, and im only showing 11 filter's weight of it (11 3x3 filter with channel=1)
Left side is original weight Right side is Pruned weight
So I was wondering how does the "TORCH.NN.UTILS.PRUNE.L1_UNSTRUCTURED" works because by the pytorch website said, it prune the lowest L1-norm unit, but as far as I know, L1-norm pruning is a filter pruning method which prune the whole filter which use this equation to fine the lowest filter value instead of pruning single weight. So I'm a bit curious about how does this function actually works?
The following is my pruning code
parameters_to_prune = (
(model.input_layer[0], 'weight'),
(model.hidden_layer1[0], 'weight'),
(model.hidden_layer2[0], 'weight'),
(model.output_layer[0], 'weight')
)
prune.global_unstructured(
parameters_to_prune,
pruning_method=prune.L1Unstructured,
amount = (pruned_percentage/100),
)
The nn.utils.prune.l1_unstructured utility does not prune the whole filter, it prunes individual parameter components as you observed in your sheet. That is components with the lower norm get masked.
Here is a minimal example as discussed in the comments below:
>>> m = nn.Linear(10,1,bias=False)
>>> m.weight = nn.Parameter(torch.arange(10).float())
>>> prune.l1_unstructured(m, 'weight', .3)
>>> m.weight
tensor([0., 0., 0., 3., 4., 5., 6., 7., 8., 9.], grad_fn=<MulBackward0>)
I was trying to write a simple function to create a random adjacency matrix in the following way :
def create_adj(a):
a[a>0.5] = 1
a[a<=0.5] = 0
return a
given that a is assumed to be a torch.Tensor() as input, but I get the following error:
TypeError: 'int' object does not support item assignment
If I do things separately (i.e. not inside a function), I simply do:
>> a = torch.rand(3,3)
>> a[a>0.5] = 1
>> a[a<=0.5] = 0
>> a
tensor([[1., 1., 1.],
[0., 0., 0.],
[1., 0., 0.]])
But I don't understand what I'm doing wrong in the function.
I would assume you are not passing the correct variable your create_adj function. As long as a is a torch.tensor, then it should work.
Alternatively, you can directly use the mask as result:
def create_adj(x):
return (a > .5).float()
I am trying to get a specific range of values from my pytorch tensor.
tensor=torch.tensor([0,1,2,3,4,5,6,7,8,9])
new_tensor=tensor[tensor>2]
print(new_tensor)
This will give me a tensor with scalars of 3-9
new_tensor2=tensor[tensor<8]
print(new_tensor2)
This will give me a tensor with scalars of 0-7
new_tensor3=tensor[tensor>2 and tensor<8]
print(new_tensor3)
However this raises an error. Would I be able to get a tensor with the values of 3-7 using something like this? I am trying to edit the tensor directly, and do not wish to change the order of the tensor itself.
grad[x<-3]=0.1
grad[x>2]=1
grad[(x>=-3 and x<=2)]=siglrelu(grad[(x>=-3 and x<=2)])*(1.0-siglrelu(grad[(x>=-3 and x<=2)]))
This is what I am really going for, and I am not exactly sure of how to go about this. Any help is appreciated, thank you!
You can use & operation,
t = torch.arange(0., 10)
print(t)
print(t[(t > 2) & (t < 8)])
Output is,
tensor([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
tensor([3., 4., 5., 6., 7.])
how to upscale an image in Pytorch without defining height and width using transforms?
('--upscale_factor', type=int, required=True, help="super resolution upscale factor")
This might do the Job
transforms.Compose([transforms.resize(ImageSize*Scaling_Factor)])
If I understand correctly that you want to upsample a tensor x by just specifying a factor f (instead of specifying target width and height) you could try this:
from torch.nn.modules.upsampling import Upsample
m = Upsample(scale_factor=f, mode='nearest')
x_upsampled = m(x)
Note that Upsample allows for multiple interpolation modes, e.g. mode='nearest' or mode='bilinear'
Here is one interesting example:
input = torch.tensor([[1.,2.],[3.,4.]])
input=input[None]
input=input[None]
output = nn.functional.interpolate(input, scale_factor=2, mode='nearest')
print(output)
Out:
tensor([[[[1., 1., 2., 2.],
[1., 1., 2., 2.],
[3., 3., 4., 4.],
[3., 3., 4., 4.]]]])
You can do
image_tensor = transforms.functional.resize(image_tensor, size=(image_tensor.shape[1] * 2, image_tensor.shape[2] * 2))
or read out width and height using color, height, width = image_tensor.size() beforehand
check this example for reference on Resize as well.