How can I get argmaxed torch tensor excluding certain index? - pytorch

I wonder if I can get torch.argmax of my input excluding certain index.
For example,
target = torch.tensor([1,2])
input = torch.tensor([[0.1,0.5,0.2,0.2], [0.1,0.5,0.1,0.3]])
I want to get the maximum value in input excluding the index on the target, so that the result would be
output = torch.tensor([[0.2],[0.5]])

You can try this
Set negative infy to the target indices in temp tensor
Then use torch.max or torch.argmax
tmp_input = input.clone()
tmp_input[range(len(input)), target] = float("-Inf")
torch.max(tmp_input, dim=1).values
tensor([0.2000, 0.5000])
torch.max(tmp_input, dim=1).indices
tensor([3, 1])
torch.argmax(tmp_input, dim=1)
tensor([3, 1])

input[target[0]-1,target[1]-1] = -1 # or use -inf
#-1 is added for python indexing style
output = torch.max(input,dim = 1)

Related

Transforms.Normalize returns values higher than 255 Pytorch

I am working on an video dataset, I read the frames as integers and convert them to a numpy array float32.
After being loaded, they appear in a range between 0 and 255:
[165., 193., 148.],
[166., 193., 149.],
[167., 193., 149.],
...
Finally, to feed them to my model and stack the frames I do the "ToTensor()" plus my transformation [transforms.Resize(224), transforms.Normalize([0.454, 0.390, 0.331], [0.164, 0.187, 0.152])]
and here the code to transform and stack the frames:
res_vframes = []
for i in range(len(v_frames)):
res_vframes.append(self.transforms((v_frames[i])))
res_vframes = torch.stack(res_vframes, 0)
The problem is that after the transformation the values appears in this way, which has values higher than 255:
[tensor([[[1003.3293, 1009.4268, 1015.5244, ..., 1039.9147, 1039.9147,
1039.9147],...
Any idea on what I am missing or doing wrong?
The behavior of torchvision.transforms.Normalize:
output[channel] = (input[channel] - mean[channel]) / std[channel]
Since the numerator of the lefthand of the above equation is greater than 1 and the denominator of it is smaller than 1, the computed value gets larger.
The class ToTensor() maps a tensor's value to [0, 1] only if some condition is satisfied. Check this code from official Pytorch docs:
if isinstance(pic, np.ndarray):
# handle numpy array
if pic.ndim == 2:
pic = pic[:, :, None]
img = torch.from_numpy(pic.transpose((2, 0, 1))).contiguous()
# backward compatibility
if isinstance(img, torch.ByteTensor):
return img.to(dtype=default_float_dtype).div(255)
else:
return img
Therefore you need to divide tensors explicitly or make to match the above condition.
Your normalization uses values between 0-1 and not 0-255.
You need to change your input frames to 0-1 or the normalization vectors to 0-255.
You can divide the frames by 255 before using the transform:
res_vframes = []
for i in range(len(v_frames)):
res_vframes.append(self.transforms((v_frames[i]/255)))
res_vframes = torch.stack(res_vframes, 0)

Pytorch dimension change

Is there any methods to change [1,512,1,1] to [1,512,2,2] tensor.
I know it is not possible just by changing the dimensions.
Are there any ways using concat or stack with PyTorch (torch.stack, torch.cat)
I make tensor with following code
a = torch.rand([1,512,1,1])
How can I change this to tensor with dimension [1,512,2,2]
That would be with torch.repeat, this will copy the data:
>>> a = a.repeat(1, 1, 2, 2)
If you do not wish to copy the data, then use torch.expand:
>>> a = a.expand(-1, -1, 2, 2)
I tried this
tmp = torch.cat([a,a],2)
a = torch.cat([tmp,tmp],3)

Backtransforming a PyTorch Tensor

I have trained a WGAN on the CelebA dataset in PyTorch following this youtube video. Since I do this on Google Cloud Platform where TensorBoard is not availabe, I save one figure of generated images by the GAN every epoch to see how the GAN is actually doing.
Now, the saved pdf files look sth like this: generated images. Unfortunately, this is not really readable, and I suspect this has to do with the preprocessing I do:
trafo = transforms.Compose(
[transforms.Resize(size = (64, 64)),
transforms.ToTensor(),
transforms.Normalize( mean = (0.5,), std = (0.5,))])
Is there any way to kind of undo this transformation when I save the image?
Currently, I save the image every epoch as follows:
visualization = torchvision.utils.make_grid(
tensor = gen(fixed_noise),
nrow = 8,
normalize = False)
plt.savefig("generated_WGAN_" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".pdf")
Also, I should probably mention that in the Jupyter notebook, I get the following warning:
"Clipping input data to the valid range for imshow with RGB data ([0..1]) for floats or [0..255] for integers)."
The torchvision.transform.Normalize function is usually used to standardize data (make mean(data)=0 and std(x)=1) while the normalize option on torchvision.utils.make_grid is used to normalize the data between [0,1] given a range. So no need to implement a function to fix this.
If True, shift the image to the range (0, 1), by the min and max values specified by range. Default: False.
Here you are looking to normalize between 0 and 1. Given a tensor x:
torchvision.utils.make_grid(x, nrow=8, normalize=True, range=(x.min(), x.max()))
Here are some examples of use provided by the PyTorch's documentation.
Back to your original question, I should mention that torchvision.transform.Normalize(mean=0.5, std=0.5) doesn't transform your data such that it has mean=0.5 and std=0.5... Neither will it standardize it to mean=0, std=1. You have to measure the mean and std from your dataset.
torchvision.transform.Normalize simply performs a shift-scale operation. To undo that just unscale-unshift with the same values:
>>> x = torch.rand(64, 3, 100, 100)*torch.rand(64, 1, 1, 1)
>>> x.mean(), x.std()
(tensor(0.2536), tensor(0.2175))
>>> t = T.Normalize(mean, std)
>>> t_inv = lambda x: x*std + mean
>>> x_after = t(x)
>>> x_after.mean(), x_after.std()
(tensor(-0.4928), tensor(0.4350))
>>> x_before = t_inv(x_after)
>>> x_before.mean(), x_before.std()
(tensor(0.2536), tensor(0.2175))
It seems like your output pixel values are in range [-1, 1] (please verify this).
Therefore, when you save the images, the negative part is being clipped (as the error message you got suggests).
Try:
visualization = torchvision.utils.make_grid(
tensor = torch.clamp(gen(fixed_noise), -1, 1) * 0.5 + 0.5, # from [-1, 1] -> [0, 1]
nrow = 8,
normalize = False)
plt.savefig("generated_WGAN_" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".pdf")

how can I insert a Tensor into another Tensor in pytorch

I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.
I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.
I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.
I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.
A = Variable(torch.zeros(batch_size, step, vec_size))
What I'm doing is like:
for i in range(batch_size):
pos = P[i]
A[i][pos] = A[i][pos] + B[i]
But I get an Error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Then, I make a clone of A each inside the loop:
for i in range(batch_size):
A_clone = A.clone()
pos = P[i]
A_clone[i][pos] = A_clone[i][pos] + B[i]
This is very slow for autograd, I wonder if there any better solutions? Thank you.
You can use a mask instead of cloning.
See the code below
# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()
# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0
# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)
This way you avoid using for loops and cloning as well.

ValueError: array must not contain infs or NaNs in SpectralCoclustering in Python3.X

data = np.genfromtxt("breastCancer.txt", delimiter=',').astype(np.float32)
data = data[~np.isnan(data).any(axis=1)]
ROW, COLUMN = data.shape
label = data[:, -1]
input = data[:, 1:COLUMN - 1]
scaler = preprocessing.MinMaxScaler(feature_range=(-1.0, 1.0))
scaler.fit(input)
input = scaler.transform(input)
model = SpectralCoClustering(n_clusters=3, random_state=0)
model.fit(input)
I tried to get biclustering of dataset that is in a range of (-1.0, 1.0). In my data I don't have any inf or nan. But it throws an error of ValueError: array must not contain infs or NaNs in SpectralCoclustering. Can you please help me? I need a dataset in a range (-1, 1) so I dont want to change it in a positive range.
I spent two days figuring out the same problem. My solution: before doing model.fit(input) I removed columns with only zeros from input:
remaining_collumns=input.getnnz(1)>0
input=input[remaining_collumns,:]
model.fit(input)
Also note, that in my case input is returned by TfidfVectorizer.fit_transform and was in form of sparse.csr.csr_matrix

Resources