I have a PyTorch tensor as (2,3,4) dimension I want to make this tensor like (3,2,4) dimension. But I want a value of (:,0,:) equal to (0,:,:). I tried torch.reshape and torch.view but I cannot get the expected result.
How can I do it in PyTorch?
Try this:
import torch
x = torch.randn(2, 0, 4)
print(x.shape)
x = x.permute(1, 0, 2)
print(x.shape)
Related
I want to train the model to sum the three inputs. So it is as simple as possible.
Firstly the weights are initialized randomly. It produces bad error estimate (approx. 0.5)
Then I initialize the weights with zeros. There are two options:
the shape of the weights tensor is [1, 3]
the shape of the weights tensor is [3]
When I choose the 1st option the model still works bad and can't learn this simple formula.
When I choose the 2nd option it works perfect with the error of 10e-12.
Why the result depends on the shape of the weights? Why do I need to initialize the model with zeros to solve this simple problem?
import torch
from torch.nn import Sequential as Seq, Linear as Lin
from torch.optim.lr_scheduler import ReduceLROnPlateau
X = torch.rand((1024, 3))
y = (X[:,0] + X[:,1] + X[:,2])
m = Seq(Lin(3, 1, bias=False))
# 1 option
m[0].weight = torch.nn.parameter.Parameter(torch.tensor([[0, 0, 0]], dtype=torch.float))
# 2 option
#m[0].weight = torch.nn.parameter.Parameter(torch.tensor([0, 0, 0], dtype=torch.float))
optim = torch.optim.SGD(m.parameters(), lr=10e-2)
scheduler = ReduceLROnPlateau(optim, 'min', factor=0.5, patience=20, verbose=True)
mse = torch.nn.MSELoss()
for epoch in range(500):
optim.zero_grad()
out = m(X)
loss = mse(out, y)
loss.backward()
optim.step()
if epoch % 20 == 0:
print(loss.item())
scheduler.step(loss)
First option doesn't learning because it fails with broadcasting: while out.shape == (1024, 1) corresponding targets y has shape of (1024, ). MSELoss, as expected, computes mean of tensor (out - y)^2, which in this case has shape (1024, 1024), clearly wrong objective for this task. At the same time, after applying 2-nd option tensor (out - y)^2 has size (1024, ) and mean of it corresponds to actual mse. Default approach, without explicit changing weights shape (through option 1 and 2), would work if set target shape to (1024, 1) for example by y = y.unsqueeze(-1) after definition of y.
I'm wondering about how to do the following thing:
If I have a torch.tensor x with shape (4,5,1) how can apply a neural network using PyTorch on the last dimension?
Using the standard procedure, the model is flattening the entire tensor into some new tensor of shape (20,1) but this is not actually what I want.
Let's say we want some output features of dimension 64, then I would like to obtain a new object of shape (4,5,64)
import torch
import torch.nn as nn
x = torch.randn(4, 5, 1)
print(x.size())
# https://pytorch.org/docs/stable/generated/torch.nn.Linear.html
m = nn.Linear(1, 64)
y = m(x)
print(y.size())
result:
torch.Size([4, 5, 1])
torch.Size([4, 5, 64])
For example, there is a 3-d tensor, I want to run the conv1d calculation on its third dimension,
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=5,stride=3,padding=0)
y = conv1d(x)
I hope the shape of y is (4,5,2,-1), but I get an error
Given groups=1, weight of size [2, 1, 5], expected input[4, 5, 6] to have 1 channels, but got 5 channels instead
Then I modified the code,
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=5,stride=3,padding=0)
x = x.unsqueeze(2)
y = conv1d(x)
There is another error:
Expected 3-dimensional input for 3-dimensional weight [2, 1, 5], but got 4-dimensional input of size [4, 5, 1, 6] instead
And if I want to run the maxpoo1d calulation in a tensor whose shape is (4,5,2,-1) ,in its last two dimension, what should I do?
I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this. Thank you all for your help.
I made an attempt, but I felt it couldn’t meet the actual needs, I wanted to know if it's good practice to do that and what would be the best way to do that?
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=2,stride=3,padding=0)
x = x.unsqueeze(2)
for i in range(4):
y = conv1d(x[i,:,:,:])
y = y.unsqueeze(0)
if i==0:
z = y
else:
z = torch.cat((z,y),0)
print(y)
print(z.size())
To use Conv1d you need your input to have 3 dimensions:
[batch_size, in_channels, data_dimension]
So, this would work:
x = torch.rand(4, 1, 50) # [batch_size=4, in_channels=1, data_dimension=50]
conv1d = nn.Conv1d(in_channels=1,out_channels=2,kernel_size=2,stride=3,padding=0)
x = conv1d(x)
print(x.shape) # Will output [4, 2, 16] 4=batch_size, 2=channels, 16=data_dimension
You can use MaxPool1d in the same way:
maxpool1d = nn.MaxPool1d(5)
x = maxpool1d(x)
print(x.shape) # Will output [4, 2, 3] 4=batch_size, 2=channels, 3=data_dimension
I sovle this question by torch.reshape(). I put the code here, hoping it could help somebody.
import torch
import torch.nn as nn
x = torch.rand(4,5,6)
conv1d =nn.Conv1d(in_channels=1,out_channels=2,kernel_size=2,stride=3,padding=0)
y = x.reshape(x.shape[0]*x.shape[1],-1)
y = y.unsqueeze(1)
y = conv1d(y)
z = y.reshape(x.shape[0],x.shape[1],2,-1)
print(z.size())
I have a Pytorch code which generates a Pytorch tensor in each iteration of for loop, all of the same size. I want to assign each of those tensors to a row of new tensor, which will include all the tensors at the end. In other works something like this
for i=1:N:
X = torch.Tensor([[1,2,3], [3,2,5]])
#Y is a pytorch tensor
Y[i] = X
I wonder how I can implement this with Pytorch.
You can concatenate the tensors using torch.cat:
tensors = []
for i in range(N):
X = torch.tensor([[1,2,3], [3,2,5]])
tensors.append(X)
Y = torch.cat(tensors, dim=0) # dim 0 is the rows of the tensor
I am trying to modify Resnet50 with my custom data as follows:
X = [[1.85, 0.460,... -0.606] ... [0.229, 0.543,... 1.342]]
y = [2, 4, 0, ... 4, 2, 2]
X is a feature vector of length 2000 for 784 images. y is an array of size 784 containing the binary representation of labels.
Here is the code:
def __classifyRenet(self, X, y):
image_input = Input(shape=(2000,1))
num_classes = 5
model = ResNet50(weights='imagenet',include_top=False)
model.summary()
last_layer = model.output
# add a global spatial average pooling layer
x = GlobalAveragePooling2D()(last_layer)
# add fully-connected & dropout layers
x = Dense(512, activation='relu',name='fc-1')(x)
x = Dropout(0.5)(x)
x = Dense(256, activation='relu',name='fc-2')(x)
x = Dropout(0.5)(x)
# a softmax layer for 5 classes
out = Dense(num_classes, activation='softmax',name='output_layer')(x)
# this is the model we will train
custom_resnet_model2 = Model(inputs=model.input, outputs=out)
custom_resnet_model2.summary()
for layer in custom_resnet_model2.layers[:-6]:
layer.trainable = False
custom_resnet_model2.layers[-1].trainable
custom_resnet_model2.compile(loss='categorical_crossentropy',
optimizer='adam',metrics=['accuracy'])
clf = custom_resnet_model2.fit(X, y,
batch_size=32, epochs=32, verbose=1,
validation_data=(X, y))
return clf
I am calling to function as:
clf = self.__classifyRenet(X_train, y_train)
It is giving an error:
ValueError: Error when checking input: expected input_24 to have 4 dimensions, but got array with shape (785, 2000)
Please help. Thank you!
1. First, understand the error.
Your input does not match the input of ResNet, for ResNet, the input should be (n_sample, 224, 224, 3) but you are having (785, 2000). From your question, you have 784 images with array of size 2000, which doesn't really align with the original ResNet50 input shape of (224 x 224) no matter how you reshape it. That means you cannot use the ResNet50 directly with your data. The only thing you did in your code is to take the last layer of ResNet50 and added you output layer to align with your output class size.
2. Then, what you can do.
If you insist to use the ResNet architecture, you will need to change the input layer rather than output layer. Also, you will need to reshape your image data to utilize the convolution layers. That means, you cannot have it in a (2000,) array, but need to be something like (height, width, channel), just like what ResNet and other architectures are doing. Of course you will also need to change the output layer as well just like you did so that you are predicting for your classes. Try something like:
model = ResNet50(input_tensor=image_input_shape, include_top=True,weights='imagenet')
This way, you can specify customized input image shape. You can check the github code for more information (https://github.com/keras-team/keras/blob/master/keras/applications/resnet50.py). Here's part of the docstring:
input_shape: optional shape tuple, only to be specified
if `include_top` is False (otherwise the input shape
has to be `(224, 224, 3)` (with `channels_last` data format)
or `(3, 224, 224)` (with `channels_first` data format).
It should have exactly 3 inputs channels,
and width and height should be no smaller than 197.
E.g. `(200, 200, 3)` would be one valid value.