I'm trying to use GradCAM with a Deeplabv3 resnet50 model preloaded from torchvision, but in Captum I need to say the name of the layer (of type nn.module). I can't find any documentation for how this is done, does anyone possibly have any ideas of how to get the name of the final ReLu layer?
Thanks in advance!
You can have a look at its representation and get an idea of where it's located by simply printing it:
>>> model = torchvision.models.segmentation.deeplabv3_resnet50()
>>> model
DeepLabV3(
(backbone): IntermediateLayerGetter(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): Bottleneck(
...
To get the actual exact name of the layer you can loop over the modules with named_modules and only pick the nn.ReLU layers:
>>> relus = [name for name, module in model.named_modules() if isinstance(module, nn.ReLU)]
>>> relus
['backbone.relu',
'backbone.layer1.0.relu',
'backbone.layer1.1.relu',
'backbone.layer1.2.relu',
'backbone.layer2.0.relu',
'backbone.layer2.1.relu',
'backbone.layer2.2.relu',
'backbone.layer2.3.relu',
'backbone.layer3.0.relu',
'backbone.layer3.1.relu',
'backbone.layer3.2.relu',
'backbone.layer3.3.relu',
'backbone.layer3.4.relu',
'backbone.layer3.5.relu',
'backbone.layer4.0.relu',
'backbone.layer4.1.relu',
'backbone.layer4.2.relu',
'classifier.0.convs.0.2',
'classifier.0.convs.1.2',
'classifier.0.convs.2.2',
'classifier.0.convs.3.2',
'classifier.0.convs.4.3',
'classifier.0.project.2',
'classifier.3']
Then pick the last one:
>>> relus[-1]
'classifier.3'
Related
I am trying to use transfer learning for an image segmentation task, and my plan is to use the first few layers of a pretrained model (VGG16 for example) as an encoder and then will add my own decoder.
So, I can load the model and see the structure by printing it:
model = torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True)
print(model)
I get like this:
ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
.....
.....
.....
I can also access the specific layers with model.layer3 for example. Now, I am struggling with certain things.
How to cut the model and take every module from the beginning to the end of any layer (model.layer3 for example)?
How to freeze only this stripped part, and keep the newly added modules available for training?
For 1): Initialize the ResNet in your LightningModule and slice it until the part that you need. Then add your own head after that, and define forward in the order that you need. See this example, based on the transfer learning docs:
import torchvision.models as models
class ImagenetTransferLearning(LightningModule):
def __init__(self):
super().__init__()
# init a pretrained resnet
backbone_tmp = models.resnet50(pretrained=True)
num_filters = backbone_tmp.fc.in_features
layers = list(backbone_tmp.children())[:-1]
self.backbone = nn.Sequential(*layers)
# use the pretrained model to classify cifar-10 (10 image classes)
num_target_classes = 10
self.classifier = nn.Linear(num_filters, num_target_classes)
For 2): Pass a BackboneFinetuning callback to your trainer. This requires that your LightningModule has a self.backbone attribute containing the modules that you want to be frozen, as shown on the snippet above. You can also use the BaseFinetuning callback if you need different freeze-unfreeze behavior.
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import BackboneFinetuning
multiplicative = lambda epoch: 1.5
backbone_finetuning = BackboneFinetuning(200, multiplicative)
trainer = Trainer(callbacks=[backbone_finetuning])
The following is true for any child module of model, but I will answer your question with model.layer3 here:
model.layer3 will give you the nn.Module associated with layer n°3 of your model. You can call it directly as you would with model
>>> z = model.layer3(torch.rand(16, 128, 10, 10))
>>> z.shape
torch.Size([16, 256, 5, 5])
To freeze the model:
you could put the layer in eval mode which disables dropouts and makes BN layers use statistics learning during training. This is done with model.layer3.eval()
you must disable training on that layer by toggling the requires_grad flag: model.layer3.requires_grad_(False), this will affect all child parameters.
You can freeze the layers with:
pretrained_model.freeze()
https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.core.lightning.html?highlight=Freeze#pytorch_lightning.core.lightning.LightningModule.freeze
I have a pytorch binary classification model that I converted to coreML. I converted my model directly and indirectly through onnx using the following tutorials/documentation respectively
https://coremltools.readme.io/docs/pytorch-conversion, and
https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb .
The output prior to the softmax function and the probabilities are similar for both the original pytorch and the onnx model converted from PyTorch. But the output for the coreML model converted from PyTorch via the tutorial documentation is completely incorrect. I had no errors compiling the coreML method from either method.
Checking the weights of the last layer for coreML and Pytorch seem to be the same. the output of the coreML model prior to softmax gives me
{'classLabel': '_xx', 'classLabelProbs': {'_xx': 29.15625, 'xx': -22.53125}}
while the output from the pytorch model give me [-3.2185674 3.4477997]
The output of the conversion from onnx to coreML looks like...
58/69: Converting Node Type Add
59/69: Converting Node Type Relu
60/69: Converting Node Type Conv
61/69: Converting Node Type BatchNormalization
62/69: Converting Node Type Relu
63/69: Converting Node Type Conv
64/69: Converting Node Type BatchNormalization
65/69: Converting Node Type Add
66/69: Converting Node Type Relu
67/69: Converting Node Type GlobalAveragePool
68/69: Converting Node Type Flatten
69/69: Converting Node Type Gemm
Translation to CoreML spec completed. Now compiling the CoreML model.
Model Compilation done.
While the output of the pytorch model when I print looks like this for the final layer....
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=2, bias=True).
How do I go about resolving the quantitative errors produced from my coreML model that was converted from PyTorch?
It's probably an issue with your image preprocessing options: https://machinethink.net/blog/help-core-ml-gives-wrong-output/
update:
using the coreml unified api I have added a scaled layer. My outputs are not giving any probabilities for my classifier.
![last couple layers of converted pytorch model][1]
[1]: https://i.stack.imgur.com/9bzd2.png
the last layer prints out a tensor instead of probabilities. So I added a softmax function via the network.builder
builder.add_softmax(name="softmax", input_name="305", output_name="307:labelProbabilityLayerName")
the previous last node had the output name equal to "307:labelProbabilityLayerName" and I changed it to "305", prior to me adding the softmax(). this way the previous last node's output is the input to my softmax. Also, now the output can be passed to my softmax can now connect to the original string class printing out the intended probabilites.
I am still getting an error saying...
"RuntimeError: Error compiling model: "Error reading protobuf spec. validator error: Layer 'softmax' consumes an input named '307' which is not present in this network."."
Which doesnt make sense because I defined my softmax to consume '305' and also updated that last layer which is an innerproduct layer to output 305.
As we know that convolution layers are expensive to calculate , I would like to compute the output of the convolution layers once and use them to train the fully connected layer of my Resnet, in order to speed up the process.
In the case of a VGG model, we can compute the output of the first convolutional part as follows
x = model_vgg.features(inputs)
But how can I do to extract features from a Resnet?
Thanks in advance
I guess you can try hacking through the net. I'll use resnet18 as an example:
import torch
from torch import nn
from torchvision.models import resnet18
net = resnet18(pretrained=False)
print(net)
You'll see something like:
....
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=1000, bias=True)
Let's store the Linear layer somewhere, and in its place, put a dummy layer. Then the output of the whole net is actually the output of the conv layers.
x = torch.randn(4,3,32,32) # dummy input of batch size 4 and 32x32 rgb images
out = net(x)
print(out.shape)
>>> 4, 1000 # batch size 4, 1000 default class predictions
store_fc = net.fc # Save the actual Linear layer for later
net.fc = nn.Identity() # Add a layer which actually does nothing
out = net(x)
print(out.shape)
>>> 4, 512 # batch size 4, 512 features that are the input to the actual fc layer.
I want to finetune an FCN_ResNet101. I would like to change the last layer as my dataset has a different number of classes. Also, finetune only the FCN head.
For the former, is it enough to only change the num_classes argument when defining the model or I need to use something like this:
model = torchvision.models.segmentation.fcn_resnet101(pretrained=True)
model.classifier=nn.identity()
model.Conv2d = nn.Conv2d(
in_channels=256,
out_channels=nb_classes,
kernel_size=1,
stride=1
)
I took this piece of code from another thread. I am not sure if it is necessary to use nn.identity(). When I do, the last layer does not change but the last layer of the one to the last FCN!
And, how many layers must be changed so my FCN head is re_trianed?
I wrote it this way but I’m mostly confused about FCN_ResNet101 architecture.
model = torchvision.models.segmentation.fcn_resnet101(pretrained=True, progress=True, num_classes=?)
#model.classifier[4] = nn.Identity()
“”"
FCNHead(
(0): Conv2d(2048, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.1)
(4): Conv2d(512, 21, kernel_size=(1, 1), stride=(1, 1))
), FCNHead(
(0): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.1)
(4): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))
)]
“”"
#setting our own number of classes
layer_list = list(model.children())[-5:]
model_small = nn.Sequential(*list(model.children()))[-5:]
for param in model_small.parameters():
param.requires_grad = False
model_small.Conv2d = nn.Conv2d( in_channels=1024,kernel_size=(3,3),stride=(1,1))
model_small.BatchNorm2d = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
model_small.ReLU = nn.ReLU()
model_small.Dropout = nn.Dropout(p=0.1)
model_small.Conv2d = nn.Conv2d(
in_channels=256,
out_channels=nb_classes,
kernel_size=1,
stride=1
)
model = model_small.to(device)
Any guidance is very much appreciated!
In Keras, after creating a model, we can see its input and output shapes using model.input_shape, model.output_shape. For weights and config we can use model.get_weights() and model.get_config(), respectively.
What are the similar alternatives for PyTorch? Also is there any other functions we need to know for inspecting a PyTorch model?
To get the summary in PyTorch, we print model print(model) but this gives lesser information than model.summary(). Is there a better summary for PyTorch?
There is no "model.summary()" method in pytorch. You need to use built_in methods and fields of the model.
For example, I have customized inception_v3 model. To get the information I need to use other many different fields. For instance:
IN:
print(model) # print network architecture
OUT
Inception3(
(Conv2d_1a_3x3): BasicConv2d(
(conv): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
)
(Conv2d_2a_3x3): BasicConv2d(
(conv): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), bias=False)
(bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
)
(Conv2d_2b_3x3): BasicConv2d(
(conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
)
(Conv2d_3b_1x1): BasicConv2d(
(conv): Conv2d(64, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(80, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
)
(Conv2d_4a_3x3): BasicConv2d(
(conv): Conv2d(80, 192, kernel_size=(3, 3), stride=(1, 1), bias=False)
...
IN:
for i in model.state_dict().keys():
print(i)
#print keys of dict with values of learned weights, bias, parameters
OUT:
Conv2d_1a_3x3.conv.weight
Conv2d_1a_3x3.bn.weight
Conv2d_1a_3x3.bn.bias
Conv2d_1a_3x3.bn.running_mean
Conv2d_1a_3x3.bn.running_var
Conv2d_1a_3x3.bn.num_batches_tracked
Conv2d_2a_3x3.conv.weight
Conv2d_2a_3x3.bn.weight
Conv2d_2a_3x3.bn.bias
Conv2d_2a_3x3.bn.running_mean
...
So if I would like to get weights for the layer CNN at Conv2d_1a_3x3, I look for the key "Conv2d_1a_3x3.conv.weight":
print("model.save_dict()["Conv2d_1a_3x3.conv.weight"])
OUT:
tensor([[[[-0.2103, -0.3441, -0.0344],
[-0.1420, -0.2520, -0.0280],
[ 0.0736, 0.0183, 0.0381]],
[[ 0.1417, 0.1593, 0.0506],
[ 0.0828, 0.0854, 0.0186],
[ 0.0283, 0.0144, 0.0508]],
...
If you want to see used hyperparameters from optimizer:
optimizer.param_groups
OUT:
[{'dampening': 0,
'lr': 0.01,
'momentum': 0.01,
'nesterov': False,
'params': [Parameter containing:
tensor([[[[-0.2103, -0.3441, -0.0344],
[-0.1420, -0.2520, -0.0280],
[ 0.0736, 0.0183, 0.0381]],
...