I am getting an error running the validate() function of pytorch lighting using the following code.
error:
ValueError: An invalid dataloader was passed to `Trainer.validate(dataloaders=...)`. Either pass the dataloader to the `.validate()` method OR implement `def val_dataloader(self):` in your LightningModule/LightningDataModule.
code:
from torchvision.datasets import MNIST
from torchvision import transforms
from torch.utils.data import DataLoader
...
mnist_val = MNIST(root='data',train=False, download=True, transform=transform)
mnist_val_loader = DataLoader(mnist_val, batch_size=256, num_workers=4)
...
trainer.validate()
I used the data loader into the validate() function but I get the following error:
Unwrapping the module did not yield a `LightningModule`
I solved with the newer versions of pytorch_lighting putting as input model & data loader to trainer.validate()
trainer.validate(model, mnist_val_loader)
Related
I'm trying to obtain the "last_hidden_state" (as explained here) for code generation models over here. I am unable to figure out how to proceed, other than manually downloading each code-generation-model and checking if its key has that attribute using the following code -
import numpy as np
from datasets import load_dataset
from transformers import AutoTokenizer
from transformers import AutoModel, AutoModelForCausalLM
import torch
from sklearn.linear_model import LogisticRegression
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = AutoModelWithLMHead.from_pretrained("codeparrot/codeparrot").to(device)
inputs = tokenizer("def hello_world():", return_tensors="pt")
inputs = {k:v.to(device) for k,v in inputs.items()}
with torch.no_grad():
outputs = model(**inputs)
print(outputs.keys())
So far, I tried this strategy on CodeParrot and InCoder with no success. Perhaps there is a better way to access the values of the hidden layers?
The hidden_states of output from CodeGenForCausalLM is already the last_hidden_state for the codegen model. See: link
where hidden_states = transformer_outputs[0] is the output of CodeGenModel (link) and the transformer_outputs[0] is the last_hidden_state
if not return_dict:
return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
return BaseModelOutputWithPast(
last_hidden_state=hidden_states,
past_key_values=presents,
hidden_states=all_hidden_states,
attentions=all_self_attentions,
)
I would like to check if model is on CUDA. How to do that?
import torch
import torchvision
model = torchvision.models.resnet18()
model.to('cuda')
Seams that model.is_cuda() is not working.
This code should do it:
import torch
import torchvision
model = torchvision.models.resnet18()
model.to('cuda')
next(model.parameters()).is_cuda
Out:
True
Note there is no is_cuda() method inside nn.Module.
Also note model.to('cuda') is the same as model.cuda() and both are inplace.
On the other hand moving the data.to('cuda') is not inplace and you typically call:
data = data.to('cuda')
to move the data to CUDA.
Following this Tutorial
to visualize images in tensorboard using torch.utils.tensorboard got error
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter('runs/fashion_mnist_experiment_1')
writer.add_embedding(features,
metadata=class_labels,
label_img=images.unsqueeze(1))
writer.close()
Error:
AttributeError: module 'tensorflow_core._api.v2.io.gfile' has no attribute 'get_filesystem'
try adding
writer = SummaryWriter(path)
img_grid = torchvision.utils.make_grid(torch.FloatTensor(imgs)) #imgs represent a batch of images with shape BxCxHxW
writer_add_image('images', img_grid)
source : https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#writing-to-tensorboard
I am trying to convert a RandomForestClassifier model of the MNIST dataset into a CoreML model using coremltools.
I use fetch_mldata from sklearn.datasets to import the data; my samples (data) are stored in a Numpy array X, and the labels (target) are stored in a Numpy array Y.
My model is generated like this:
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier()
rfc.fit(X, Y)
I then try to convert the model:
import coremltools
model = coremltools.converters.sklearn.convert(rfc)
The CoreML model input then looks like this:
input {
name: "input"
type {
multiArrayType {
shape: 784
dataType: DOUBLE
}
}
}
The problem is the multiArrayType. It is much easier to deal with a pixel-buffer in iOS, so various sources point to this syntax (however, they use Caffe rather than sklearn):
model = coremltools.converters.sklearn.convert(rfc, image_input_names='data')
However, this gives me an error message:
TypeError: convert() got an unexpected keyword argument 'image_input_names'
I have tried to find the documentation for these parameters, but I have only found some few example using Caffe, and they do not seem to get this error.
I want to initialize a 4*11 matrix using glorot uniform in Keras, using following code:
import keras
keras.initializers.glorot_uniform((4,11))
I get this output :
<keras.initializers.VarianceScaling at 0x7f9666fc48d0>
How can I visualize the output? I have tried c[1] and got output 'VarianceScaling' object does not support indexing.
The glorot_uniform() creates a function, and later this function will be called with a shape. So you need:
# from keras.initializers import * #(tf 1.x)
from tensorflow.keras.initializers import *
unif = glorot_uniform() #this returns a 'function(shape)'
mat_as_tensor = unif((4,11)) #this returns a tensor - use this in keras models if needed
mat_as_numpy = K.eval(mat) #this returns a numpy array (don't use in models)
print(mat_as_numpy)