I am using facenet model.... When i am doing classifier training it shows this message, but image alignment process with this model is going good...
def load_model(model):
# Check if the model is a model directory (containing a metagraph and a checkpoint file)
# or if it is a protobuf file with a frozen graph
model_exp = os.path.expanduser(model)
if (os.path.isfile(model_exp)):
print('Model filename: %s' % model_exp)
with gfile.FastGFile(model_exp,'rb') as f:
graph_def = tf.GraphDef()
print("Graph def value: ",graph_def)
print(type(graph_def))
graph_def.ParseFromString(f.read())
tf.import_graph_def(graph_def, name='')
Can anyone help me to clear this issue?
And also the above code works well in local the issue occurs in heroku server
In the above code the print statement shows an op as,
Graph def value:
<class 'tensorflow.core.framework.graph_pb2.GraphDef'>
Below is a screenshot for an issue:
The error is due to the model serving support is not working on heroku... Better you need to use the paid account on heroku with machinelearning dependencies... Or you can go with some other online deployment which supports tensorflow model serve.
Related
I am having this problem trying to use LayoutParser but I believe it is actually an issue with Pytorch checkpoint. I get the following assert
The directory contains model.pth?dl=1 and model.pth.lock
It looks like the downloading creates a file called model.pth?dl=1 and and a .lock file. But it fails to finish by renaming it to model.pth. It seems to happen with all LayoutParser pretrained models.
Anyone seen this?
Thank you
Peter
The assert is thrown from this call with any of the pretrained models
The full code to reproduce this is:
import layoutparser as lp
if __name__ == "__main__":
model = lp.Detectron2LayoutModel(
config_path="lp://PubLayNet/faster_rcnn_R_50_FPN_3x/config",
label_map={0:"Text",1:"Title",2:"List",3:"Table",4:"Figure"},
extra_config=["MODEL.ROI_HEADS.SCORE_THRESH_TEST",0.1]
)
Having followed Chris McCormick's tutorial for creating a BERT Fake News Detector (link here), at the end he saves the PyTorch model using the following code:
output_dir = './model_save/'
if not os.path.exists(output_dir):
os.makedirs(output_dir)
# Save a trained model, configuration and tokenizer using `save_pretrained()`.
# They can then be reloaded using `from_pretrained()`
model_to_save = model.module if hasattr(model, 'module') else model
model_to_save.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)
As he says himself, it can be reloaded using from_pretrained(). Currently, what the code does is create an output directory with 6 files:
config.json
merges.txt
pytorch_model.bin
special_tokens_map.json
tokenizer_config.json
vocab.json
So how can I use the from_pretrained() method to load the model with all of its arguments and respective weights, and which files do I use from the six?
I understand that a model can be loaded as such (from PyTorch documentation):
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()
but how can I make use of the files in the output directory to do this?
Any help is appreciated!
I am using fastai with pytorch to fine tune XLMRoberta from huggingface.
I've trained the model and everything is fine on the machine where I trained it.
But when I try to load the model on another machine I get OSError - Not Found - No such file or directory pointing to .cache/torch/transformers/. The issue is the path of a vocab_file.
I've used fastai's Learner.export to export the model in .pkl file, but I don't believe that issue is related to fastai since I found the same issue appearing in flairNLP.
It appears that the path to the cache folder, where the vocab_file is stored during the training, is embedded in the .pkl file:
The error comes from transformer's XLMRobertaTokenizer __setstate__:
def __setstate__(self, d):
self.__dict__ = d
self.sp_model = spm.SentencePieceProcessor()
self.sp_model.Load(self.vocab_file)
which tries to load the vocab_file using the path from the file.
I've tried patching this method using:
pretrained_model_name = "xlm-roberta-base"
vocab_file = XLMRobertaTokenizer.from_pretrained(pretrained_model_name).vocab_file
def _setstate(self, d):
self.__dict__ = d
self.sp_model = spm.SentencePieceProcessor()
self.sp_model.Load(vocab_file)
XLMRobertaTokenizer.__setstate__ = MethodType(_setstate, XLMRobertaTokenizer(vocab_file))
And that successfully loaded the model but caused other problems like missing model attributes and other unwanted issues.
Can someone please explain why is the path embedded inside the file, is there a way to configure it without reexporting the model or if it has to be reexported how to configure it dynamically using fastai, torch and huggingface.
I faced the same error. I had fine tuned XLMRoberta on downstream classification task with fastai version = 1.0.61. I'm loading the model inside docker.
I'm not sure about why the path is embedded, but I found a workaround. Posting for future readers who might be looking for workaround as retraining is usually not possible.
I created /home/.cache/torch/transformer/ inside the docker image.
RUN mkdir -p /home/<username>/.cache/torch/transformers
Copied the files (which were not found in docker) from my local /home/.cache/torch/transformer/ to docker image /home/.cache/torch/transformer/
COPY filename:/home/<username>/.cache/torch/transformers/filename
I have been trying to use this Github (https://github.com/AntixK/PyTorch-VAE) and call the CelebA dataset using the config file listed. Specifically under the vae.yaml I have placed the path of the unzipped file where I have downloaded the celeba dataset (https://www.kaggle.com/jessicali9530/celeba-dataset) on my computer. And every time I run the program, I keep getting these errors:
File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/celeba.py", line 67, in init
' You can use download=True to download it')
RuntimeError: Dataset not found or corrupted. You can use download=True to download it
AttributeError: 'VAEXperiment' object has no attribute '_lazy_train_dataloader'
I have tried to download the dataset, but nothing changes. So I have no idea why the program is not running.
The run.py calls the experiment.py which uses this dataloader to retrieve the information:
def train_dataloader(self):
transform = self.data_transforms()
if self.params['dataset'] == 'celeba':
dataset = CelebA(root = self.params['data_path'],
split = "train",
transform=transform,
download=False)
else:
raise ValueError('Undefined dataset type')
self.num_train_imgs = len(dataset)
return DataLoader(dataset,
batch_size= self.params['batch_size'],
shuffle = True,
drop_last=True)
The config file grabs the information passed on the root. So what I did was upload a few files to google colab (some .jpg files) and when I run the command stated in the GItHub, python run.py -c config/vae.yaml, it states that the dataset is not found or is corrupt. I have tried this on my linux machine and the same error occurs, even when I used the downloaded and unzip link. I have gone further to attempt to change the self.params['data_path'] to the actual path and that still does not work. Any ideas what I can do?
My pytorch version is 1.6.0.
There are two issues which I have faced. The below is my solution. It is not official but it works for me. Hope the next pytorch version will update it.
Issue: Dataset not found or corrupted.'
When I checked file celeba.py in pytorch library. I found this line:
if ext not in [".zip", ".7z"] and not check_integrity(fpath, md5):
return False
This part will make self._check_integrity() return False and the program provides the message error as we got.
Solve: You can ignore this part by add "if False" immediately in front of this line
if False:
if ext not in [".zip", ".7z"] and not check_integrity(fpath, md5):
return False
celeba.py downloads dataset if you choose download=True but these two files are broken "list_landmarks_align_celeba.txt" and "list_attr_celeba.txt"
You need to find somewhere, download and replace them
Hope these solutions will help you !!!!
https://learn.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/python-tutorial
I followed the above tutorial for using the Azure Custom Vision Python SDK. Instead of using an image on the internet for prediction (as shown in the tutorial), I would like to use an image file from my computer. How can I do that? Thanks!
There is a sample in the Github project hosted for the tutorial you mentioned:
It is for Object Detection but the call is the same for Classification, the difference is in the content of the result (here you have bounding_box items because object detection is predicting zones in the image):
def predict_project(prediction_key, project, iteration):
predictor = CustomVisionPredictionClient(prediction_key, endpoint=ENDPOINT)
# Open the sample image and get back the prediction results.
with open(os.path.join(IMAGES_FOLDER, "Test", "test_od_image.jpg"), mode="rb") as test_data:
results = predictor.predict_image(project.id, test_data, iteration.id)
# Display the results.
for prediction in results.predictions:
print ("\t" + prediction.tag_name + ": {0:.2f}%".format(prediction.probability * 100), prediction.bounding_box.left, prediction.bounding_box.top, prediction.bounding_box.width, prediction.bounding_box.height)
See source here: https://github.com/Azure-Samples/cognitive-services-python-sdk-samples/blob/master/samples/vision/custom_vision_object_detection_sample.py#L122