I want to hand write a framework to perform inference of a given neural network. The network is so complicated, so to make sure my implementation is correct, I need to know how exactly the inference process is done on device.
I tried to use torchviz to visualize the network, but what I got seems to be the back propagation compute graph, which is really hard to understand.
Then I tried to convert the pytorch model to ONNX format, following the instruction enter link description here, but when I tried to visualize it, it seems that the original layers of the model had been seperated into very small operators.
I just want to get the result like this
How can I get this? Thanks!
Have you tried saving the model with torch.save (https://pytorch.org/tutorials/beginner/saving_loading_models.html) and opening it with Netron? The last view you showed is a view of the Netron app.
You can try also the package torchview, which provides several features (useful especially for large models). For instance you can set the display depth (depth in nested hierarchy of moduls).
It is also based on forward prop
github repo
Disclaimer: I am the author of the package
Note: The accepted format for tool is pytorch model
Related
I have an image dataset with soft labels (i.e. the images don't belong to a single class, but rather I have a probability distribution saying that there's a 66% chance this image belong in one class and 33% chance it belongs in some other class).
I am struggling to figure out how to setup my PyTorch code to allow this to be represented by the model and outputted correctly. The probabilities are saved in a csv file. I have looked at the PyTorch docs and other resources which mention the cross entropy loss function but I am still unclear how to import the data successfully and make use of soft labels.
What you are trying to solve is a multi-label classification task, i.e. instances can be classified with more than one label at a time. You cannot use torch.CrossEntropyLoss since it only allows for single-label targets. So you have two options:
Either use a soft version of the nn.CrossEntropyLoss function, this can be done by implementing the loss by hand allowing for soft targets. You can find such implementation on Soft Cross Entropy in PyTorch.
Or consider the task a multiple "independent" binary classification tasks, in this case, you would use nn.BCEWithLogitsLoss (this layer contains a sigmoid function).
Pytorch CrossEntropyLoss Supports Soft Labels Natively Now
Thanks to the Pytorch team, I believe this problem has been solved with the current version of the torch CROSSENTROPYLOSS.
You can directly input probabilities for each class as target (see the doc).
Here is the forum discussion that pushed this enhancement.
I would like to use AllenNLP Interpret (code + demo) with a PyTorch classification model trained with HuggingFace (electra base discriminator). Yet, it is not obvious to me, how I can convert my model, and use it in a local allen-nlp demo server.
How should I proceed ?
Thanks in advance
If your task is binary classification, you can look at the BoolQ example in https://github.com/allenai/allennlp-models/blob/main/training_config/classification/boolq_roberta.jsonnet. You can change that configuration to use a different model (such as Electra).
We also just put some new documentation out for the Interpret functionality: https://guide.allennlp.org/interpret
To give you a more specific answer, I'll need to know some more details, like what the task is you're trying to solve, how you trained the original model, etc.
I have a model, exported from pytorch, I'll call main_model.onnx. It has an input node I'll call main_input that expects a list of integers. I can load this in onnxruntime and send a list of ints and it works great.
I made another ONNX model I'll call pre_model.onnx with input pre_input and output pre_output. This preprocesses some text so input is the text, and pre_output is a list of ints, exactly as main_model.onnx needs for input.
My goal here is, using the Python onnx.helper tools, create one uber-model that accepts text as input, and runs through my pre-model.onnx, possibly some connector node (Identity maybe?), and then through main_model.onnx all in one big combined.onnx model.
I have tried using pre_model.graph.node+Identity connector+main_model.graph.node as nodes in a new graph, but the parameters exported from pytorch are lost this way. Is there a way to keep all those parameters and everything around, and export this one even larger combined ONNX model?
This is possible to achieve albeit a bit tricky. You can explore the Python APIs offered by ONNX (https://github.com/onnx/onnx/blob/master/docs/PythonAPIOverview.md). This will allow you to load models to memory and you'll have to "compose" your combined model using the APIs exposed (combine both the GraphProto messages into one - this is easier said than done - you' ll have to ensure that you don't violate the onnx spec while doing this) and finally store the new Graphproto in a new ModelProto and you have your combined model. I would also run it through the onnx checker on completion to ensure the model is valid post its creation.
If you have static size inputs, sclblonnx package is an easy solution for merging Onnx models. However, it does not support dynamic size inputs.
For dynamic size inputs, one solution would be writing your own code using ONNX API as stated earlier.
Another solution would be converting the two ONNX models to a framework(Tensorflow or PyTorch) using tools like onnx-tensorflow or onnx2pytorch. Then pass the outputs of one network as inputs of the other network and export the whole network to Onnx format.
I have built and trained neural network in Pytorch and is ready for production to a website but how do I deploy it?
There is multiple ways to do it.
Yet first, I add a PS : I noticed that you were asking specifically about reinforcement learning after having posted my answer. Know that even though I have written this answer with a static neural network model in mind, I offer at the end of the post a solution to apply the ideas of this answer to reinforcement learning.
The different options :
From what I know, using PyTorch in production is not especially recommended for big scale production.
It is more common to convert the PyTorch model to the ONNX format (a format to make ai models interchangeable between frameworks). Here is a tutorial if you want to operate this way : https://github.com/onnx/tutorials/blob/master/tutorials/PytorchOnnxExport.ipynb .
Then run it using the ONNX runtime, Caffe2 (by Facebook) or with TensorFlow (by Google).
My answer is not going to explore those solutions (and i did not include tutorials to those options), because i recently did the same as you are trying to do (building a neural network architecture and wanting to deploy it, and also allowing users to train their neural network with the architecture), yet i did not converted my neural network for the following reasons :
ONNX is evolving quickly, yet is currently not supporting all the operations you can possibly do in a PyTorch model. So if you have a highly custom or specific neural network (like in my case), you might not be able to convert it to ONNX with ease. You might need to change your architecture, or maybe have to re-write a big part of it so that it can be converted to ONNX.
You will need to use one or two additional tools, where most tutorials are not going really deep, or not explaining the logic behind what they are doing.
Note that you might want to convert your neural network if you call your network billions or trillions of times a day, otherwise i think you can stick with PyTorch without issues even for production, and avoid the fallback of converting to ONNX.
First let's see how we can save a trained neural network, load it back trough the architecture of the network, and re-run the trained network.
Second how we can deploy a network to a website, and also how you can allow users to train their networks. It is likely not the best or most efficient way, yet it sure works.
Saving the network :
First, you clearly need to have imported pyTorch with "import torch". Inside your neural network file you should save the stateDict (basically a dictionary of the operations and weights of your network) of the network you want to re-use. You could for example only save the stateDict of the model with the smallest loss of your epoch.
# network is the variable containing your neural network class
network_stateDict = network.state_dict()
# Saving network stateDict to a variable
Then when you want to save the stateDict to a file that you can re-use later, use :
torch.save(network_stateDict, "folderPath/myStateDict.pt)
# Saving the stateDict variable to a file
# The pt extension is just a convention in the PyTorch community, ptr is also used a lot
Finally when you will want to re-use your trained network later on, you will need to :
network = myNetwork(1, 2, 3)
# Load the architecture of the network in a variable (use the same architecture
# and the same network parameters as the ones used to create the stateDict)
network.load_state_dict(torch.load(folderPath/myStateDict.pt))
# Loading the file containing the stateDict of the trained network into a format
# pyTorch can read with the torch.load function. Then load the stateDict inside the
# network architecture with the load_state_dict function, applied to your network
# object with network.load_state_dict .
network.eval()
# To make sure that the stateDict has correctly been loaded.
output = network(input_data)
# You should now be able to get output data from your
# trained network, by feeding it a single set of input data.
For more infos on saving models and the stateDicts : https://pytorch.org/tutorials/beginner/saving_loading_models.html
Deploying the network:
Now that we know how to save, restore and feed input data to a network, all that there is left to do is to deploy it so that this process is done trough the website.
You first need to get (likely from your user) the inputs that your neural network will use. I am not going to include any link, since there is so many different web frameworks.
You would then need to either use a framework (like Django) that allow you to do in Python the logic of :
import torch
network = myNetwork(1, 2, 3)
network.load_state_dict(torch.load(folderPath/myStateDict.pt))
network.eval()
input_data = data_fromMyUser
output = network(input_data)
Then you would collect the output to display it, or do whatever you want it.
If your framework is not giving you the ability to use Python, i think it would be a good idea to have a tiny Python script, to which you would give the input data, and which would return the output.
If you would like to give the possibility to the user to train networks, you should just give them the possibility to start the training of one, and then use torch.save on a stateDict object to save the stateDict to a file.
You or they could later use the trained networks (you should also need to create a little function to make sure that you do not override previous stateDict files).
How to apply it to reinforcement learning :
I did not deploy a reinforcement learning model, yet i can offer you some ideas and leads to explore to deploy one.
You could store and add the inputs that you get from your user to a file or a database, and write a little program, that say every 24 hours or every hour, re-run the neural network with the now bigger dataset.
You could then totally apply the suggestions in this answer, of running the network, saving the stateDict of the model and then changing the stateDict that your network is using in production.
This is a bit hacky, yet would allow you to save in a "static way" your trained networks, and still have them evolving and changing their stateDicts.
Conclusion
This is clearly not the most mass-scale production approach that you could employ, yet it is in my opinion the easiest to put in place.
You also know that the output that you will get, will be the actual output of your neural network, without any distorsions or errors in the values.
Have a great day !
save the trained model however you want (HD5 or with pickle)
write the program to handle in production by loading the trained model
deploy the program on distributed system for real time computation like on Apache storm, Flink, Alink, Apache Samoa etc..
if you feel you need to retrain the model depending on feedback then retrain the model on different cluster or parallel environment and observe the model accuracy if looks good then move the model to production (initial days you need to retrain multiple times and it will decrease time goes on if your model is designed in a good way)
I´m new here, so please be kind and teach me if I did not provide all the information you need :)
I would like to compare Edge TPU with other edge device such as Myriad. I would like to select one object detection model and one image segmentation model. Considering the following link which shows supported operations, I have noticed that yolov3 cannot be compiled for EdgeTPU because it includes LeakyRelu.
https://coral.withgoogle.com/docs/edgetpu/models-intro/
For image segmentation, I'd like to use Deeplab. But I'm still don't know if operations included in deeplab v3+, such as atrous convolution or feature pyramid network, are supported.
I'd appreciate if someone teach me what models are usable on edgeTPU. Are there any models of image segmentation?
Did you already found below?
https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/quantize.md
"mobilenetv2_coco_voc_trainaug_8bit":
deeplabv3_mnv2_pascal_train_aug_8bit/frozen_inference_graph.pb
This model is possible to converting to TFLite FlatBuffer.
And also possible to compile for edgetpu with edgetpu_compiler.
Note. edgetpu_api environment had updated.
You can find it below.
https://coral.withgoogle.com/news/updates-07-2019/
Yes. There are prepackaged segmentation models and code examples how to use them.
Here they are https://coral.ai/models/
Please share if you know where to find something similar for Movidius based VPU devices.
Here you can find all supported layers for edgetpu: https://coral.ai/docs/edgetpu/models-intro/#supported-operations.
And for Conv2D it says "Must use the same dilation in x and y dimensions.". So implementing a version of deeplab v3+ is possible for the edgetpu.