I'm trying to train a quantize model in pytorch and convert it to ONNX.
I employ the quantized-aware-training technique with help of pytorch_quantization package.
I used the below code to convert my model to ONNX:
from pytorch_quantization import nn as quant_nn
from pytorch_quantization import calib
from pytorch_quantization.tensor_quant import QuantDescriptor
from pytorch_quantization import quant_modules
import onnxruntime
import torch
import torch.utils.data
from torch import nn
import torchvision
def export_onnx(model, onnx_filename, batch_onnx, per_channel_quantization):
model.eval()
quant_nn.TensorQuantizer.use_fb_fake_quant = True # We have to shift to pytorch's fake quant ops before exporting the model to ONNX
if per_channel_quantization:
opset_version = 13
else:
opset_version = 12
# Export ONNX for multiple batch sizes
print("Creating ONNX file: " + onnx_filename)
dummy_input = torch.randn(batch_onnx, 3, 224, 224, device='cuda') #TODO: switch input dims by model
input_names = ['input']
output_names = ['Linear[fc]'] ### ResNet34
dynamic_axes = {'input': {0: 'batch_size'}}
try:
torch.onnx.export(model, dummy_input, onnx_filename, input_names=input_names,
export_params=True, output_names=output_names, opset_version=opset_version,
verbose=True, enable_onnx_checker=False, do_constant_folding=True)
except ValueError:
warnings.warn(UserWarning("Per-channel quantization is not yet supported in Pytorch/ONNX RT (requires ONNX opset 13)"))
print("Failed to export to ONNX")
return False
return True
After conversion, I get the following warnings:
warnings.warn("'enable_onnx_checker' is deprecated and ignored. It will be removed in "
W0305 12:39:40.472136 140018114328384 tensor_quantizer.py:280] Use Pytorch's native experimental fake quantization.
/usr/local/lib/python3.8/dist-packages/pytorch_quantization/nn/modules/tensor_quantizer.py:285: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
Also, the accuracy is not valid for ONNX model!
Accuracy summary:
+-----------+-------+
| Stage | Top1 |
+-----------+-------+
| Finetuned | 38.03 |
| ONNX | -1.00 |
+-----------+-------+
More info is here:
pytorch 1.10.2+cu102
torchvision 0.11.3+cu102
TensorRT 8.2.3-1+cuda11.4
ONNX 1.11.0
ONNX Runtime 1.10.0
cuda 11.6
python 3.8
What is the problem with ONNX conversion?
After some tries, I found that there is a version conflict. I changed the versions accordingly:
onnx == 1.9.0
onnxruntime == 1.8.1
pytorch == 1.9.0+cu111
torchvision == 0.10.0+cu111
Related
I'm trying to convert a Pytorch model to ONNX usin this code:
import onnx
import torch
import onnxruntime
import numpy as np
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
batch_size = 1
model = torch.jit.load('../saved_weights/model.pt')
x = torch.randn(1, 3, 224, 224, requires_grad=True)
x = x.to(device)
torch_out = model(x)
torch_out = torch_out['out']
torch.onnx.export(model, x, "../saved_weights/model.onnx",
export_params=True,
opset_version=12,
do_constant_folding=True,
input_names=['input'],
output_names=['out'],
dynamic_axes={ 'input': {0 : 'batch_size'},
'output': {0 : 'batch_size'}})
However I'm getting the following error:
raise errors.UnsupportedOperatorError(
torch.onnx.errors.UnsupportedOperatorError: Exporting the operator
'aten::dict' to ONNX opset version 12 is not supported.
I'm not sure why, but could it be that the model output is in the form of a dictionary?
Referring Pytorch doc, you can try:
Change the model to not use that operator (here, dictionary).
Create a symbolic function to convert the operator and register it as a custom symbolic function.
Contribute to PyTorch to add the same symbolic function to torch.onnx itself.
My recommendation is to change the output type to a tensor.
Also, you can visit here and check supported operators.
Solution: I had to save the model using model.save instead of a scripted model:
model_scripted = torch.jit.script(model) # Export to TorchScript
model_scripted.save('model_scripted.pt') # Save
It is a multi-class classification model with sklearn.
I am using OneVsOneClassifier model to train and predict 150 intents. Its a multi-class classification problem.
Data:
text intents
text1 int1
text2 int2
I convert these intents in labels using:
le = LabelEncoder()
y_train = le.fit_transform(y_train)
y_test = le.fit_transform(y_test)
Expectation:
Without changing the training pipeline or parameters, note the inference time. Currently, it's slow, ~1second for 1 inference. So to convert pipeline to ONNX format and then use for inferencing on 1 example.
Code:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.compose import ColumnTransformer
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC,LinearSVC
def create_pipe(clf):
# Each pipeline uses the same column transformer.
column_trans = ColumnTransformer(
[('Text', TfidfVectorizer(), 'text')
],
remainder='drop')
pipeline = Pipeline([('prep',column_trans),
('clf', clf)])
return pipeline
def fit_and_print(pipeline):
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
print(metrics.classification_report(y_test, y_pred,
target_names=le.classes_,
digits=3))
clf = OneVsOneClassifier(LinearSVC(random_state=42, class_weight='balanced'))
pipeline = create_pipe(clf)
%time fit_and_print(pipeline)
# convert input to df
def create_test_data(x):
d = {'text' : x}
df = pd.DataFrame(d, index=[0])
return df
revs=[]
for idx in [948, 5717, 458]:
cur = test.loc[idx, 'text']
revs.append(cur)
print(revs)
revs=sam['text'].values
%%time
for rev in revs:
c_res = pipeline.predict(create_test_data(rev))
print(rev, '=', labels[c_res[0]])
ONNX conversion code
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType, StringTensorType
initial_type = [('UTTERANCE', StringTensorType([None, 2]))]
model_onnx = convert_sklearn(pipeline, initial_types=initial_type)
Error
MissingShapeCalculator: Unable to find a shape calculator for type '<class 'sklearn.multiclass.OneVsOneClassifier'>'.
It usually means the pipeline being converted contains a
transformer or a predictor with no corresponding converter
implemented in sklearn-onnx. If the converted is implemented
in another library, you need to register
the converted so that it can be used by sklearn-onnx (function
update_registered_converter). If the model is not yet covered
by sklearn-onnx, you may raise an issue to
https://github.com/onnx/sklearn-onnx/issues
to get the converter implemented or even contribute to the
project. If the model is a custom model, a new converter must
be implemented. Examples can be found in the gallery.
How to resolve this? Also how to do prediction after converting to ONNX format?
I am trying to use Sentiment Classification with LSTM and pre-trained BERT embeddings, and later language translation with Transformer
first of all I downloaded
!pip install ktrain
!pip install tensorflow_text
And I imported the necessary lib
import pathlib
import random
import numpy as np
from typing import Tuple, List
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
from sklearn.model_selection import train_test_split
# tensoflow imports
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import (
TextVectorization, LSTM, Dense, Embedding, Dropout,
Layer, Input, MultiHeadAttention, LayerNormalization)
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.initializers import Constant
from tensorflow.keras import backend as K
import tensorflow_text as tf_text
import ktrain
from ktrain import text
And I downloaded and extracted Large Movie dataset from Stanford
!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
!tar -xzf aclImdb_v1.tar.gz
1- I try to use LSTM with train by Creating the training and test sets with the texts_from_folder function of the ktrain.text module
%reload_ext autoreload
%autoreload 2
%matplotlib inline
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID";
os.environ["CUDA_VISIBLE_DEVICES"]="0";
DATADIR ='/content/aclImdb'
trn, val, preproc = text.texts_from_folder(DATADIR,max_features=20000, maxlen=400, ngram_range=1, preprocess_mode='standard', train_test_names=['train', 'test'],classes=['pos', 'neg'])
And I am trying to build LSTM model her
K.clear_session()
def build_LSTM_model(
embedding_size: int,
total_words: int,
lstm_hidden_size: int,
dropout_rate: float) -> Sequential:
model.add(Embedding(input_dim = total_words,output_dim=embedding_size,input_length=total_words))
model.add(LSTM(lstm_hidden_size,return_sequences=True,name="lstm_layer"))
model.add(GlobalMaxPool1D())
# model.add(Dense(total_words, activation='softmax'))
model.add(Dropout(dropout_rate))
model.add(Dense(MAX_SEQUENCE_LEN, activation="relu"))
# adam = Adam(lr=0.01)
model.compile(loss='CategoricalCrossentropy', optimizer=Adam(lr=0.01), metrics=['CategoricalAccuracy'])
model.summary()
model = Sequential()
with the following requirements for a sequential model The model should include:
One Embedding layer at the beginning. (Watch out for proper parameterization!)
At least one LSTM layer.At least one Dropout layer for regularization.One final Dense layer mapping to the outputs.
compile model, with categorical_crossentropy loss and the adam optimizer. or might want to add other types of metrics for example CategoricalAccuracy makes sense here.
And then I want to use the ktrain library's get_learner method to create an easily trainable version of the previous model. and to use test set as the val_data, to see the performance. ( not include the proper train-validation-test split, but it could be extended if required.)
I am using the learner's lr_find and lr_plot methods to determine the most effective learning rate for the model. by Specifying the max_epochs parameter of lr_find to limit the time this takes. a couple of epochs! to determine the best learning rate based on the plot. and find balance between the fastest convergence and stability
learner: ktrain.Learner
model = text.text_classifier('bert', trn , preproc=preproc)
learner.lr_find()
learner.lr_plot()
learner.fit_onecycle(1e-4, 1)
I faced following errors
ValueError Traceback (most recent call last)
in ()
6 # workers=8, use_multiprocessing=False, batch_size=64)
7
----> 8 model = text.text_classifier('bert', trn , preproc=preproc)
10 # learner.lr_find()
1 frames
/usr/local/lib/python3.7/dist-packages/ktrain/text/models.py in _text_model(name, train_data, preproc, multilabel, classification, metrics, verbose)
109 raise ValueError(
110 "if '%s' is selected model, then preprocess_mode='%s' should be used and vice versa"
--> 111 % (BERT, BERT)
112 ) 113 is_huggingface = U.is_huggingface(data=train_data)
ValueError: if 'bert' is selected model, then preprocess_mode='bert' should be used and vice versa
And next step to make it with LSTM with pretrained static word vectors
If you're using BERT for pretrained word vectors supplied as features to an LSTM, then you don't need to build a separate BERT classification model. You can use TransformerEmbedding to generate word vectors for your dataset (or use sentence-transformers):
In [1]: from ktrain.text import TransformerEmbedding
In [2]: te = TransformerEmbedding('bert-base-cased')
In [3]: te.embed('George Washington went to Washington .').shape
Out[3]: (1, 6, 768)
This is what the included NER models in ktrain do under-the-hood.
Also, the input feature format for a BERT model is completely different than input features for an LSTM. As the error message indicates, to preprocess your texts for BERT classification model, you'll need to supply preprocess_mode='bert' to texts_from_folder.
Newbie question. I've been trying to convert this PyTorch model into CoreML model. I've followed the guide here but couldn't make it work. I tried tracing and scripting but faced errors which hint that there might be an operation not supported in TorchScript:
Error on torch.jit.trace: RuntimeError: PyTorch convert function for op 'pythonop' not implemented
Error on torch.jit.script: RuntimeError: Python builtin <built-in method apply of FunctionMeta object at 0x7fa37e2ad600> is currently not supported in Torchscript
I suspect that it just might not be possible to convert any PyTorch model into CoreML one. Is this the case? Can I somehow overcome the errors without diving deep into PyTorch operations and layers?
My python script just in case (model is loaded locally):
import warnings
import torch
import torch.nn as nn
import coremltools as ct
from efficientnet_pytorch import EfficientNet
from torchvision import datasets, models, transforms
from PIL import Image
# Simple loading the model
# model = torch.load('food308_efnetb2_91.31.pth', map_location=torch.device('cpu'))
# ends up with RuntimeError("Could not get name of python class object")
# Load the model
model = EfficientNet.from_pretrained('efficientnet-b2')
num_ftrs = model._fc.in_features
model._fc = nn.Linear(num_ftrs, 308)
prev_state = torch.load('food308_efnetb2_91.31.pth', map_location=torch.device('cpu'))
model.load_state_dict(prev_state)
model.eval()
# Model tracing
example_input = torch.rand(1, 3, 224, 224)
traced_model = torch.jit.trace(model, example_input)
mlmodel = ct.convert(
traced_model,
inputs=[ct.TensorType(name="input", shape=(1, 3, 64, 64))],
)
# Model scripting
scripted_model = torch.jit.script(model)
mlmodel2 = ct.convert(
scripted_model,
inputs=[ct.TensorType(name="input", shape=(1, 3, 64, 64))],
)
I am trying to add layers to a Sequential model to train it with the fashion_mnist data. However, I am receiving this error repeatedly.
#import statements
import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
from tensorflow.keras import models
#Creating a sequential model
model=models.Sequential([tf.keras.layers.Flatten(),tf.keras.layers.Dense(128,activation=tf.nn.relu),tf.keras.layers.Dense(10,activation=tf.nn.softmax)])
#Compiling the model with optimizer and loss function
model.compile(optimizer='tf.train.AdamOptimizer',loss='sparse_categorical_crossentropy')
The error:
TypeError Traceback (most recent call last)
<ipython-input-4-ffa2750d675a> in <module>()
1 #creating a sequential model
----> 2 model=models.Sequential([tf.keras.layers.Flatten(),tf.keras.layers.Dense(128,activation=tf.nn.relu),tf.keras.layers.Dense(10,activation=tf.nn.softmax)])
c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\keras\models.py in __init__(self, layers, name)
439 if layers:
440 for layer in layers:
--> 441 self.add(layer)
442
443 def add(self, layer):
c:\users\admin\appdata\local\programs\python\python36\lib\site-packages\keras\models.py in add(self, layer)
458 raise TypeError('The added layer must be '
459 'an instance of class Layer. '
--> 460 'Found: ' + str(layer))
461 if not self.outputs:
462 # First layer in model: check that it is an input layer.
TypeError: The added layer must be an instance of class Layer. Found: <tensorflow.python.keras.layers.core.Flatten object at 0x00000254C648FE48>
Can anyone help me out with this please?
Please show us the import statements at the top. If you used
from keras import models
then this error is expected as you are adding tf.keras layers to keras model.
Unfortunately, keras and tf.keras are not compatible.
If you change the import to:
from tensorflow.keras import models
then your code should work.
I found the similar error in my code:
import tensorflow as tf
from keras import Sequential, Embedding
model = Sequential(name='embedding')
model.add(Embedding(2, 2, input_length=7))
model.compile('rmsprop', 'mse')
model.predict(np.array([[0, 1, 0, 1, 1, 0, 0]]))
Then I found this article about tensorflow 2.0 and keras.
In this article, the author said:
Now that TensorFlow 2.0 is released both keras and tf.keras are in sync,
implying that keras and tf.keras are still separate projects; however,
developers should start using tf.keras moving forward as the keras package
will only support bug fixes.
Then I changed my code and everything was okay
import tensorflow as tf
import tensorflow.keras as k
from tensorflow.keras.layers import Embedding
import numpy as np
model: k.Sequential = k.Sequential(name='embedding')
em: Embedding = Embedding(2, 2, input_length=7)
model.add(em)
model.compile('rmsprop', 'mse')
model.predict(np.array([[0, 1, 0, 1, 1, 0, 0]]))
print("model.weights: ", model.weights)