I'm trying to train a quantize model in pytorch and convert it to ONNX.
I employ the quantized-aware-training technique with help of pytorch_quantization package.
I used the below code to convert my model to ONNX:
from pytorch_quantization import nn as quant_nn
from pytorch_quantization import calib
from pytorch_quantization.tensor_quant import QuantDescriptor
from pytorch_quantization import quant_modules
import onnxruntime
import torch
import torch.utils.data
from torch import nn
import torchvision
def export_onnx(model, onnx_filename, batch_onnx, per_channel_quantization):
model.eval()
quant_nn.TensorQuantizer.use_fb_fake_quant = True # We have to shift to pytorch's fake quant ops before exporting the model to ONNX
if per_channel_quantization:
opset_version = 13
else:
opset_version = 12
# Export ONNX for multiple batch sizes
print("Creating ONNX file: " + onnx_filename)
dummy_input = torch.randn(batch_onnx, 3, 224, 224, device='cuda') #TODO: switch input dims by model
input_names = ['input']
output_names = ['Linear[fc]'] ### ResNet34
dynamic_axes = {'input': {0: 'batch_size'}}
try:
torch.onnx.export(model, dummy_input, onnx_filename, input_names=input_names,
export_params=True, output_names=output_names, opset_version=opset_version,
verbose=True, enable_onnx_checker=False, do_constant_folding=True)
except ValueError:
warnings.warn(UserWarning("Per-channel quantization is not yet supported in Pytorch/ONNX RT (requires ONNX opset 13)"))
print("Failed to export to ONNX")
return False
return True
After conversion, I get the following warnings:
warnings.warn("'enable_onnx_checker' is deprecated and ignored. It will be removed in "
W0305 12:39:40.472136 140018114328384 tensor_quantizer.py:280] Use Pytorch's native experimental fake quantization.
/usr/local/lib/python3.8/dist-packages/pytorch_quantization/nn/modules/tensor_quantizer.py:285: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
Also, the accuracy is not valid for ONNX model!
Accuracy summary:
+-----------+-------+
| Stage | Top1 |
+-----------+-------+
| Finetuned | 38.03 |
| ONNX | -1.00 |
+-----------+-------+
More info is here:
pytorch 1.10.2+cu102
torchvision 0.11.3+cu102
TensorRT 8.2.3-1+cuda11.4
ONNX 1.11.0
ONNX Runtime 1.10.0
cuda 11.6
python 3.8
What is the problem with ONNX conversion?
After some tries, I found that there is a version conflict. I changed the versions accordingly:
onnx == 1.9.0
onnxruntime == 1.8.1
pytorch == 1.9.0+cu111
torchvision == 0.10.0+cu111
Newbie question. I've been trying to convert this PyTorch model into CoreML model. I've followed the guide here but couldn't make it work. I tried tracing and scripting but faced errors which hint that there might be an operation not supported in TorchScript:
Error on torch.jit.trace: RuntimeError: PyTorch convert function for op 'pythonop' not implemented
Error on torch.jit.script: RuntimeError: Python builtin <built-in method apply of FunctionMeta object at 0x7fa37e2ad600> is currently not supported in Torchscript
I suspect that it just might not be possible to convert any PyTorch model into CoreML one. Is this the case? Can I somehow overcome the errors without diving deep into PyTorch operations and layers?
My python script just in case (model is loaded locally):
import warnings
import torch
import torch.nn as nn
import coremltools as ct
from efficientnet_pytorch import EfficientNet
from torchvision import datasets, models, transforms
from PIL import Image
# Simple loading the model
# model = torch.load('food308_efnetb2_91.31.pth', map_location=torch.device('cpu'))
# ends up with RuntimeError("Could not get name of python class object")
# Load the model
model = EfficientNet.from_pretrained('efficientnet-b2')
num_ftrs = model._fc.in_features
model._fc = nn.Linear(num_ftrs, 308)
prev_state = torch.load('food308_efnetb2_91.31.pth', map_location=torch.device('cpu'))
model.load_state_dict(prev_state)
model.eval()
# Model tracing
example_input = torch.rand(1, 3, 224, 224)
traced_model = torch.jit.trace(model, example_input)
mlmodel = ct.convert(
traced_model,
inputs=[ct.TensorType(name="input", shape=(1, 3, 64, 64))],
)
# Model scripting
scripted_model = torch.jit.script(model)
mlmodel2 = ct.convert(
scripted_model,
inputs=[ct.TensorType(name="input", shape=(1, 3, 64, 64))],
)
Doing basic things with the Keras Functional API seems to produce errors. For example, the following fails:
from keras.layers import InputLayer, Embedding
input = InputLayer(name="input", input_shape=(1, ))
embedding = Embedding(10000, 64)(input)
This produces the error:
AttributeError: 'str' object has no attribute 'base_dtype'
I can then "cheat" by using the input_length argument but this then fails when I try to concatenate two such embeddings:
from keras.layers import InputLayer, Embedding, Concatenate
embedding1 = Embedding(10000, 64, input_length=1)
embedding2 = Embedding(10000, 64, input_length=1)
concat = Concatenate()([embedding1 , embedding2])
This gives the error:
TypeError: 'NoneType' object is not subscriptable
Same error when I use "concatenate" (lower case) instead (some sources seem to say that this should be used instead if using the functional API).
What am I doing wrong?
I am on tensorflow version 2.3.1, keras version 2.4.3, python version 3.6.7
I strongly suggest to use tf.keras and not keras.
It doesn't work because InputLayer is an instance of keras.Layer, whereas keras.layers.Input is an instance of Tensor. The argument to layer.__call__() should be Tensor and not keras.Layer.
import tensorflow as tf
inputs = tf.keras.layers.Input((1,))
print(type(inputs)) # <class 'tensorflow.python.framework.ops.Tensor'>
input_layer = tf.keras.layers.InputLayer(input_shape=(1,))
print(type(input_layer)) # <class 'tensorflow.python.keras.engine.input_layer.InputLayer'>
You use InputLayer with Sequential API. When you use functional API you should use tf.keras.layers.Input() instead:
import tensorflow as tf
inputs = tf.keras.layers.Input((1, ), name="input", )
embedding = tf.keras.layers.Embedding(10000, 64)(inputs)
Same with the second example:
import tensorflow as tf
inputs = tf.keras.layers.Input((1, ), name="input", )
embedding1 = tf.keras.layers.Embedding(10000, 64)(inputs)
embedding2 = tf.keras.layers.Embedding(10000, 64)(inputs)
concat = tf.keras.layers.Concatenate()([embedding1, embedding2])
I'm trying to convert the input tensor to a numpy array inside a custom keras loss function, after following the instructions here.
The above code runs on my machine with no errors. Now, I want to extract a numpy array with values from the input tensor. However, I get the following error:
"tensorflow.python.framework.errors_impl.InvalidArgumentError: You
must feed a value for placeholder tensor 'input_1' with dtype float
[[Node: input_1 = Placeholderdtype=DT_FLOAT, shape=[],
_device="/job:localhost/replica:0/task:0/cpu:0"]]"
I need to convert to a numpy array because I have other keras models that must operate on the input - I haven't shown those lines below in joint_loss, but even the code sample below doesn't run at all.
import numpy as np
from keras.models import Model, Sequential
from keras.layers import Dense, Activation, Input
import keras.backend as K
def joint_loss_wrapper(x):
def joint_loss(y_true, y_pred):
x_val = K.eval(x)
return y_true - y_pred
return joint_loss
input_tensor = Input(shape=(6,))
hidden1 = Dense(30, activation='relu')(input_tensor)
hidden2 = Dense(40, activation='sigmoid')(hidden1)
out = Dense(1, activation='sigmoid')(hidden2)
model = Model(input_tensor, out)
model.compile(loss=joint_loss_wrapper(input_tensor), optimizer='adam')
I figured it out!
What you want to do is use the Functional API for Keras.
Then your submodels outputs as tensors can be obtained as y_pred_submodel = submodel(x).
This is similar to how a Keras layer operates on a tensor.
Manipulate only tensors within the loss function. That should work fine.
I'm predicting image classes using Keras. It works in Google Cloud ML (GCML), but for efficiency need change it to pass base64 strings instead of json array. Related Documentation
I can easily run python code to decode a base64 string into json array, but when using GCML I don't have the opportunity to run a preprocessing step (unless maybe use a Lambda layer in Keras, but I don't think that is the correct approach).
Another answer suggested adding tf.placeholder with type of tf.string, which makes sense, but how to incorporate that into the Keras model?
Here is complete code for training the model and saving the exported model for GCML...
import os
import numpy as np
import tensorflow as tf
import keras
from keras import backend as K
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.preprocessing import image
from tensorflow.python.platform import gfile
IMAGE_HEIGHT = 138
IMAGE_WIDTH = 106
NUM_CLASSES = 329
def preprocess(filename):
# decode the image file starting from the filename
# end up with pixel values that are in the -1, 1 range
image_contents = tf.read_file(filename)
image = tf.image.decode_png(image_contents, channels=1)
image = tf.image.convert_image_dtype(image, dtype=tf.float32) # 0-1
image = tf.expand_dims(image, 0) # resize_bilinear needs batches
image = tf.image.resize_bilinear(image, [IMAGE_HEIGHT, IMAGE_WIDTH], align_corners=False)
image = tf.subtract(image, 0.5)
image = tf.multiply(image, 2.0) # -1 to 1
image = tf.squeeze(image,[0])
return image
filelist = gfile.ListDirectory("images")
sess = tf.Session()
with sess.as_default():
x = np.array([np.array( preprocess(os.path.join("images", filename)).eval() ) for filename in filelist])
input_shape = (IMAGE_HEIGHT, IMAGE_WIDTH, 1) # 1, because preprocessing made grayscale
# in our case the labels come from part of the filename
y = np.array([int(filename[filename.index('_')+1:-4]) for filename in filelist])
# convert class labels to numbers
y = keras.utils.to_categorical(y, NUM_CLASSES)
########## TODO: something here? ##########
image = K.placeholder(shape=(), dtype=tf.string)
decoded = tf.image.decode_jpeg(image, channels=3)
# scores = build_model(decoded)
model = Sequential()
# model.add(decoded)
model.add(Conv2D(32, kernel_size=(2, 2), activation='relu', input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
model.fit(
x,
y,
batch_size=64,
epochs=20,
verbose=1,
validation_split=0.2,
shuffle=False
)
predict_signature = tf.saved_model.signature_def_utils.build_signature_def(
inputs={'input_bytes':tf.saved_model.utils.build_tensor_info(model.input)},
########## TODO: something here? ##########
# inputs={'input': image }, # input name must have "_bytes" suffix to use base64.
outputs={'formId': tf.saved_model.utils.build_tensor_info(model.output)},
method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
)
builder = tf.saved_model.builder.SavedModelBuilder("exported_model")
builder.add_meta_graph_and_variables(
sess=K.get_session(),
tags=[tf.saved_model.tag_constants.SERVING],
signature_def_map={
tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: predict_signature
},
legacy_init_op=tf.group(tf.tables_initializer(), name='legacy_init_op')
)
builder.save()
This is related to my previous question.
Update:
The heart of the question is how to incorporate the placeholder that calls decode into the Keras model. In other words, after creating the placeholder that decodes the base64 string to a tensor, how to incorporate that into what Keras runs? I assume it needs to be a layer.
image = K.placeholder(shape=(), dtype=tf.string)
decoded = tf.image.decode_jpeg(image, channels=3)
model = Sequential()
# Something like this, but this fails because it is a tensor, not a Keras layer. Possibly this is where a Lambda layer comes in?
model.add(decoded)
model.add(Conv2D(32, kernel_size=(2, 2), activation='relu', input_shape=input_shape))
...
Update 2:
Trying to use a lambda layer to accomplish this...
import keras
from keras.models import Sequential
from keras.layers import Lambda
from keras import backend as K
import tensorflow as tf
image = K.placeholder(shape=(), dtype=tf.string)
model = Sequential()
model.add(Lambda(lambda image: tf.image.decode_jpeg(image, channels=3), input_shape=() ))
Gives the error: TypeError: Input 'contents' of 'DecodeJpeg' Op has type float32 that does not match expected type of string.
first of all I use tf.keras but this should not be a big problem.
So here is an example of how you can read a base64 decoded jpeg:
def preprocess_and_decode(img_str, new_shape=[299,299]):
img = tf.io.decode_base64(img_str)
img = tf.image.decode_jpeg(img, channels=3)
img = tf.image.resize_images(img, new_shape, method=tf.image.ResizeMethod.BILINEAR, align_corners=False)
# if you need to squeeze your input range to [0,1] or [-1,1] do it here
return img
InputLayer = Input(shape = (1,),dtype="string")
OutputLayer = Lambda(lambda img : tf.map_fn(lambda im : preprocess_and_decode(im[0]), img, dtype="float32"))(InputLayer)
base64_model = tf.keras.Model(InputLayer,OutputLayer)
The code above creates a model that takes a jpeg of any size, resizes it to 299x299 and returns as 299x299x3 tensor. This model can be exported directly to saved_model and used for Cloud ML Engine serving. It is a little bit stupid, since the only thing it does is the convertion of base64 to tensor.
If you need to redirect the output of this model to the input of an existing trained and compiled model (e.g inception_v3) you have to do the following:
base64_input = base64_model.input
final_output = inception_v3(base64_model.output)
new_model = tf.keras.Model(base64_input,final_output)
This new_model can be saved. It takes base64 jpeg and returns classes identified by the inception_v3 part.
Another answer suggested adding tf.placeholder with type of tf.string, which makes sense, but how to incorporate that into the Keras model?
In Keras you can access your selected Backend (in this case Tensorflow) by doing:
from keras import backend as K
This you already seem to import on your code. That will enable you to access some native methods and resources available on the backend of your choice. It is the case that Keras backend includes a method for creating placeholders, among other utilities. Regarding placeholders, we can see what the Keras docs indicates about them:
placeholder
keras.backend.placeholder(shape=None, ndim=None, dtype=None, sparse=False, name=None)
Instantiates a placeholder tensor and returns it.
It also gives some example on its use:
>>> from keras import backend as K
>>> input_ph = K.placeholder(shape=(2, 4, 5))
>>> input_ph._keras_shape
(2, 4, 5)
>>> input_ph
<tf.Tensor 'Placeholder_4:0' shape=(2, 4, 5) dtype=float32>
As you can see, this is returning a Tensorflow tensor, with shape (2,4,5) and of dtype float. If you had another backend while doing the example you would get another tensor object (a Theano one surely). You can therefore use this placeholder() to adapt the solution you got on your previous question.
In conclusion, you can use your backend imported as K (or whatever you want) to do calls on the methods and objects available on the backend of your choice, by doing K.foo.bar() on the desired method. I suggest you give a read to what the Keras Backend to explore more things that can be useful for you on future situations.
Update: As per your edit. Yes, this placeholder should be a layer in your model. Specifically, it should be the Input Layer of your model, as it holds your decoded image (as Keras needs it that way) to classify.