I want to do inference using openvino.
But I got an error while using openvino.
Any way to solve it?enter code here
model = keras.models.load_model('/resnet50.h5')
onnx_model, _ = tf2onnx.convert.from_keras(model, opset=16)
onnx.save(onnx_model, '/t1_model.onnx')
ie = IECore()
net = ie.read_network("/t1_model.onnx")
input_name = list(net.input_info.keys())[0]
output_name = list(net.outputs.keys())[0]
net.input_info[input_name].precision = 'FP32'
net.outputs[output_name].precision = 'FP32'
exec_net = ie.load_network(network=net, device_name='CPU')
I faced these problems.
RuntimeError: Check 'std::get<0>(valid)' failed at C:\j\workspace\private-ci\ie\build-windows-vs2019#3\b\repos\openvino\src\inference\src\ie_core.cpp:1414:
InferenceEngine::Core::LoadNetwork doesn't support inputs having dynamic shapes. Use ov::Core::compile_model API instead. Dynamic inputs are :{ input:'input_1,input_1', shape={?,256,256,3}}
input_shape = (None, 256,256,3)
The IECore API doesn't support dynamic shapes so you need to make your model static before you load it into the plugin. You can use the reshape() method on the imported model.
As an alternative you can switch to the 2022.1 version of OV where the dynamic shapes are supported. You have to switch from IECore to Core, read_network -> read_model, load_network -> compile_model.
Related
I try to quantize nn.LogSoftmax in my model. But ı don't find right configs. Any idea ?
it's dynamic quantize, ı can try static quantize if it possible.
qconfig_spec = {
nn.Embedding: float_qparams_weight_only_qconfig,
nn.Linear: default_dynamic_qconfig,
nn.LSTM: default_dynamic_qconfig,
nn.LogSoftmax: ????????????????
}
model_to_quantize = copy.deepcopy(MODEL)
quantized_model = torch.quantization.quantize_dynamic(model_to_quantize, qconfig_spec, dtype=torch.qint8)
Try to quantize LogSoftmax, my real purpose is using the quantized model on tensorrt
I'm trying to implement Double DQN (not to be confused with DQN with a slightly delayed Q-target network) in PyTorch to train an agent to play an Atari OpenAI Gym game. Here I discuss the implementation of the following formula:
Update of Q-network, formula taken from Sutton & Barto.
My first implementation is:
Q_pred = self.Q_1.forward(s_now)[T.arange(batch_size), actions.long()]
Q_next_all = self.Q_1.forward(s_next)
maxA_id = T.argmax(Q_next_all, dim=1)
Q_pred2 = self.Q_2.forward(s_next)[T.arange(batch_size), maxA_id]
Q_target = (rewards + (~dones) * self.GAMMA * Q_pred2).detach()
self.Q_1.optimizer.zero_grad()
self.Q_1.loss(Q_target, Q_pred).backward()
self.Q_1.optimizer.step()
(Q_1 and Q_2 are nn.Module classes, and all of the variables involved here are already torch tensors lying in the GPU.)
I noticed that my program ran much slower than a previous implementation which used plain DQN.
I realized that I can combine the batches entering Q_1, so there will be one combined batch being forwarded in the neural network, instead of two batches in sequence. The code becomes:
s_combined = T.cat((s_now, s_next))
Q_combined = self.Q_1.forward(s_combined)
Q_pred = Q_combined[T.arange(batch_size), actions.long()]
Q_next_all = Q_combined[batch_size:]
Q_pred2_all = self.Q_2.forward(s_next)
maxA_id = T.argmax(Q_next_all, dim=1)
Q_pred2 = Q_pred2_all[T.arange(batch_size), maxA_id]
Q_target = (rewards + (~dones) * self.GAMMA * Q_pred2).detach()
self.Q_1.optimizer.zero_grad()
self.Q_1.loss(Q_target, Q_pred).backward()
self.Q_1.optimizer.step()
(This proves that I understand how to do batch training in PyTorch, so don't mark this as a duplicate of this question.)
Furthermore, I realized that Q_1 and Q_2 can process their batches in parallel. So I looked up how to do multiprocessing in PyTorch. Unfortunately, I couldn't find a good example. I tried to adapt a code that looks similar to my scenario, and my code becomes:
def spawned():
s_combined = T.cat((s_now, s_next))
Q_combined = self.Q_1.forward(s_combined)
Q_pred = Q_combined[T.arange(batch_size), actions.long()]
Q_next_all = Q_combined[batch_size:]
mp.set_start_method('spawn', force=True)
p = mp.Process(target=spawned)
p.start()
Q_pred2_all = self.Q_2.forward(s_next)
p.join()
maxA_id = T.argmax(Q_next_all, dim=1)
Q_pred2 = Q_pred2_all[T.arange(batch_size), maxA_id]
Q_target = (rewards + (~dones) * self.GAMMA * Q_pred2).detach()
self.Q_1.optimizer.zero_grad()
self.Q_1.loss(Q_target, Q_pred).backward()
self.Q_1.optimizer.step()
This crashes with the error message:
AttributeError: Can't pickle local object 'Agent.learn.<locals>.spawned'
So how do I make this work?
(Achieving this in CUDA programming is trivial. One simply launches two device kernels using a sequential host code, and the two kernels are automatically computed in parallel in the GPU.)
I'm trying to replicate the network described in this link: https://arxiv.org/pdf/1806.07492.pdf. I've been able to replicate the patch networks of the model so far, so now I have 9 models of (32,32,3) input size each.
However, the next step is to merge those 9 models into a single one to obtain a single (96,96,3) input network. The idea is that of the 27 neurons of the first layer, the first 3 correspond to the pretrained model of the first patch, and so on. Here is an image of how it should result (each colored row is a different model):
Final concatenated model
In this image, each row before the fully-connected layer represents a previously trained model with the following characteristics:
(32,32,3) architecture
As you can see, this network analyzes images of (32,32,3). However, the complete model needs to follow this structure that uses a (96,96,3) model:
(96,96,3) architecture
I already have the (32,32,3) models in their respective hdf5 file, but I do not know how to merge them into a single model to obtain the one of size (96,96,3). I tried using the concatenate function of Keras, but I got an error that said that my inputs needed to be tensors.
Here is the code that I used:
in1 = Input(shape=(32,32,3))
model_patch1 = load_model('patch1.hdf5')
in2 = Input(shape=(32,32,3))
model_patch2 = load_model('patch2.hdf5')
in3 = Input(shape=(32,32,3))
model_patch3 = load_model('patch3.hdf5')
in4 = Input(shape=(32,32,3))
model_patch4 = load_model('patch4.hdf5')
in5 = Input(shape=(32,32,3))
model_patch5 = load_model('patch5.hdf5')
in6 = Input(shape=(32,32,3))
model_patch6 = load_model('patch6.hdf5')
in7 = Input(shape=(32,32,3))
model_patch7 = load_model('patch7.hdf5')
in8 = Input(shape=(32,32,3))
model_patch8 = load_model('patch8.hdf5')
in9 = Input(shape=(32,32,3))
model_patch9 = load_model('patch9.hdf5')
model_final_concat = Concatenate(axis=-1)([model_patch1, model_patch2,
model_patch3, model_patch4, model_patch5, model_patch6, model_patch7,
model_patch8, model_patch9])
model_final_dense_1 = Dense(1, activation='sigmoid')(model_final_concat)
lsCnn_faces = Model(inputs=[in1,in2,in3,in4,in5,in6,in7,in8,in9],
outputs=model_final_dense_1) lsCnn_faces.summary()
Any help will be very much appreciated.
I've been playing around with BERT and TensorFlow following the example here and have a trained working model.
I then wanted to save and deploy the model, so used the export_saved_model function, which requires you build a serving_input_fn to handle any incoming requests when the model is reloaded.
I wanted to be able to pass a single string for sentiment analysis to the deployed model, rather than having a theoretical client side application do the tokenisation and feature generation etc, so tried to write an input function that would handle that and pass the constructed features to the model. Is this possible? I wrote the following which I feel should do what I want:
import json
import base64
def plain_text_serving_input_fn():
input_string = tf.placeholder(dtype=tf.string, shape=None, name='input_string_text')
# What format to expect input in.
receiver_tensors = {'input_text': input_string}
input_examples = [run_classifier.InputExample(guid="", text_a = str(input_string), text_b = None, label = 0)] # here, "" is just a dummy label
input_features = run_classifier.convert_examples_to_features(input_examples, label_list, MAX_SEQ_LENGTH, tokenizer)
variables = {}
for i in input_features:
variables["input_ids"] = i.input_ids
variables["input_mask"] = i.input_mask
variables["segment_ids"] = i.segment_ids
variables["label_id"] = i.label_id
feature_spec = {
"input_ids" : tf.FixedLenFeature([MAX_SEQ_LENGTH], tf.int64),
"input_mask" : tf.FixedLenFeature([MAX_SEQ_LENGTH], tf.int64),
"segment_ids" : tf.FixedLenFeature([MAX_SEQ_LENGTH], tf.int64),
"label_ids" : tf.FixedLenFeature([], tf.int64)
}
string_variables = json.dumps(variables)
encode_input = base64.b64encode(string_variables.encode('utf-8'))
encode_string = base64.decodestring(encode_input)
features_to_input = tf.parse_example([encode_string], feature_spec)
return tf.estimator.export.ServingInputReceiver(features_to_input, receiver_tensors)
I would expect that this would allow me to call predict on my deployed model with
variables = {"input_text" : "This is some test input"}
predictor.predict(variables)
I've tried a range of variations of this (putting it in an array, converting to base 64 etc), but I get a range of errors either telling me
"error": "Failed to process element: 0 of 'instances' list. Error: Invalid argument: JSON Value: {\n \"input_text\": \"This is some test input\"\n} not formatted correctly for base64 data" }"
or
Object of type 'bytes' is not JSON serializable
I suspect I'm formatting my requests incorrectly, but I also can't find any examples of something similar being done in a serving_input_fn, so has anyone ever done something similar?
I am trying to use a QRNN based encoder for text classification by tuning a QRNN pretrained LM.
Here is the configuration of qrnn
emb_sz:int = 400
nh: int = 1550
nl: int = 3
qrnn_config = copy.deepcopy(awd_lstm_lm_config)
dps = dict(output_p=0.25, hidden_p=0.1, input_p=0.2, embed_p=0.02, weight_p=0.15)
qrnn_config.update({'emb_sz':emb_sz, 'n_hid':nh, 'n_layers':nl, 'pad_token':1, 'qrnn':True})
qrnn_config
I am passing configuration to lm_learner
lm_learner = language_model_learner(data_lm, AWD_LSTM, config=qrnn_config, pretrained=False,drop_mult=.1,pretrained_fnames=(pretrained_lm_fname,pretrained_itos_fname))
What I am getting is:
ImportError: No module named 'forget_mult_cuda'
Fast-ai version is: '1.0.51.dev0'
Try cleaning cuda cash using
gc.collect()
torch.cuda.empty_cache()
Use this for updating QRnn to true
language model
config = awd_lstm_lm_config.copy()
config['qrrn']=True
Classification model
config = awd_lstm_clas_config.copy()
config['qrrn']=True
config
You need not copy anything from source code.
It seems that you're missing ninja package.
Use:
pip install ninja
And restart your notebook, if you're using it.