The issue
I developed a simple NodeJS app for object detection using #tensorflow/tfjs-node. Everything works fine on my development PC (Windows 10 Pro), but trying to execute on my Raspberry Pi 2B (Raspbian 10), I got the following error:
Overriding the gradient for 'Max'
Overriding the gradient for 'OneHot'
Overriding the gradient for 'PadV2'
Overriding the gradient for 'SpaceToBatchND'
Overriding the gradient for 'SplitV'
2020-07-31 11:25:12.068892: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: ./assets/saved_model
2020-07-31 11:25:12.643852: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-07-31 11:25:13.206821: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: fail. Took 1137915 microseconds.
Error: Failed to load SavedModel: Op type not registered 'NonMaxSuppressionV5' in binary running on raspberrypi. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
at NodeJSKernelBackend.loadSavedModelMetaGraph (/home/pi/storage/tensorflow-test-node/node_modules/#tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1588:29)
at Object.<anonymous> (/home/pi/storage/tensorflow-test-node/node_modules/#tensorflow/tfjs-node/dist/saved_model.js:429:45)
at step (/home/pi/storage/tensorflow-test-node/node_modules/#tensorflow/tfjs-node/dist/saved_model.js:48:23)
at Object.next (/home/pi/storage/tensorflow-test-node/node_modules/#tensorflow/tfjs-node/dist/saved_model.js:29:53)
at fulfilled (/home/pi/storage/tensorflow-test-node/node_modules/#tensorflow/tfjs-node/dist/saved_model.js:20:58)
I can reproduce it with the following lines:
const tf = require('#tensorflow/tfjs-node');
// Native SavedModel: ./assets/saved_model/saved_model.pb
const objectDetectionModel = await tf.node.loadSavedModel('./assets/saved_model'); // Error
// ...
I supose that the error is related with the SavedModel version, but I don't know how convert it to use in the Rapsberry Pi or why the NodeJS app needs different SavedModel if I execute in Windows or Raspbian.
Details
Enviroment
Development:
OS: Windows 10 Pro
NodeJS: v12.16.2
NPM: 6.11.3
Target (Raspberry PI):
OS: Raspbian 10
NodeJS: v12.18.3
NPM: 6.14.6
NodeJS app
#tensorflow/tfjs-node#2.0.1 is the only dependency declared in the package.json.
Training
The model was trained on Python following this guide (TensorFlow version used was 1.15.2).
SavedModel
Details of SavedModel (command saved_model_cli show --dir saved_model --tag_set serve --signature_def serving_default executed):
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_INT32
shape: (-1, -1, -1, 3)
name: image_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 300, 4)
name: detection_boxes:0
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 300)
name: detection_classes:0
outputs['detection_multiclass_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 300, 37)
name: detection_multiclass_scores:0
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 300)
name: detection_scores:0
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (-1)
name: num_detections:0
outputs['raw_detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 300, 4)
name: raw_detection_boxes:0
outputs['raw_detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 300, 37)
name: raw_detection_scores:0
Method name is: tensorflow/serving/predict
You need to convert your model for Tensorflow Lite (with reduced ops). The error received is due to the lack of ops available on raspberry pi when loading a desktop compiled model (with higher ops available). Read more about ops here: https://www.tensorflow.org/lite/guide/ops_select
There's already is a build script that exports the model to TF Lite, similar to the one you're using (same folder in official examples repo). The functionality is the same, however the input format is slightly different. Check it out: https://www.github.com/tensorflow/models/tree/master/research%2Fobject_detection%2Fexport_tflite_ssd_graph.py
Related
After running the neural network in the browser, an error appears('The dtype of dict['input_tensor'] provided in model.execute(dict) must be int32, but was float32'), but it seems to me that the problem is not in the input_tensor, but in the neural network.
Either I trained it incorrectly, or converted it incorrectly.
I trained a pre-trained neural network 'ssd_mobilenet_v2_fpnlite_320x320_coco17' in a colobarator.
Saved the network in tensorflow format - save_model:
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_tensor'] tensor_info:
dtype: DT_UINT8
shape: (1, -1, -1, 3)
name: serving_default_input_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_anchor_indices'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:0
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 4)
name: StatefulPartitionedCall:1
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:2
outputs['detection_multiclass_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 249)
name: StatefulPartitionedCall:3
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:4
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (1)
name: StatefulPartitionedCall:5
outputs['raw_detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 130944, 4)
name: StatefulPartitionedCall:6
outputs['raw_detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 130944, 249)
name: StatefulPartitionedCall:7
Method name is: tensorflow/serving/predict
Concrete Functions:
Function Name: '__call__'
Option #1
Callable with:
Argument #1
input_tensor: TensorSpec(shape=(1, None, None, 3), dtype=tf.uint8, name='input_tensor')
I checked the quality of the model (MobileNetv2, finds one class in the image), it is wonderful!
After that, I converted, save the model to tensorflow js, with the following code
tensorflowjs_converter \
--input_format=tf_saved_model \
--output_node_names='detection_boxes','detection_classes','detection_features','detection_multiclass_scores','num_detections','raw_detection_boxes','raw_detection_scores' \
--output_format=tfjs_graph_model \
/content/gdrive/MyDrive/model_scoarbord/export/inference_graph/saved_model
/content/gdrive/MyDrive/model_scoarbord/web_model
When loading the network into the browser, I created a zero tensor to test the performance of the neural network.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs#3.12.0/dist/tf.min.js"></script>
<title>Document</title>
</head>
<body onload="">
<script>
async function loadModel() {
const modelUrl ='model.json';
const model = await tf.loadGraphModel(modelUrl);
console.log('Model loaded')
//create a zero tensor to test the model
const zeros = tf.zeros([1, -1, -1, 3]);
const zeros2 = zeros.toInt()
//checking the performance of the model
model.predict(zeros).print();
return model
}
loadModel()
</script>
</body>
</html>
Accordingly, my directory looks like this:
group1-shard1of3.bin
group1-shard2of3.bin
group1-shard3of3.bin
index.html
model.json
After starting the live server in visual code, I see the following error:
util_base.js:153 Uncaught (in promise) Error: The dtype of dict['input_tensor'] provided in model.execute(dict) must be int32, but was float32
I tried to explicitly specify the type of tensor const zeros2 = zeros.toInt()
And made a test prediction with zeros2
And got other errors:
graph_executor.js:166 Uncaught (in promise) Error: This execution contains the node 'StatefulPartitionedCall/map/while/exit/_435', which has the dynamic op 'Exit'. Please use model.executeAsync() instead. Alternatively, to avoid the dynamic ops, specify the inputs [StatefulPartitionedCall/map/TensorArrayV2Stack_1/TensorListStack]
Please tell me what am I doing wrong?
How else can you check the performance of a neural network in the tfjs_graph_model format?
I'm new to Computer Vision model structure, and I'm using Tensorflow for Node JS #tensorflow/tfjs-node to make some models detect some objects. With Mobilenet and Resnet SSD, the models are using the Channels Last format, so when I create a Tensor with tf.node.decodeImage the format is by default Channels Last, like shape: [1, 1200, 1200, 3] for 3 channels, and the predictions data work great, able to recognize objects.
But model from Pytorch, converted to ONNX, then to Protobuf PB format, the saved_model.pb has the Channels First format, like shape: [1, 3, 1200, 1200].
Now I need to create Tensor from image but with Channels First format. I found many exemple of creating conv1d, conv2d specifying the format dataFormat='channelsFirst'. But I don't know how to apply it to an image data. Here is the API https://js.tensorflow.org/api/latest/#layers.conv2d .
Here is the Tensor codes:
const tf = require('#tensorflow/tfjs-node');
let imgTensor = tf.node.decodeImage(new Uint8Array(subBuffer), 3);
imgTensor = imgTensor.cast('float32').div(255);
imgTensor = imgTensor.expandDims(0); // to add the most left axis of size 1
console.log('tensor', imgTensor);
This gives me a shape with channels last that is not compatible with the Model shape with channels first:
tensor Tensor {
kept: false,
isDisposedInternal: false,
shape: [ 1, 1200, 1200, 3 ],
dtype: 'float32',
size: 4320000,
strides: [ 4320000, 3600, 3 ],
dataId: {},
id: 7,
rankType: '4',
scopeId: 4
}
I know of tf.shape, but it reshapes without converting to channels first, and the result seems useless in predictions results. Don't know what I'm missing.
you can use something like this:
const nchw = tf.transpose(nhwc, [0, 3, 1, 2]);
I managed to get the server to work, but I can't POST the image to my network. My network is a modification of the example, and when I post it it gives the following error.
"error": "inputs is a plain value/list, but expecting an object as multiple input tensors required as per tensorinfo_map"
My cliente side is:
import requests
import json
import cv2
import numpy as np
from PIL import Image
import nsvision as nv
img = cv2.imread(r'./temp.png')
_, img_encoded = cv2.imencode('.png', img)
headers = {"content-type": "application/json"}
data = json.dumps({"signature_name": "serving_default", "inputs": [img_encoded.tolist()] })
json_response = requests.post(url="http://172.104.198.143:8501/v1/models/API_model:predict", data = data, headers = headers)
print(json_response.text)
My signature:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['image'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 200, 50, 1)
name: serving_default_image:0
inputs['label'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1)
name: serving_default_label:0
The given SavedModel SignatureDef contains the following output(s):
outputs['ctc_loss'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 50, 37)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
I'm a beginner in deep learning and python,
I tried to run keras R-FCN I google colab and i got an error like this
tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
(0) Failed precondition: Error while reading resource variable _AnonymousVar404 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar404/N10tensorflow3VarE does not exist.
[[node regr_vote/crop_to_bounding_box_7/stack/ReadVariableOp_1 (defined at usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3009) ]]
(1) Cancelled: Function was cancelled before it was started
0 successful operations.
0 derived errors ignored. [Op:__inference_keras_scratch_graph_16906]
I think function crop_to_bounding_box has a something that uninitialized
and this is the code that called crop_to_bounding_box
# position-sensitive ROI pooling + classify
score_map_bins = []
for channel_step in range(self.k*self.k):
bin_x = K.variable(int(channel_step % self.k) *
self.pool_shape, dtype='int32')
print(bin_x)
bin_y = K.variable(int(channel_step / self.k) *
self.pool_shape, dtype='int32')
channel_indices = K.variable(list(range(
channel_step*self.channel_num, (channel_step+1)*self.channel_num)), dtype='int32')
croped = tf.image.crop_to_bounding_box(
tf.gather(pooled, indices=channel_indices, axis=-1), bin_y, bin_x, self.pool_shape, self.pool_shape)
# [pool_shape, pool_shape, channel_num] ==> [1,1,channel_num] ==> [1, channel_num]
croped_mean = K.pool2d(croped, (self.pool_shape, self.pool_shape), strides=(
1, 1), padding='valid', data_format="channels_last", pool_mode='avg')
# [batch * num_rois, 1,1,channel_num] ==> [batch * num_rois, 1, channel_num]
croped_mean = K.squeeze(croped_mean, axis=1)
score_map_bins.append(croped_mean)
I'm using tensorflow-GPU v2.1.0 and Keras 2.3.1
UPDATED:
probably because I add K.variable inside loop, if I tried comment K.variable the code work perfectly, but I need to change K.variable based on channel_step.
how to update K.variable so i can defined K.variable outside loop, and update value inside the loop based on channel_step ?
I found the solution for this problem, I just need to downgrade Keras and TensorFlow version. Now I'm using tensorflow-GPU 1.15.0 and Keras 2.2.4. Apparently, this code does not support TensorFlow 2 and later.
I have fixed this by changing the load function. If you are loading your model from .h5 or .pb
change it from this:
model = tf.saved_model.load(self.filename, [tf.saved_model.SERVING])
to this:
from tensorflow.python.keras.models import load_model
model = load_model(self.filename)
This is with regards to TF 2.0.
Please find below my code that performs GridSearch along with Cross Validation using sklearn.model_selection.GridSearchCV for the mnist dataset that works perfectly fine.
# Build Function to create model, required by KerasClassifier
def create_model(optimizer_val='RMSprop',hidden_layer_size=16,activation_fn='relu',dropout_rate=0.1,regularization_fn=tf.keras.regularizers.l1(0.001),kernel_initializer_fn=tf.keras.initializers.glorot_uniform,bias_initializer_fn=tf.keras.initializers.zeros):
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(units=hidden_layer_size, activation=activation_fn,kernel_regularizer=regularization_fn,kernel_initializer=kernel_initializer_fn,bias_initializer=bias_initializer_fn),
tf.keras.layers.Dropout(dropout_rate),
tf.keras.layers.Dense(units=hidden_layer_size,activation='softmax',kernel_regularizer=regularization_fn,kernel_initializer=kernel_initializer_fn,bias_initializer=bias_initializer_fn)
])
optimizer_val_final=optimizer_val
model.compile(optimizer=optimizer_val, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
return model
#Create the model with the wrapper
model = tf.keras.wrappers.scikit_learn.KerasClassifier(build_fn=create_model, epochs=100, batch_size=10, verbose=2)
#Initialize the parameter grid
nn_param_grid = {
'epochs': [10],
'batch_size':[128],
'optimizer_val': ['Adam','SGD'],
'hidden_layer_size': [128],
'activation_fn': ['relu'],
'dropout_rate': [0.2],
'regularization_fn':['l1','l2','L1L2'],
'kernel_initializer_fn':['glorot_normal', 'glorot_uniform'],
'bias_initializer_fn':[tf.keras.initializers.zeros]
}
#Perform GridSearchCV
grid = GridSearchCV(estimator=model, param_grid=nn_param_grid, verbose=2, cv=3,scoring=precision_custom,return_train_score=False,n_jobs=-1)
grid_result = grid.fit(x_train, y_train)
My idea is to pass different optimizers with different learning rates , say Adam for learning rates 0.1,0.01 and 0.001. I also want to try out SGD with different learning rates and momentum values.
In that case , when I pass 'optimizer_val': [tf.keras.optimizers.Adam(0.1)], , I get the error as given below:
Cannot clone object <tensorflow.python.keras.wrappers.scikit_learn.KerasClassifier object at 0x7fe08b210e10>, as the constructor either does not set or modifies parameter optimizer_val
Please advise as to how can I rectify this error.
This is sklearn bug. You should reduce the version of sklearn:
conda install scikit-learn==0.21.2
It's OK!
You can fix the issue with changing the list into tuple.
If there is any single valued instance then you can use list.
#Initialize the parameter grid
nn_param_grid = {
'epochs': [10],
'batch_size':[128],
'optimizer_val': ('Adam','SGD'),
'hidden_layer_size': [128],
'activation_fn': ['relu'],
'dropout_rate': [0.2],
'regularization_fn':('l1','l2','L1L2'),
'kernel_initializer_fn':('glorot_normal', 'glorot_uniform'),
'bias_initializer_fn':[tf.keras.initializers.zeros]
}
Found this comment online and it helped!
For those who are getting following error due to above statement:
Cannot clone object <keras.wrappers.scikit_learn.KerasClassifier object at 0x7f93ddc5d1d0>, as the constructor either does not set or modifies parameter layers
Change the layers from array of list to array of tuple:
layers => [(20,), (45, 30, 15), (40, 20)]
Don't forget to add comma after
(20,) otherwise another error/warning will appear - FitFailedWarning:
Estimator fit failed. The score on this train-test partition for these
parameters will be set to nan. Details: TypeError: 'int' object is
not iterable Because single tuple without comma is treated as int.
Only installing TensorFlow 2.8 helped with this issue. Notice that it is available only via pip Anaconda TensorFlow 2.7 vs. Pypi TensorFlow 2.8
To check your version of Tensorflow type: conda list tensorflow
(base) C:\Users\User> conda list tensorflow-gpu
# Name Version Build Channel
tensorflow-gpu 2.4.1 pyhd8ed1ab_3 conda-forge
To uninstall type: conda uninstall tensorflow and to install version 2.8 type:
pip install tensorflow-gpu==2.8