I'm trying to find an example of using the C-API (not C++) asynchronous batch interface for OpenVino image inference.
I'm able to do inference on a single image at a time no problem, but its not clear to me how to expand this to batch inference with the C-API.
Does anyone have an example or reference for doing so? The OpenVino documentation is limited on this front, nor do they provide any C-based examples for doing so that I've been able to find.
EDIT: Per comment below, clarifying that the challenge is in understanding how to load up the input blobs with multiple images. The existing examples either assume the C++ interface and use vectors to move things around, or are unclear in what is idiomatic to the example (e.g. the object detection sample using the C-API)
You can use the following functions: ie_infer_request_infer_async() and ie_infer_request_set_batch().
Relevant information is available at the following link:
https://docs.openvinotoolkit.org/2020.4/ie_c_api/group__InferRequest.html
I suggest you refer to the following two C API based samples:
Hello Classification C Sample
https://docs.openvinotoolkit.org/2020.4/openvino_inference_engine_ie_bridges_c_samples_hello_classification_README.html
Hello NV12 Input Classification C Sample
https://docs.openvinotoolkit.org/2020.4/openvino_inference_engine_ie_bridges_c_samples_hello_nv12_input_classification_README.html
High-level description of the process of integrating the Inference Engine into your application is available at the following page:
https://docs.openvinotoolkit.org/2020.4/openvino_docs_IE_DG_Integrate_with_customer_application_new_API.html
Related
Torchscript provides torch.jit.trace and torch.jit.script to convert pytorch code from eager mode to script model. From the documentation, I can understand torch.jit.trace cannot handle control flows and other data structures present in the python. Hence torch.jit.script was developed to overcome the problems in torch.jit.trace.
But it looks like torch.jit.script works for all the cases, then why do we need torch.jit.trace?
Please help me understand the difference between these two methods
If torch.jit.script works for your code, then that's all you should need. Code that uses dynamic behavior such as polymorphism isn't supported by the compiler torch.jit.script uses, so for cases like that, you would need to use torch.jit.trace.
I am trying to do research on batch normalization, and had to make some modifications for the pytorch BN code. I dig into the pytorch code and got stuck with torch.nn.functional.batch_norm, which references torch.batch_norm.
The problem is that torch.batch_norm cannot be further found in the torch library. Is there any way I can find the source code of this built-in function and re-implement it? Thanks!
It's there, but it's not defined in Python. They're defined in C++ in the aten/ directories.
For CPU, the implementation (one of them, it depends on whether or not the input is contiguous) is here: https://github.com/pytorch/pytorch/blob/420b37f3c67950ed93cd8aa7a12e673fcfc5567b/aten/src/ATen/native/Normalization.cpp#L61-L126
For CUDA, the implementation is here: https://github.com/pytorch/pytorch/blob/7aae51cdedcbf0df5a7a8bf50a947237ac4b3ee8/aten/src/ATen/native/cudnn/BatchNorm.cpp#L52-L143
I am working on a project that requires inversion of multiple large dense matrix. I would like to know if ojAlgo supports multi-threading.
I do notice the github page for ojAlgo is tagged with multi-threading, however I am unable to find any documentation about it.
Thanks
I wanted to see how the conv1d module is implemented
https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv1d. So I looked at functional.py but still couldn’t find the looping and cross-correlation computation.
Then I searched Github by keyword ‘conv1d’, checked conv.cpp https://github.com/pytorch/pytorch/blob/eb5d28ecefb9d78d4fff5fac099e70e5eb3fbe2e/torch/csrc/api/src/nn/modules/conv.cpp 1 but still couldn’t locate where the computation is happening.
My question is two-fold.
Where is the source code that "conv1d” is implemented?
In general, if I want to check how the modules are implemented, where is the best place to find? Any pointer to the documentation will be appreciated. Thank you.
It depends on the backend (GPU, CPU, distributed etc) but in the most interesting case of GPU it's pulled from cuDNN which is released in binary format and thus you can't inspect its source code. It's a similar story for CPU MKLDNN. I am not aware of any place where PyTorch would "handroll" it's own convolution kernels, but I may be wrong. EDIT: indeed, I was wrong as pointed out in an answer below.
It's difficult without knowing how PyTorch is structured. A lot of code is actually being autogenerated based on various markup files, as explained here. Figuring this out requires a lot of jumping around. For instance, the conv.cpp file you're linking uses torch::conv1d, which is defined here and uses at::convolution which in turn uses at::_convolution, which dispatches to multiple variants, for instance at::cudnn_convolution. at::cudnn_convolution is, I believe, created here via a markup file and just plugs in directly to cuDNN implementation (though I cannot pinpoint the exact point in code when that happens).
Below is an answer that I got from pytorch discussion board:
I believe the “handroll”-ed convolution is defined here: https://github.com/pytorch/pytorch/blob/master/aten/src/THNN/generic/SpatialConvolutionMM.c 3
The NN module implementations are here: https://github.com/pytorch/pytorch/tree/master/aten/src
The GPU version is in THCUNN and the CPU version in THNN
I have an Alloy model. The model is of some decision making logic in software I wrote. In that model I have a few predicates that create examples. The predicate creates instances that are expected behavior and outside expected behavior. I would love to take those examples as inputs to a unit test for my code.
Does anyone have an example of some software that interacts with Alloy to dump many examples generated to a single file? I would love to run a program, get a file with many instances in it, and then use that file as input to my test program.
This interests me because the examples and counter examples created are often not what I would think to do when hand writing my test inputs.
Thoughts?
You can export an instance in the File/Export To menu.
If you can work in Java, then maybe it is interesting to know we're setting up an open source repo on github: https://github.com/AlloyTools/
I think it is quite easy to link your code with this code and generate your test cases or provide them from proper files.
I am extremely interested in this kind of applications for Alloy so please keep us posted on https://groups.google.com/forum/#!forum/alloytools