It seems that Pytorch tensor does not support unsigned int32 and int64 currently. While Numpy and TensorFlow both support them. I am wondering whether Pytorch has any plan about it? If not, any short cut i could do to make extension from Numpy? Or write C++ code is necessary?
Related
I am using the OpenVINO model optimizer framework to convert an ONNX model containing a single ConvInteger operation to OpenVINO IR format.
mo --input_model {onnx_model}
The ONNX ConvInteger operator has input and weight tensors with INT8/UINT8 precision, and an output tensor with INT32 precision - this output precision is the only supported precision.
When the model is converted to OpenVINO, the input and weight tensors are converted to INT32 precision automatically, and convert operators are added to the model to make this change in precision.
Is it possible to force the int8/uint8 precision for the openvino model? Alternatively, is there a simple way to convert the precisions to int8/uint8 once the openvino model has been created?
Thanks
You can convert the FP32 or FP16 precision into INT8 without model retraining or fine-tuning by using OpenVINO Post-training Optimization Tool (POT). This tool supports the uniform integer quantization method.
There are two main quantization methods:
Default Quantization: a recommended method that provides fast and accurate results in most cases. It requires only an unannotated dataset for quantization.
Accuracy-aware Quantization: an advanced method that allows keeping accuracy at a predefined range at the cost of performance improvement in case when Default Quantization cannot guarantee it. The method requires annotated representative dataset and may require more time for quantization.
Is it possible to create an ONNX convolution operator which has int8/uint8 precision for the input, weight and output tensors?
The ONNX Conv only supports fp precisions and ConvInteger requires an int32 output tensor. Is there simple way of converting one of these operators so that all tensors have int8/uint8 precision?
I have a tensor and want to convert in into a Quantized Binary form.
x= torch.Tensor([-9.0387e-01, 1.4811e-01, 2.8242e-01, 3.6679e-01, 3.2012e-01])
PyTorch only supports qint8 type.
You can convert the tensor to a quantized version with torch.quantize_per_tensor, you can check the wiki here.
I am trying to do a text classification using pytorch and torchtext on paperspace.
I get
RuntimeError: ‘lengths’ argument should be a 1D CPU int64 tensor, but got 1D cuda:0 Long tensor
My PyTorch version is 1.10.1+cu102
I just had this problem yesterday, in my case the rnn pad sequences wants length to be on the cpu, so just put the lengths to CPU in your function call like this:
packed_sequences = nn.utils.rnn.pack_padded_sequence(padded_tensor, valid_frames.to('cpu'), batch_first=True, enforce_sorted=True)
This might not be the exact function you're using but I think it will apply to most of the rnn utils functions.
I am trying to run my PyTorch model for ASR on an arm based device without gpu. As far as I know, arm does not support MKL which ATen uses. Naturally, I am getting the following error when I try to make inference:
RuntimeError: fft: ATen not compiled with MKL support
How can I solve this problem? Are there any alternatives that I can use?
If your target device is mobile, it's reasonable to try converting it to TorchScript with Pytorch Mobile first. TorchScript is an intermediate representation of a PyTorch model that can then be run in mobile environment.
https://pytorch.org/mobile/home/
I solved this issue by bypassing PyTorch's stft implementation. This may not be feasible for everyone, but in my case it allowed me to make predictions using my model with no issues on arm device.
The problem stemmed from _VF.stft call in packages/torch/functional.py.
I changed the line
return _VF.stft(input, n_fft, hop_length, win_length, window, normalized, onesided, return_complex)
with:
librosa_stft = librosa.stft(input.cpu().detach().numpy().reshape(-1), n_fft, hop_length, win_length, window="hann", center=True, pad_mode=pad_mode)
librosa_stft = np.array([[a.real, a.imag] for a in librosa_stft])
librosa_stft = np.transpose(librosa_stft, axes=[0, 2, 1])
librosa_stft = np.expand_dims(librosa_stft, 0)
librosa_stft = torch.from_numpy(librosa_stft)
return librosa_stft
This code may be optimized further. I just tried to replicate what PyTorch did by using Librosa. Resulting output is same in both versions in my case. But you should check your outputs to be sure if you decide to use this method.