In Python programming, which using Pytorch i got this error
4 and >5 are printing the type of x to check the type of tensor, but returning different results
why does this happen?My Code of Execution
x = x.type(torch.int64)
print(x.type())
type(X)
This is a known behavior of PyTorch since version 0.4, though the maintainers don't offer a justification.
Note also that the type() of a Tensor no longer reflects the data type. Use isinstance() or x.type() instead
x = torch.DoubleTensor([1, 1, 1])
print(type(x)) # was torch.DoubleTensor
# "<class 'torch.Tensor'>"
print(x.type()) # OK: 'torch.DoubleTensor'
# 'torch.DoubleTensor'
print(isinstance(x, torch.DoubleTensor)) # OK: True
# True
Related
I am new to pytorch, and I have been trying some examples with autograd, to see if I understand it. I am confused about why the following code does not work:
def Loss(a):
return a**2
a=torch.tensor(3.0, requires_grad=True )
L=Loss(a)
L.backward()
with torch.no_grad(): a=a+1.0
L=Loss(a)
L.backward()
print(a.grad)
Instead of outputing 8.0, we get "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn"
There are two things to note regarding your code:
You are performing two back propagation up to leaf a which means the gradients should accumulate. In other words, you should get a gradient equal to da²/da + d(a+1)²/da which is equal to 2a + 2(a+1) which is 2(2a + 1). If a=3, then a.grad will be equal to 14.
You are using a torch.no_grad context manager which means you will be unable to perform backpropagation from any resulting tensor i.e. here a itself.
Here is a snippet which yields the desired result, that is 14 as the accumulation of both gradients:
>>> L = Loss(a)
>>> L.backward()
>>> a.grad
6
>>> L = Loss(a+1)
>>> L.backward()
>>> a.grad
14 # as 6 + 8
Consider the following code,
import numpy as np
xx = np.asarray([1,0,1])
def ff(x):
return np.sin(x)/x
# this throws an error because of division by zero
# C:\Users\User\AppData\Local\Temp/ipykernel_2272/525615690.py:4:
# RuntimeWarning: invalid value encountered in true_divide
# return np.sin(x)/x
yy = ff(xx)
# to avoid the error, I did the following
def ff_smart(x):
if (x==0):
# because sin(x)/x = 1 as x->0
return 1
else:
return np.sin(x)/x
# but then I cannot do
# yy_smart = ff_smart(xx)
# because of ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
# I therefore have to do:
*yy_smart, = map(ff_smart,xx)
yy_smart = np.asarray(yy_smart)
Is there a way (some numpy magic) to write ff_smart such that I can call it without using map and ff_smart remains operable on scalars (non numpy arrays). I'd like to avoid type-checking in ff_smart.
you can do:
yy = [np.sin(x)/x if x != 0 else 1 for x in xx]
If you want to use the power of numpy, a different answer, still useful to know is to use masked arrays:
# initialize x
x = np.array([2, 3, 1, 0, 2])
# compute the masked array of x, masking out 0s
masked_x = np.ma.array(x, mask= x == 0, dtype=x.dtype)
# perform operation only on non-zero values
y = np.sin(masked_x) / masked_x
# get the value back, filling the masked out values with 1s.
y = np.ma.filled(y, fill_value=1)
For conditional operations as you describe numpy has the numpy where function.
You can do
np.where(x==0, 1, np.sin(x)/x)
I am new to Numba and I need to use Numba to speed up some Pytorch functions. But I find even a very simple function does not work :(
import torch
import numba
#numba.njit()
def vec_add_odd_pos(a, b):
res = 0.
for pos in range(len(a)):
if pos % 2 == 0:
res += a[pos] + b[pos]
return res
x = torch.tensor([3, 4, 5.])
y = torch.tensor([-2, 0, 1.])
z = vec_add_odd_pos(x, y)
But the following error appears
def vec_add_odd_pos(a, b):
res = 0.
^
This error may have been caused by the following argument(s):
argument 0: cannot determine Numba type of <class 'torch.Tensor'>
argument 1: cannot determine Numba type of <class 'torch.Tensor'>
Can anyone help me? A link with more examples would be also appreciated. Thanks.
Pytorch now exposes an interface on GPU tensors which can be consumed by numba directly:
numba.cuda.as_cuda_array(tensor)
The test script provides a few usage examples: https://github.com/pytorch/pytorch/blob/master/test/test_numba_integration.py
As others have mentioned, numba currently doesn't support torch tensors, only numpy tensors. However there is TorchScript, which has a similar goal. Your function can then be rewritten as such:
import torch
#torch.jit.script
def vec_add_odd_pos(a, b):
res = 0.
for pos in range(len(a)):
if pos % 2 == 0:
res += a[pos] + b[pos]
return res
x = torch.tensor([3, 4, 5.])
y = torch.tensor([-2, 0, 1.])
z = vec_add_odd_pos(x, y)
Beware: although you said your code snippet was just a simple example, for loops are really slow and running TorchScript might not help you much, you should avoid them at any cost and only use then when no other solution exist. That being said, here's how to implement your function in a more performant way:
def vec_add_odd_pos(a, b):
evenids = torch.arange(len(a)) % 2 == 0
return (a[evenids] + b[evenids]).sum()
numba supports numpy-arrays but not torch's tensors. There is however a bridge Tensor.numpy():
Returns self tensor as a NumPy ndarray. This tensor and the returned
ndarray share the same underlying storage. Changes to self tensor will
be reflected in the ndarray and vice versa.
That means you have to call jitted functions as:
...
z = vec_add_odd_pos(x.numpy(), y.numpy())
If z should be a torch.Tensor as well, torch.from_numpy is what we need:
Creates a Tensor from a numpy.ndarray.
The returned tensor and ndarray share the same memory. Modifications
to the tensor will be reflected in the ndarray and vice versa. The
returned tensor is not resizable.
...
For our code that means
...
z = torch.from_numpy(vec_add_odd_pos(x.numpy(), y.numpy()))
should be called.
I'm trying to convert input_image which is a tensor to numpy array.Following the already answered questions here and several others that suggested to use input_image.eval() or equivalently sess.run() for this conversion, I did the same, but it throws an error and apparently expects a feed_dict value for the sess.run(). But since here I'm not trying to run an operation dependent on unknown values, I don't see the need for the feed_dict here because all I'm doing here is just conversion.
Besides, just so as to check I also tried converting a tf.constant([1,2,3]) value right above it using the same method and it got successfully compiled despite its data type being the same as input_image. Here's my code which is the part of larger script:
def call(self, x):
input_image = Input(shape=(None, None, 3))
print(input_image.shape)
print(type(tf.constant([1,2,3])))
print(type(input_image))
print(type(K.get_session().run(tf.constant([1,2,3]))))
print(type(K.get_session().run(input_image)))
and here's the error:
(?, ?, ?, 3)
<class 'tensorflow.python.framework.ops.Tensor'>
<class 'tensorflow.python.framework.ops.Tensor'>
<class 'numpy.ndarray'>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,?,?,3]
[[{{node input_1}}]]
[[input_1/_1051]]
(1) Invalid argument: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,?,?,3]
[[{{node input_1}}]]
0 successful operations.
0 derived errors ignored.
I wonder why the former would work and the latter won't.
There is no such thing as "converting" a symbolic tensor to a numpy array, as the latter cannot hold the same kind of information as the former.
When you use eval() or session.run(), what you are doing is evaluating a symbolic expression to get a numerical result, which is a numpy array, but this is not a conversion. Evaluating an expression might or might not require additional input data (that's what the feed_dict is for), depending on the expression.
Evaluating a constant (tf.constant) does not require any input data, but evaluating your other expression does require the input data, so you cannot "convert" this to a numpy array.
Just adding to (or elaborating on) what #MatiasValdenegro said,
TensorFlow follows something called graph execution (or define-then-run). In other words, when you write a TensorFlow program it defines something called a data-flow graph which shows how the operations you defined are related to each other. And then you execute bits and pieces of that graph depending on the results you're after.
Let's consider two examples. (I am switching to a simple TensorFlow program instead of Keras bits as it makes things more clear - After all K.get_session() returns a Session object).
Example 1
Say you have the following program.
import tensorflow as tf
a = tf.placeholder(shape=[2,2], dtype=tf.float32)
b = tf.constant(1, dtype=tf.float32)
c = a * b
# Wrong: This is what you're doing essentially when you do sess.run(input_image)
with tf.Session() as sess:
print(sess.run(c))
# Right: You need to feed values that c is dependent on
with tf.Session() as sess:
print(sess.run(c, feed_dict={a: np.array([[1,2],[2,3]])}))
Whenever a resulting tensor (e.g. c) is dependent on a placeholder you cannot execute it and get the result without feeding values to all the dependent placeholders.
Example 2
When you define a tf.constant(1) this is not dependent on anything. In other words you don't need a feed_dict and can directly run eval() or sess.run() on it.
Update: Further explanation on why you need a feed_dict for input_image
TLDR: You need a feed_dict because your resulting Tensor is produced by an Input layer.
Your input_image is basically the resulting tensor you get by feeding something to the Input layer. Usually in Keras, you are not exposed to the internal placeholder level details. But you would do that via using model.fit() or model.evaluate(). You can see that Keras Input layer in fact uses a placeholder by analysing this line.
Hope I made my point clear that you do need to feed in a value to the placeholder to successfully evaluate the output of an Input layer. Because that basically holds a placeholder.
Update 2: How to feed to your Input layer
So, appears you can use feed_dict with Keras Input layer in the following manner. Instead of defining shape argument you straight away pass a placeholder to the tensor argument, which will bypass the internal placeholder creation in the layer.
from tensorflow.keras.layers import InputLayer
import numpy as np
import tensorflow.keras.backend as K
x = tf.placeholder(shape=[None, None, None, 3], dtype=tf.float32)
input_image = Input(tensor=x)
arr = np.array([[[[1,1,1]]]])
print(arr.shape)
print(K.get_session().run(input_image, feed_dict={x: arr}))
I am training a neural network and a part of my code has returned the following error:
def plot_confusion_matrix(truth,
predictions,
classes,
normalize=False,
save=False,
cmap=plt.cm.Oranges,
path="confusion_matrix.png"):
acc = (np.array(truth) == np.array(predictions))
size = float(acc.shape[0]) #error
acc = np.sum(acc.astype("int32")) / size
(...)
AttributeError: 'bool' object has no attribute 'shape'
function call
pred = pred.numpy()
plot_confusion_matrix(truth=labels.numpy(),
predictions=pred,
save=False,
path="logref_confusion_matrix.png",
classes=["forward", "left", "right"])
Where the thuth represents the labels of Y and predictions the array of prediction, both with shape 32, 3. I checked the update on numpy, ipython etc and all are updated, tried some modification, but without success.
The only reason that acc would be a boolean and not a numpy array of booleans is that you are passing in a singular value for truth and predictions. In the code you provided, there would be no error for an actual array of 32x3. Look at the rest of your code and make sure you actually pass in an array to np.array() instead of singular values.