Is it possible to run map_fn on a tensor with a single value?
The following works:
import tensorflow as tf
a = tf.constant(1.0, shape=[3])
tf.map_fn(lambda x: x+1, a)
#output: [2.0, 2.0, 2.0]
However this does not:
import tensorflow as tf
b = tf.constant(1.0)
tf.map_fn(lambda x: x+1, b)
#expected output: 2.0
Is it possible at all?
What am I doing wrong?
Any hints will be greatly appreciated!
Well, I see you accepted an answer, which correctly states that tf.map_fn() is applying a function to elements of a tensor, and a scalar tensor has no elements. But it's not impossible to do this for a scalar tensor, you just have to tf.reshape() it before and after, like this code (tested):
import tensorflow as tf
b = tf.constant(1.0)
if () == b.get_shape():
c = tf.reshape( tf.map_fn(lambda x: x+1, tf.reshape( b, ( 1, ) ) ), () )
else:
c = tf.map_fn(lambda x: x+1, b)
#expected output: 2.0
with tf.Session() as sess:
print( sess.run( c ) )
will output:
2.0
as desired.
This way you can factor this into an agnostic function that can take both scalar and non-scalar tensors as argument.
No, this is not possible. As you probably saw it throws an error:
ValueError: elems must be a 1+ dimensional Tensor, not a scalar
The point of map_fn is to apply a function to each element of a tensor, so it makes no sense to use this for a scalar (single-element) tensor.
As to "what you are doing wrong": This is difficult to say without knowing what you're trying to achieve.
Related
I want to compute a symbolic gradient with sympy, e.g.,
import sympy as sym
x, y, z = sym.symbols("x y z", real=True)
T = sym.cos(x**2+y**2)
gradT = sym.Matrix([sym.diff(T, x), sym.diff(T,y), sym.diff(T,z)])
Now I would like to create a lamddify function with this expression:
func = lambdify((x,y,z), gradT,'numpy')
To use the function I have:
gradT_exact = func(np.linspace(0,2,100), np.linspace(0,2,100), np.linspace(0,2,100))
and I receive the following error:
<lambdifygenerated-3>:2: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
return (array([[-2*x*sin(x**2 + y**2)], [-2*y*sin(x**2 + y**2)], [0]]))
If I change T to be a function of x,y,z it gives me no problems...
Why is it giving warnings when T only depends on x and y and z is set to zero.
Thanks in advance!
The gradT expression:
In [84]: gradT
Out[84]:
⎡ ⎛ 2 2⎞⎤
⎢-2⋅x⋅sin⎝x + y ⎠⎥
⎢ ⎥
⎢ ⎛ 2 2⎞⎥
⎢-2⋅y⋅sin⎝x + y ⎠⎥
⎢ ⎥
⎣ 0 ⎦
and its conversion to numpy:
In [87]: print(func.__doc__)
Created with lambdify. Signature:
func(x, y, z)
Expression:
Matrix([[-2*x*sin(x**2 + y**2)], [-2*y*sin(x**2 + y**2)], [0]])
Source code:
def _lambdifygenerated(x, y, z):
return (array([[-2*x*sin(x**2 + y**2)], [-2*y*sin(x**2 + y**2)], [0]]))
If x and y are arrays, then 2 terms will reflect their dimension(s), but the last is [0]. That's why you get the ragged warning.
lambdify does a rather simple lexical translation. It does not implement any deep understanding of numpy arrays. At some level it's your responsibility to check that the numpy code looks reasonable.
with scalar inputs:
In [88]: func(1,2,3)
Out[88]:
array([[1.91784855],
[3.8356971 ],
[0. ]])
but if one input is an array:
In [90]: func(np.array([1,2]),2,3)
<lambdifygenerated-1>:2: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
return (array([[-2*x*sin(x**2 + y**2)], [-2*y*sin(x**2 + y**2)], [0]]))
Out[90]:
array([[array([ 1.91784855, -3.95743299])],
[array([ 3.8356971 , -3.95743299])],
[0]], dtype=object)
The result is object dtype containing 2 arrays, plus that [0] list.
To avoid this problem, the lambdify would have to produce a function like:
In [95]: def f(x,y,z):
...: temp = 0*x*y
...: return np.array([-2*x*np.sin(x**2 + y**2), -2*y*np.sin(x**2 + y**2)
...: , temp])
where temp is designed to give 0 value, but with a shape that reflects the broadcasted operations on x and y in the other terms. I think that's asking too much of lambdify.
In [96]:
In [96]: f(np.array([1,2]),2,3)
Out[96]:
array([[ 1.91784855, -3.95743299],
[ 3.8356971 , -3.95743299],
[ 0. , 0. ]])
I am new to Numba and I need to use Numba to speed up some Pytorch functions. But I find even a very simple function does not work :(
import torch
import numba
#numba.njit()
def vec_add_odd_pos(a, b):
res = 0.
for pos in range(len(a)):
if pos % 2 == 0:
res += a[pos] + b[pos]
return res
x = torch.tensor([3, 4, 5.])
y = torch.tensor([-2, 0, 1.])
z = vec_add_odd_pos(x, y)
But the following error appears
def vec_add_odd_pos(a, b):
res = 0.
^
This error may have been caused by the following argument(s):
argument 0: cannot determine Numba type of <class 'torch.Tensor'>
argument 1: cannot determine Numba type of <class 'torch.Tensor'>
Can anyone help me? A link with more examples would be also appreciated. Thanks.
Pytorch now exposes an interface on GPU tensors which can be consumed by numba directly:
numba.cuda.as_cuda_array(tensor)
The test script provides a few usage examples: https://github.com/pytorch/pytorch/blob/master/test/test_numba_integration.py
As others have mentioned, numba currently doesn't support torch tensors, only numpy tensors. However there is TorchScript, which has a similar goal. Your function can then be rewritten as such:
import torch
#torch.jit.script
def vec_add_odd_pos(a, b):
res = 0.
for pos in range(len(a)):
if pos % 2 == 0:
res += a[pos] + b[pos]
return res
x = torch.tensor([3, 4, 5.])
y = torch.tensor([-2, 0, 1.])
z = vec_add_odd_pos(x, y)
Beware: although you said your code snippet was just a simple example, for loops are really slow and running TorchScript might not help you much, you should avoid them at any cost and only use then when no other solution exist. That being said, here's how to implement your function in a more performant way:
def vec_add_odd_pos(a, b):
evenids = torch.arange(len(a)) % 2 == 0
return (a[evenids] + b[evenids]).sum()
numba supports numpy-arrays but not torch's tensors. There is however a bridge Tensor.numpy():
Returns self tensor as a NumPy ndarray. This tensor and the returned
ndarray share the same underlying storage. Changes to self tensor will
be reflected in the ndarray and vice versa.
That means you have to call jitted functions as:
...
z = vec_add_odd_pos(x.numpy(), y.numpy())
If z should be a torch.Tensor as well, torch.from_numpy is what we need:
Creates a Tensor from a numpy.ndarray.
The returned tensor and ndarray share the same memory. Modifications
to the tensor will be reflected in the ndarray and vice versa. The
returned tensor is not resizable.
...
For our code that means
...
z = torch.from_numpy(vec_add_odd_pos(x.numpy(), y.numpy()))
should be called.
I was wondering if someone can maybe have a quick look at the following code snippet and point me in a direction to find my misunderstanding in calculating the probability of a sample for each class in the model and my related code bug. I tried to manually calculate the results provided by the sklearn function lm.predict_proba(X) , sadly the results are different, so i did a mistake.
I think the bug will be in part "d" of the following code walkthrough. Maybe in the math, but I could not see why.
a) Creating and training a logistic regression model ( works fine )
lm = LogisticRegression(random_state=413, multi_class='multinomial', solver='newton-cg')
lm.fit(X, train_labels)
b) Saving coefficient and bias ( works fine )
W = lm.coef_
b = lm.intercept_
c) Using lm.predict_proba(X) ( works fine)
def reshape_single_element(x,num):
singleElement = x[num]
nx,ny = singleElement.shape
return singleElement.reshape((1,nx*ny))
select_image_number = 6
X_select_image_data=reshape_single_element(train_dataset,select_image_number)
Y_probabilities = lm.predict_proba(X_select_image_data)
Y_pandas_probabilities = pd.Series(Y_probabilities[0], index=['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'])
print"estimate probabilities for each class: \n" ,Y_pandas_probabilities , "\n"
print"all probabilities by lm.predict_proba(..) sum up to ", np.sum(Y_probabilities) , "\n"
The output was:
estimate probabilities for each class:
a 0.595426
b 0.019244
c 0.001343
d 0.004033
e 0.017185
f 0.004193
g 0.160380
h 0.158245
i 0.003093
j 0.036860
dtype: float64
all probabilities by lm.predict_proba(..) sum up to 1.0
d) Manually performing the calculation done by lm.predict_proba ( no error/warning, but results are not the same )
manual_calculated_probabilities = []
for select_class_k in range(0,10): #a=0. b=1, c=3 ...
z_for_class_k = (np.sum(W[select_class_k] *X_select_image_data) + b[select_class_k] )
p_for_class_k = 1/ (1 + math.exp(-z_for_class_k))
manual_calculated_probabilities.append(p_for_class_k)
print "formula: ", manual_calculated_probabilities , "\n"
def softmax(x):
"""Compute softmax values for each sets of scores in x."""
e = np.exp(x)
dist = e / np.sum(np.exp(x),axis=0)
return dist
abc = softmax(manual_calculated_probabilities)
print "softmax:" , abc
The output was:
formula: [0.9667598370531315, 0.48453459121301334, 0.06154496922245115, 0.16456194859398865, 0.45634781280053394, 0.16999340794727547, 0.8867996361191054, 0.8854473986336552, 0.13124464656251109, 0.642913996162282]
softmax: [ 0.15329642 0.09464644 0.0620015 0.0687293 0.0920159 0.069103610.14151607 0.14132483 0.06647715 0.11088877]
Softmax was used, because of a comment at github logistic.py
For a multi_class problem, if multi_class is set to be "multinomial" the softmax function is used to find the predicted probability of each class.
Note:
print "shape of X: " , X_select_image_data.shape
print "shape of W: " , W.shape
print "shape of b: " , b.shape
shape of X: (1, 784)
shape of W: (10, 784)
shape of b: (10,)
I found a very similar question here, but sadly I could not adapted it to my code so the predictions got the same. I tried many different combinations to calculate the variables 'z_for_class_k' and 'p_for_class_k' but sadly without success to reproduce the prediction values from 'predict_proba(X)'.
I think the problem is with
p_for_class_k = 1/ (1 + math.exp(-z_for_class_k))
1 / (1 + exp(-logit)) is a simplification that works only on binary problems.
The real equation, before being simplified, looks like this:
p_for_classA =
exp(logit_classA) /
[1 + exp(logit_classA) + exp(logit_classB) ... + exp(logit_classC)]
In other words, when calculating a probability for a specific class, you must incorporate ALL the weights and biases from the other classes as well into your formula.
I didn't have the data to test this out, but hopefully this points you in the right direction.
change
p_for_class_k = 1/ (1 + math.exp(-z_for_class_k))
manual_calculated_probabilities.append(p_for_class_k)
to
manual_calculated_probabilities.append(z_for_class_k)
aka the input for softmax is "z"s instead of "p"s, in your notation. multinomial logistic
I was able to replicate the method lr.predict_proba by doing the following:
>>> sigmoid = lambda x: 1/(1+np.exp(-x))
>>> sigmoid(lr.intercept_+np.sum(lr.coef_*X.values, axis=1))
Assuming that X is a numpy array and lr is an object from sklearn.
I don't understand why do we need tensor.reshape() function in Theano. It is said in the documentation:
Returns a view of this tensor that has been reshaped as in
numpy.reshape.
As far as I understood, theano.tensor.var.TensorVariable is some entity that is used for computation graphs creation. And it is absolutely independent of shapes. For instance when you create your function you can pass there matrix 2x2 or matrix 100x200. As I thought reshape somehow restricts this variety. But it is not. Suppose the following example:
X = tensor.matrix('X')
X_resh = X.reshape((3, 3))
Y = X_resh ** 2
f = theano.function([X_resh], Y)
print(f(numpy.array([[1, 2], [3, 4]])))
As I understood, it should give an error since I passed matrix 2x2 not 3x3, but it computes element-wise squares perfectly.
So what is the shape of the theano tensor variable and where should we use it?
There is an error in the provided code though Theano fails to point this out.
Instead of
f = theano.function([X_resh], Y)
you should really use
f = theano.function([X], Y)
Using the original code you are actually providing the tensor after the reshape so the reshape command never gets executed. This can be seen by adding
theano.printing.debugprint(f)
which prints
Elemwise{sqr,no_inplace} [id A] '' 0
|<TensorType(float64, matrix)> [id B]
Note that there is no reshape operation in this compiled execution graph.
If one changes the code so that X is used as the input instead of X_resh then Theano throws an error including the message
ValueError: total size of new array must be unchanged Apply node that
caused the error: Reshape{2}(X, TensorConstant{(2L,) of 3})
This is expected because one cannot reshape a tensor with shape (2, 2) (i.e. 4 elements) into a tensor with shape (3, 3) (i.e. 9 elements).
To address the broader question, we can use symbolic expressions in the target shape and those expressions can be functions of the input tensor's symbolic shape. Here's some examples:
import numpy
import theano
import theano.tensor
X = theano.tensor.matrix('X')
X_vector = X.reshape((X.shape[0] * X.shape[1],))
X_row = X.reshape((1, X.shape[0] * X.shape[1]))
X_column = X.reshape((X.shape[0] * X.shape[1], 1))
X_3d = X.reshape((-1, X.shape[0], X.shape[1]))
f = theano.function([X], [X_vector, X_row, X_column, X_3d])
for output in f(numpy.array([[1, 2], [3, 4]])):
print output.shape, output
I am really new to Theano, and I am just trying to figure out some basic functionality. I have a tensor variable x, and i would like the functio to return a tensor variable y of the same shape, but filled with value 0.2. I am not sure how to define y.
For example if x = [1,2,3,4,5], then I would like y = [0,2, 0,2, 0,2, 0,2, 0.2]
from theano import tensor, function
y = tensor.dmatrix('y')
masked_array = function([x],y)
There's probably a dozen different ways to do this and which is best will depend on the context: how this piece of code/functionality fits into the wider program.
Here's one approach:
import theano
import theano.tensor as tt
x = tt.vector()
y = tt.ones_like(x) * 0.2
f = theano.function([x], outputs=y)
print f([1, 2, 3, 4, 5])