I am trying to optimize some code which has some loops and matrix operations. However, I am running into some errors. Please find the code and output below.
Code:
#njit
def list_of_distance(d1): #d1 was declared as List()
list_of_dis = List()
for k in range(len(d1)):
sum_dist = List()
for j in range(3):
s = np.sum(square(np.reshape(d1[k][:,:,j].copy(),d1[k][:,:,j].shape[0]*d1[k][:,:,j].shape[1])))
sum_dist.append(s) # square each value in the resulting list (dimenstion)
distance = np.sum(sum_dist) # adding the total value for each dimension to a list
list_of_dis.append(np.round(np.sqrt(distance))) # Sum the values to get the total squared values of residual images
return list_of_dis
Output:
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<function sum at 0x7f898814bd08>) with argument(s) of type(s): (list(int64))
* parameterized
In definition 0:
All templates rejected with literals.
In definition 1:
All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: Function(<function sum at 0x7f898814bd08>)
[2] During: typing of call at <ipython-input-18-8c787cc8deda> (7)
File "<ipython-input-18-8c787cc8deda>", line 7:
def list_of_distance(d1):
<source elided>
for j in range(3):
s = np.sum(square(np.reshape(d1[k][:,:,j].copy(),d1[k][:,:,j].shape[0]*d1[k][:,:,j].shape[1])))
^
This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.
To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html
For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile
If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new
Would anyone be able to help me out regarding this issue.
Thanks & Best Regards
Michael
I had to make a few changes to get this to work and mocked up "d1", but this does work for me with Numba. This main issue that caused the runtime error appears to be that np.sum does not work on list with Numba, although it did run correctly when I commented out #jit. wrapping sumdist with np.array() resolves this issue.
d1 = [np.arange(27).reshape(3,3,3), np.arange(27,54).reshape(3,3,3)]
#njit
def list_of_distance(d1): #d1 was declared as List()
list_of_dis = [] #List() Changed - would not compile
for k in range(len(d1)):
sum_dist = [] #List() #List() Changed - would not compile
for j in range(3):
s = np.sum(np.square(np.reshape(d1[k][:,:,j].copy(),d1[k][:,:,j].shape[0]*d1[k][:,:,j].shape[1]))) #Added np. to "square"
sum_dist.append(s) # square each value in the resulting list (dimenstion)
distance = np.sum(np.array(sum_dist)) # adding the total value for each dimension to a list - Wrapped list in np.array
list_of_dis.append(np.round(np.sqrt(distance))) # Sum the values to get the total squared values of residual images
return list_of_dis
list_of_distance(d1)
Out[11]: [79.0, 212.0]
Related
Code:
a=training_dataset.map(lambda x,y: (tf.pad(x,tf.constant([[13-int(tf.shape(x)[0]),0],[0,0]])),y))
gives the following error:
TypeError: in user code:
<ipython-input-32-b25101c2110a>:1 None *
a=training_dataset.map(lambda x,y: (tf.pad(tensor=x,paddings=tf.constant([[13-int(tf.shape(x)[0]),0],[0,0]]),mode="CONSTANT"),y))
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py:264 constant **
allow_broadcast=True)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py:282 _constant_impl
allow_broadcast=allow_broadcast))
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_util.py:456 make_tensor_proto
_AssertCompatible(values, dtype)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_util.py:333 _AssertCompatible
raise TypeError("Expected any non-tensor type, got a tensor instead.")
TypeError: Expected any non-tensor type, got a tensor instead.
However, when I use:
a=training_dataset.map(lambda x,y: (tf.pad(x,tf.constant([[1,0],[0,0]])),y))
Above code works fine.
This brings me to the conclusion that something is wrong with: 13-tf.shape(x)[0] but cannot understand what.
I tried converting the tf.shape(x)[0] to int(tf.shape(x)[0]) and still got the same error.
What I want the code to do:
I have a tf.data.Dataset object having variable length sequences of size (None,128) where the first dimension(None) is less than 13. I want to pad the sequences such that the size of every collection is 13 i.e (13,128).
Is there any alternate way (if the above problem cannot be solved)?
A solution that works:
using:
paddings = tf.concat(([[13-tf.shape(x)[0],0]], [[0,0]]), axis=0)
instead of using:
paddings = tf.constant([[13-tf.shape(x)[0],0],[0,0]])
works for me.
However, I still cannot figure out why the latter one did not work.
I am trying to execute following 'for loop'.
However, after waiting for several hours, what I've got was the error message that says out of memory.
Thus, I want to convert the for loop to numpy in order to reduce time and memory require for this calculation.
Unfortunately, I failed to do it.
It would be a great help if anyone can suggest the way to convert the following code into the one that uses the numpy array.
The type of la,lo is 'pandas.core.series.Series',
and the type of ste is 'obspy.core.stream.Stream'.
def distance(a,i):
return math.sqrt(((ste[a].stats.sac.stla-la[i])**2)+((ste[a].stats.sac.stlo-lo[i])**2))
def traveltime(a):
return model.get_travel_times(source_depth_in_km=35, distance_in_degree=a, phase_list=["S"], receiver_depth_in_km=0)[0].time
di=[(la[i],lo[i],distance(a,i), distance(b,i)) for i in range(len(lo))
for a in range(len(ste))
for b in range(len(ste)) if a<b]
didf=pd.DataFrame(di)
latot=didf[0]
lotot=didf[1]
dia=didf[2]
dib=didf[3]
tt=[]
for i in range(len(di)):
try:
tt.append((latot[i],lotot[i],traveltime(dia[i])-traveltime(dib[i])))
except IndexError:
continue
ttdf=pd.DataFrame(tt)
final=[(win[j],ttdf[0][i],ttdf[1][i],(ttdf[2][i]-shift[j])**2) for i in range(len(ttdf))
for j in range(len(ccdf))]
P.S)
For more information, I add the codes that are executed after the codes above. These worked well when I put smaller amount of data. Also, I add a image I get after executing all these codes.
fidf=pd.DataFrame(final)
titles = {0: 'winnum', 1: 'latitude', 2: 'longitude', 3: 'difference'}
fidf.rename(columns=titles, inplace=True)
gls={}
for i in range(10):
fidf=fidf.sample(frac=0.9,replace=True,random_state=1)
grouped=fidf['difference'].groupby([fidf['winnum'],fidf['latitude'],fidf['longitude']]).sum()
grouped=grouped.loc[grouped.groupby('winnum').idxmin()]
gls[i]=grouped.reset_index()
median=pd.Panel(gls).median(axis=0)
std=pd.Panel(gls).std(axis=0)
print(median)
print(std)
I'm working with some old Python2 code from a differential power analysis (DPA) contest from way back (http://www.dpacontest.org/home/index.html). The code I'm modifying can be found here (https://svn.comelec.enst.fr/dpacontest/code/reference/)
Current issue that is driving me nuts:
Essentially, Some python objects in this code will "erase" themselves or go to their default values, and I have no idea why. I cannot find any code in the logic flow that does this, and I don't think it's a scope issue.
I've tried rewriting parts of the code to use numpy instead of mapping lambda functions. I've also tried a myriad of different orderings of code and pulling methods out of their classes and trying to run them locally/inline.
main.py:
loc_subkeys = brk.get_subkeys()
des_breaker.py
def get_subkeys(self):
"""
Returns a vector of currently best sboxes subkeys.
This is an array of 8 integers.
"""
sk = np.array([])
for i in range(8):
sk = np.append(sk, self.__sbox_breakers[i].get_key())
return sk
sbox_breaker.py
def get_key(self):
"Gives the current best key"
if self.__best_key is None:
marks = np.array([])
print("p0: ", len(list(self.__key_estimators[0]._key_estimator__p0)))
print("p1: ", len(list(self.__key_estimators[0]._key_estimator__p1)))
print("p0: ", len(list(self.__key_estimators[0]._key_estimator__p0)))
print("p1: ", len(list(self.__key_estimators[0]._key_estimator__p1)))
for i in range(64):
ke = self.__key_estimators[i]
marks = np.append(marks, ke.get_mark())
self.__best_key = np.argmax(marks)
return self.__best_key
key_estimator.py - attributes
class key_estimator:
"""
Provides methods to give a mark to the key relatively to the probability f
the correctness of the key.
"""
__sbox = None
__key = None
__cnt0 = 0 # The accumulated traces count in partition 0
__cnt1 = 0 # The accumulated traces count in partition 1
__p0 = None # The bit=0 estimated partition
__p1 = None # The bit=1 estimated partition
__diff = np.array([]) # The differential trace
Print statements in sbox_breaker are mine. Their output is the only clue I have right now:
p0: 5003 (Good)
p1: 5003 (Good)
p0: 0 (???)
p1: 0
What gives? The second time around the attributes of the key_estimator class have seemed to erase themselves. This happens to all the attributes, not just p0 and p1.
The first loop through this program works, but on the second iteration (starting from main) it fails because the attributes have erased themselves. I can "erase" them manually just by printing the object's attributes.
So I seemed to have fixed the problem after sleeping on it. The class attributes were being created by map which returns a list in Python2, but not Python3. Making them into lists with list() solves the persistence issue. I couldn't tell you why printing a map attribute causes it to clear itself though.
Sadly I get this error when I try to do max() tried with multiple [] () combination and the error keeps on coming .
Looks like this is minor issue and easily solvable. Before posting it here referred some of the existing posts still could not figure out the way.
Any help much appreciated. The code fails after if is evaluated (last line)
for i in range(1, len(df)):
if(df[source].iat[i] > df[trail].iat[i - 1]) and (df[source].iat[i-1] > df[trail].iat[i-1]):
df[trail].iat[i] = [df[trail].iat[i-1],df[source].iat[i]- df['nLoss'].iat[i]].max()
error :'list' object has no attribute 'max'
Thanks for your support in advance.
Need function max function working with iterables in python:
for i in range(1, len(df)):
if(df[source].iat[i] > df[trail].iat[i - 1]) and (df[source].iat[i-1] > df[trail].iat[i-1]):
df[trail].iat[i] = max([df[trail].iat[i-1],df[source].iat[i]- df['nLoss'].iat[i]])
I'd like to build a tensorflow graph in a separate function get_graph(), and to print out a simple ops a in the main function. It turns out that I can print out the value of a if I return a from get_graph(). However, if I use get_operation_by_name() to retrieve a, it print out None. I wonder what I did wrong here? Any suggestion to fix it? Thank you!
import tensorflow as tf
def get_graph():
graph = tf.Graph()
with graph.as_default():
a = tf.constant(5.0, name='a')
return graph, a
if __name__ == '__main__':
graph, a = get_graph()
with tf.Session(graph=graph) as sess:
print(sess.run(a))
a = sess.graph.get_operation_by_name('a')
print(sess.run(a))
it prints out
5.0
None
p.s. I'm using python 3.4 and tensorflow 1.2.
Naming conventions in tensorflow are subtle and a bit offsetting at first.
The thing is, when you write
a = tf.constant(5.0, name='a')
a is not the constant op, but its output. Names of op outputs derive from the op name by adding a number corresponding to its rank. Here, constant has only one output, so its name is
print(a.name)
# `a:0`
When you run sess.graph.get_operation_by_name('a') you do get the constant op. But what you actually wanted is to get 'a:0', the tensor that is the output of this operation, and whose evaluation returns an array.
a = sess.graph.get_tensor_by_name('a:0')
print(sess.run(a))
# 5