Filled with numpy.zero but getting nan instead - python-3.x

Question

I have a Class that contains various methods with include the following:
def _doc_mean(self, doc):
doc_vector_values = []
for w in doc:
#print(w)
if w.lower().strip() in self._E:
Q = np.zeros((1, 200), dtype=np.float64) #this is a zero array for when a word doesnt have a vector representation in our pretrained embeddings
doc_vector_values.append(self._E.get(w, Q))
with warnings.catch_warnings():
warnings.simplefilter("ignore", category=RuntimeWarning)
return np.mean(np.array(doc_vector_values, dtype=np.float64), axis=0)
def fit(self, X, y=None):
return self
def transform(self, X):
return np.array([self._doc_mean(doc) for doc in X])
def fit_transform(self, X, y=None):
return self.fit(X).transform(X)
in _doc_mean, i compare w with the keys in a dictionary E_, if there is a match, then load the value of the key-value pair which contains a 1*200 vector into a list, if there is no match, then load numpy.zeros((1,200)) into a list. This list is now converted to an array and the mean calculated.
When i instantiate the class and fit-transform my 'doc' data
mc = MeanClass()
X_ = mc.fit_transform(doc)
X_ is of dtype "object" and the places where there was a mismatch was replaced with nan instead of numpy.zero.
This leads to multiple other problems in my code that i cant fix. What am i doing wrong?
EDIT:
The E_ dictionary looks like this :
{'hello': array([ 5.84850e-02, 6.20640e-02, ..... -2.08990e-02])
'good': array([ -4.80050e-02, 2.80610e-02, ..... -5.04991e-02])
while doc looks like this :
['hello', 'bye', 'good']
['good', 'bye', 'night']

Answer 1

Since you haven't given a [mcve], I'll create something simple:
In [125]: E_ = {'foo':np.arange(5), 'bar':np.arange(1,6), 'baz':np.arange(5,10)}
In [126]: doc = ['foo','bar','sub','baz','foo']
Now do the dictionary lookup:
In [127]: alist = []
In [128]: for w in doc:
...: alist.append(E_.get(w,np.zeros((1,5),int)))
...:
In [129]: alist
Out[129]:
[array([0, 1, 2, 3, 4]),
array([1, 2, 3, 4, 5]),
array([[0, 0, 0, 0, 0]]),
array([5, 6, 7, 8, 9]),
array([0, 1, 2, 3, 4])]
In [130]: np.array(alist)
Out[130]:
array([array([0, 1, 2, 3, 4]), array([1, 2, 3, 4, 5]),
array([[0, 0, 0, 0, 0]]), array([5, 6, 7, 8, 9]),
array([0, 1, 2, 3, 4])], dtype=object)
The arrays in E_ are all shape (5,). The 'fill' array is (1,5). Due to the mismatch in shapes, the Out[130] array is 1d object.
I think you are trying to avoid the 'fill' case, but you test w.lower().strip() in self._E, and then use w in the get. So you might get the Q value sometimes. I got it with the 'sub' string.
If instead I make the 'fill' be (5,):
In [131]: alist = []
In [132]: for w in doc:
...: alist.append(E_.get(w,np.zeros((5,),int)))
...:
In [133]: alist
Out[133]:
[array([0, 1, 2, 3, 4]),
array([1, 2, 3, 4, 5]),
array([0, 0, 0, 0, 0]),
array([5, 6, 7, 8, 9]),
array([0, 1, 2, 3, 4])]
In [134]: np.array(alist)
Out[134]:
array([[0, 1, 2, 3, 4],
[1, 2, 3, 4, 5],
[0, 0, 0, 0, 0],
[5, 6, 7, 8, 9],
[0, 1, 2, 3, 4]])
The result is a (n,5) numeric array.
I can take two different means. One is the mean across all words, with a value for each 'attribute'. The other is the mean for each word, which I could just as well have gotten by taking the mean in E_.
In [135]: np.mean(_, axis=0)
Out[135]: array([1.2, 2. , 2.8, 3.6, 4.4])
In [137]: np.mean(__, axis=1)
Out[137]: array([2., 3., 0., 7., 2.]) # mean for each 'word'
mean of the object array in Out[130]:
In [138]: np.mean(_130, axis=0)
Out[138]: array([[1, 2, 2, 3, 4]])
The result is (1,5) and looks like Out[135] truncated, but I'd have to dig a bit further to be sure.
Hopefully this gives you an idea of what to watch out for. And an idea of the kind of 'minimal reproducable concrete example' that we find most useful.