I read all posts about the Dot Layer but none explains how this and so the output shape is computed! It seems so standard though!
How exactly are the values computed with a along a specific axis?
val = np.random.randint(2, size=(2, 3, 4))
a = K.variable(value=val)
val2 = np.random.randint(2, size=(2, 2, 3))
b = K.variable(value=val)
print("a")
print(val)
print("b")
print(val2)
out = Dot(axes = 2)([a,b])
print(out.shape)
print("DOT")
print(K.eval(out))
I get:
a
[[[0 1 1 1]
[1 1 0 0]
[0 0 1 1]]
[[1 1 1 0]
[0 0 1 0]
[0 1 0 0]]]
b
[[[1 0 1]
[1 0 1]]
[[1 0 1]
[1 1 0]]]
(2, 3, 3)
DOT
[[[ 3. 1. 2.]
[ 1. 2. 0.]
[ 2. 0. 2.]]
[[ 3. 1. 1.]
[ 1. 1. 0.]
[ 1. 0. 1.]]]
I cannot understand with my mathematical and algebraic matrix know-how how the heck this is computed?
Here's how the Dot product works. Internally it is calling K.batch_dot.
First, I think you might have intended to do,
val = np.random.randint(2, size=(2, 3, 4))
a = K.variable(value=val)
val2 = np.random.randint(2, size=(2, 2, 3))
b = K.variable(value=val2) # You have val here
But fortunately, you had (or could have been your initial intention too. Anyway just pointing out)
b = K.variable(value=val)
If you had the intended code, it will throw an error because the dimension you want the dot product on, doesn't match. Moving on,
How dot product is computed
You have
a.shape = (2,3,4)
b.shape = (2,3,4)
First you are only performing element-wise dot over the batch dimension. So that dimension stays that way.
Now you can ignore the first dimension of both a and b and consider the dot product between two matrices (3,4) and (3,4) and do the dot product over the last axis, which results in a (3,3) matrix. Now add the batch dimension you get a,
(2, 3, 3) tensor
Let's now take your example. You got,
a
[[[0 1 1 1]
[1 1 0 0]
[0 0 1 1]]
[[1 1 1 0]
[0 0 1 0]
[0 1 0 0]]]
b
[[[0 1 1 1]
[1 1 0 0]
[0 0 1 1]]
[[1 1 1 0]
[0 0 1 0]
[0 1 0 0]]]
Then you do the following two dot products.
# 1st sample
[0 1 1 1] . [0 1 1 1]
[1 1 0 0] . [1 1 0 0]
[0 0 1 1] . [0 0 1 1]
# 2nd sample
[1 1 1 0] . [1 1 1 0]
[0 0 1 0] . [0 0 1 0]
[0 1 0 0] . [0 1 0 0]
This gives,
# 1st sample
[3 1 2]
[1 2 0]
[2 0 2]
# 2nd sample
[ 3 1 1]
[ 1 1 0]
[ 1 0 1]
Finally by adding the missing batch dimension you get,
[[[ 3. 1. 2.]
[ 1. 2. 0.]
[ 2. 0. 2.]]
[[ 3. 1. 1.]
[ 1. 1. 0.]
[ 1. 0. 1.]]]
Related
I'm following documentation here:
https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
>>> from sklearn.feature_extraction.text import CountVectorizer
>>> corpus = [
... 'This is the first document.',
... 'This document is the second document.',
... 'And this is the third one.',
... 'Is this the first document?',
... ]
>>> vectorizer = CountVectorizer()
>>> X = vectorizer.fit_transform(corpus)
>>> print(vectorizer.get_feature_names())
['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
>>> print(X.toarray())
[[0 1 1 1 0 0 1 0 1]
[0 2 0 1 0 1 1 0 1]
[1 0 0 1 1 0 1 1 1]
[0 1 1 1 0 0 1 0 1]]
Suppose I already have a term frequency matrix like the one given in X.toarray(), but I didn't use CountVectorizer to obtain it.
I want to apply a TfIDF to this matrix. Is there a way for me to take a count array + a dictionary and apply some inverse of this function as a constructor to get a fit_transformed X?
I'm looking for...
>>> print(X.toarray())
[[0 1 1 1 0 0 1 0 1]
[0 2 0 1 0 1 1 0 1]
[1 0 0 1 1 0 1 1 1]
[0 1 1 1 0 0 1 0 1]]
>>> V = CountVectorizerConstructorPrime(array=(X.toarray()),
vocabulary=['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this'])
such that:
>>> V == X
True
The X constructed by the CountVectorizer is a sparse matrix in SciPy's compressed sparse row (csr) format. So you can construct it directly from any word count matrix with the appropriate SciPy function:
from scipy.sparse import csr_matrix
V = csr_matrix(X.toarray())
Now V and X are equal, although this may not be obvious, because V == X will give you another sparse matrix (or rather complain that the matrix is not sparse despite the intended format, see this question). But you can check it like this:
(V != X).toarray().any()
False
Note that the word list was not needed, because the matrix only encodes the frequencies of all distinct words, no matter what they are.
torch.nn.Conv2d(7, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))
In the above line of code for convolution in pytorch what is happening with the padding = (3,3) parameter. How can we implement the same padding in tensorflow.
tf.keras.layers.Conv2D(7, 64, stride = (2,2), padding="same padding as pytorch")
How can we implement this in tensorflow ?
The Padding is adding extra bytes on each dimension in the image. As per the documentation,
padding controls the amount of implicit zero-paddings on both sides
for padding number of points for each dimension.
In Tensorflow, you can use tf.pad to do the same. For example to do padding=(3, 3) similar to pytorch, you can use below code -
import tensorflow as tf
image = tf.constant([[1, 2, 3], [4, 5, 6]])
paddings = tf.constant([[3, 3,], [3, 3]])
image = tf.pad(image, paddings, "CONSTANT")
print(image)
Output-
tf.Tensor(
[[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 1 2 3 0 0 0]
[0 0 0 4 5 6 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]], shape=(8, 9), dtype=int32)
I am working with a pandas.Dataframe that looks as follows:
A B C D
index
1 0 0 0 1
2 1 0 0 1
3 ...
4 ...
...
And I am creating a numpy.arrays that have the same shape as a row within this dataframe. I want to check if the array I am creating 'is present' within the dataframe.
In this case, for example, my array would look like this, if it is in the dataframe:
a= [0,0,0,1]
It is not if it looks like this:
b = [1,1,1,1]
Any help, even if it is a link to the right answer, is much appreciated as I have looked through stackoverflow and fortunately I did not miss anything.
df = pd.DataFrame({'A':[0, 1, 0, 0],
'B':[0, 0, 1, 1],
'C':[0, 0, 0, 0],
'D':[1, 1, 0, 1]})
# A B C D
# 0 0 0 0 1
# 1 1 0 0 1
# 2 0 1 0 0
# 3 0 1 0 1
>>> a = [0, 0, 0, 1]
>>> (df == a).all(axis=1).any()
True
>>> b = [1, 1, 1, 1]
>>> (df == b).all(axis=1).any()
False
I am using OPL CPLEX 12.9
and I want to write my results in an excel file.
It is a two dimension array and I want to write the results in two columns but is it only possible to write the Data in lines?
Example Code in .dat
Product_Cost to SheetWrite (Daten, "Result!A1:B10");
What can I write for A1:B10 to get the results in two columns?
before doing SheetWrite you could transpose your matrix:
int M=2;
int N=5;
int A[i in 1..M][j in 1..N] = rand(4);
int B [j in 1..N][i in 1..M]=A[i][j];
execute
{
writeln("A=",A);
writeln("B=",B);
}
gives
A= [[0 0 0 0 1]
[3 2 3 2 0]]
B= [[0 3]
[0 2]
[0 3]
[0 2]
[1 0]]
I tried different ways but it seems impossible for me to do it efficiently without looping through.
Input is an array y and a percentage x.
e.g. input is
y=np.random.binomial(1,1,[10,10])
x=0.5
output
[[0 0 0 0 1 1 1 1 0 1]
[1 0 1 0 0 1 0 1 0 1]
[1 0 1 1 1 1 0 0 0 1]
[0 1 0 1 1 0 1 0 1 1]
[0 1 1 0 0 1 1 1 0 0]
[0 0 1 1 1 0 1 1 0 1]
[0 1 0 0 0 0 1 0 1 1]
[0 0 0 1 1 1 1 1 0 0]
[0 1 1 1 1 0 0 1 0 0]
[1 0 1 0 1 0 0 0 0 0]]
Here's one based on masking -
def set_nonzeros_to_zeros(a, setz_ratio):
nz_mask = a!=0
nz_count = nz_mask.sum()
z_set_count = int(np.round(setz_ratio*nz_count))
idx = np.random.choice(nz_count,z_set_count,replace=False)
mask0 = np.ones(nz_count,dtype=bool)
mask0.flat[idx] = 0
nz_mask[nz_mask] = mask0
a[~nz_mask] = 0
return a
We are skipping the generation all the indices with np.argwhere/np.nonzero in favor of a masking based one to focus on performance.
Sample run -
In [154]: np.random.seed(0)
...: a = np.random.randint(0,3,(5000,5000))
# number of non-0s before using solution
In [155]: (a!=0).sum()
Out[155]: 16670017
In [156]: a_out = set_nonzeros_to_zeros(a, setz_ratio=0.2) #set 20% of non-0s to 0s
# number of non-0s after using solution
In [157]: (a_out!=0).sum()
Out[157]: 13336014
# Verify
In [158]: 16670017 - 0.2*16670017
Out[158]: 13336013.6
There are a few vectorized methods that might help you, depending on what you want to do:
# Flatten the 2D array and get the indices of the non-zero elements
c = y.flatten()
d = c.nonzero()[0]
# Shuffle the indices and set the first 100x % to zero
np.random.shuffle(d)
x = 0.5
c[d[:int(x*len(d))]] = 0
# reshape to the original 2D shape
y = c.reshape(y.shape)
No doubt there are some efficiency improvements to be made here.