Having two tensors :inputs_tokens is a batch of 20x300 of token ids
and seq_A is my model output with size of [20, 300, 512] (512 vector for each of the tokens in the batch)
seq_A.size()
Out[1]: torch.Size([20, 300, 512])
inputs_tokens.size()
torch.Size([20, 300])
I would like to get only the vectors of the token 101 (CLS) as follow:
cls_tokens = (inputs_tokens == 101)
cls_tokens
Out[4]:
tensor([[ True, False, False, ..., False, False, False],
[ True, False, False, ..., False, False, False],
[ True, False, False, ..., False, False, False], ...
How do I slice seq_A to get only the vectors which are true in cls_tokens for each batch?
when I do
seq_A[cls_tokens].size()
Out[7]: torch.Size([278, 512])
but I still need it to bee in the size of [20 x N x 512 ] (otherwise I don't know to which sample it belongs)
TLDR; You can't, all sequences must have the same size along a given axis.
Take this simplified example:
>>> inputs_tokens = torch.tensor([[ 1, 101, 18, 101, 9],
[ 1, 2, 101, 101, 101]])
>>> inputs_tokens.shape
torch.Size([2, 5])
>>> cls_tokens = inputs_tokens == 101
tensor([[False, True, False, True, False],
[False, False, True, True, True]])
Indexing inputs_tokens with the cls_tokens mask comes down to reducing inputs_tokens to cls_tokens's true values. In a general case where there is a different number of true values per batch, keeping the shape is impossible.
Following the above example, here is seq_A:
>>> seq_A = torch.rand(2, 5, 1)
tensor([[[0.4644],
[0.7656],
[0.3951],
[0.6384],
[0.1090]],
[[0.6754],
[0.0144],
[0.7154],
[0.5805],
[0.5274]]])
According to your example, you would expect to have an output shape of (2, N, 1). What would N be? 3? What about the first batch which only as 2 true values? The resulting tensor can't have different sizes (2 and 3 on axis=1). Hence: "all sequences on axis=1 must have the same size".
If however, you are expecting each batch to have the same number of tokens 101, then you could get away with a broadcast of your indexed tensor:
>>> inputs_tokens = torch.tensor([[ 1, 101, 101, 101, 9],
[ 1, 2, 101, 101, 101]])
>>> inputs_tokens.shape
>>> N = cls_tokens[0].sum()
3
Here remember, I'm assuming you have:
>>> assert all(cls_tokens.sum(axis=1) == N)
Therefore the desired output (with shape (2, 3, 1)) is:
>>> seq_A[cls_tokens].reshape(seq_A.size(0), N, -1)
tensor([[[0.7656],
[0.3951],
[0.6384]],
[[0.7154],
[0.5805],
[0.5274]]])
Edit - if you really want to do this though you would require the use of a list comprehension:
>>> [seq_A[i, cls_tokens[i]] for i in range(cls_tokens.size(0))]
[ tensor([[0.7656],
[0.6384]]),
tensor([[0.7154],
[0.5805],
[0.5274]]) ]
Related
I have two tensors.
tensor_1 = tensor([[10, 8, 7],
[ 4, 5, 9]])
target_tensor = tensor([[ 8, 8, 6, 11, 0],
[ 7, 2, 10, 4, 5]])
and torch.isin(tensor_1, target_tensor) returns
tensor([[ True, True, True],
[ True, True, False]])
I want to apply isin to each row and get result like below
tensor([[False, True, False],
[ True, True, False]])
I can do this using for loop, but I don't want to use loop.
tmp_result = torch.zeros_like(tensor_1, dtype=torch.bool)
for i in range(tensor_1.size(0)):
tmp_result[i] = torch.isin(tensor_1[i], target_tensor[i])
How can I apply torch.isin() to each row in 2D tensor using GPU without loop?
I'm trying to solve an image search issue using NumPy and Pandas for weeks now. Would like to seek for some advice regarding same as I'm stuck and feeling back to square one with any attempt.
There're 2 image sets. The first set of images are full images and another set is smaller / cut version (chunks). The smaller chunks can be in any order (flipped, transposed, rotated, etc).
Converted both to corresponding NumPy matrices.
For simplicity, consider the below 2 matrices. I'm using a smaller size for illustration, but the actuals are 10000x12000 or more.
array([[ 2, 15, 9, 16, 4, 3, 12, 8],
[ 9, 9, 0, 16, 0, 1, 11, 12],
[ 9, 10, 6, 3, 2, 12, 19, 2],
[16, 2, 0, 6, 7, 5, 8, 8],
[18, 17, 3, 19, 5, 10, 1, 18],
[10, 7, 0, 0, 8, 17, 6, 4],
[ 2, 12, 8, 9, 6, 1, 11, 1],
[ 6, 7, 15, 15, 18, 15, 17, 15]])
and I'm trying to locate the following 1-D array in the earlier matrix.
array([6, 7, 5, 8])
It's in location (3,3) -> (3,4) -> (4,4) -> (5,4), which isn't in a straight line, rather in L-shape as in below:
array([[ False, False, False, False, False, False, False, False],
[ False, False, False, False, False, False, False, False],
[ False, False, False, False, False, False, False, False],
[ False, False, False, True, True, False, False, False],
[ False, False, False, False, True, False, False, False],
[ False, False, False, False, True, False, False, False],
[ False, False, False, False, False, False, False, False],
[ False, False, False, False, False, False, False, False]])
The elements from 1-D array aren't in a straight line always, rather can follow any order, like straight-line, L-shape, slanting line, etc. since the image chunks are transformed. This leads to a larger permutations & combinations, so need an efficient way.
So far, I tried to formalize few patterns and locating the position of the each element in the dataframe / matrix by indexing methods and checking for True per element:
np.any(a == 6, axis=0)
np.any(a == 7, axis=1)
It's taking forever to identify 1 pattern and forcing to consider some other solution, which I'm not aware of.
What would be the best way to locate the 1-D array in this multi-dimensional array in any order as mentioned earlier using NumPy and/or Pandas library? Any advice is appreciated. Thank you.
IIUC, one way is to convolve a 4 by 4 filter on the main 2-D array. The 4 by 4 squares which contains all of the 4 wanted values are the candidates for further inspection.
for i in range(len(main)-len(to_match)+1):
for j in range(len(main)-len(to_match)+1):
filter4 = main[i:i+4, j:j+4]
if np.sum(np.isin(to_match, filter4))==4:
print(np.isin(filter4, to_match))
print('------')
Suppose I have the following two matching shape tensors:
a = tensor([[ 0.0113, -0.1666, 0.5960, -0.0667], [-0.0977, -0.1984, 0.5153, 0.0420]])
selectors = tensor([[ True, True, False, False], [ True, False, True, False]])
When using torch.masked_select to find the values in a that match True indices in selectors like this:
torch.masked_select(a, selectors)
The output will be in 1D shape instead of the original 2D shape:
tensor([ 0.0113, -0.1666, -0.0977, 0.5153])
This is consistent with masked_select behavior as it is given in the documentation (torch.masked_select). However, my goal is to get a result that matches the shape of the two original tensors. I.e.:
tensor([[0.0113, -0.1666], [-0.0977, 0.5153]])
Is there a way to get this without having to loop over all the elements in the tensors and find the mask for each one? Please note that I have also looked into using torch.where, but it doesn't fit the case I have as I see it.
As #jodag pointed out, for general inputs, each row on the desired masked result might have a different number of elements, depending on how many True values there are on the same row in selectors. However, you could overcome this by allowing trailing zero padding in the result.
Basic solution:
indices = torch.masked_fill(torch.cumsum(selectors.int(), dim=1), ~selectors, 0)
masked = torch.scatter(input=torch.zeros_like(a), dim=1, index=indices, src=a)[:,1:]
Explanation:
By applying cumsum() row-wise over selectors, we compute for each unmasked element in a the target column number it should be copied to in the output tensor. Then, scatter() performs a row-wise scattering of a's elements to these computed target locations. We leave all masked elements with the index 0, so that the first element in each row of the result would contain one of the masked elements (maybe arbitrarily. we don't care which). We then ignore these un-wanted 1st values by taking the slice [:,1:]. The output resulting masked tensor has the exact same size as the input a (this is the maximum needed size, for the case where there is a row of full True values in selectors).
Usage example:
>>> a = Torch.tensor([[ 1, 2, 3, 4, 5, 6], [10, 20, 30, 40, 50, 60]])
>>> selectors = Torch.tensor([[ True, False, False, True, False, True], [False, False, True, True, False, False]])
>>> torch.cumsum(selectors.int(), dim=1)
tensor([[1, 1, 1, 2, 2, 3],
[0, 0, 1, 2, 2, 2]])
>>> indices = torch.masked_fill(torch.cumsum(selectors.int(), dim=1), ~selectors, 0)
>>> indices
tensor([[1, 0, 0, 2, 0, 3],
[0, 0, 1, 2, 0, 0]])
>>> torch.scatter(input=torch.zeros_like(a), dim=1, index=indices, src=a)
tensor([[ 5, 1, 4, 6, 0, 0],
[60, 30, 40, 0, 0, 0]])
>>> torch.scatter(input=torch.zeros_like(a), dim=1, index=indices, src=a)[:,1:]
tensor([[ 1, 4, 6, 0, 0],
[30, 40, 0, 0, 0]])
Adapting output size: Here, the length of dim=1 of the output resulting masked tensor is the max number of un-masked items in a row. For your original show-case, the output shape would be (2,2) as you desired. Note that if this number is not previously known and a is on CUDA, it would cause an additional host-device synchronization that might affect the performance.
To do so, instead of allocating input=torch.zeros_like(a) for scatter(), allocate it by a.new_zeros(size=(a.size(0), torch.max(indices).item() + 1)). The +1 is for the 1st place which is later sliced-out. The host-device synchronization would occur by accessing the result of max() to calculate the allocated output size.
Example:
>>> torch.scatter(input=a.new_zeros(size=(a.size(0), torch.max(indices).item() + 1)), dim=1, index=indices, src=a)[:,1:]
tensor([[ 1, 4, 6],
[30, 40, 0]])
Changing the padding value: If another custom default value is wanted as a padding, one could use torch.full_like(my_custom_value) rather than torch.zeros_like() when allocating the output for scatter().
I've been going over the numpy docs looking for a specific operation. The words I would use for this are "overlay" or "mask" but the numpy concepts of those words don't seem to match mine.
I want to take two arrays, one dense and one sparse and combine them thusly:
[ 1, 2, 3, 4, 5 ]
X [ N, N, 10, N, 12 ]
= [ 1, 2, 10, 4, 12 ]
where X is the operation and N is None, or Null, -1, or some other special character.
How is this achieved in numpy/python3?
You can use np.where:
# pick special value
N = -1
dns = [ 1, 2, 3, 4, 5 ]
sprs = [ N, N, 10, N, 12 ]
# this is important otherwise the comparison below
# is not done element by element
sprs = np.array(sprs)
# tada!
np.where(sprs==N,dns,sprs)
# array([ 1, 2, 10, 4, 12])
When called with three arguments m,a,b where "mixes" a and b taking elements from a where m is True and from b where it is False.
You can "fill" the masked array, with np.ma.filled(..) [numpy-doc], for example:
>>> a
array([1, 2, 3, 4, 5])
>>> b
masked_array(data=[--, --, 10, --, 12],
mask=[ True, True, False, True, False],
fill_value=999999)
>>> b.filled(a)
array([ 1, 2, 10, 4, 12])
>>> np.ma.filled(b, a)
array([ 1, 2, 10, 4, 12])
Here we thus fill the masked values from b with the corresponding values of a.
I want to compare two array(4 floating point)and print mismatched items.
I used this code:
>>> from numpy.testing import assert_allclose as np_assert_allclose
>>> x=np.array([1,2,3])
>>> y=np.array([1,0,3])
>>> np_assert_allclose(x,y, rtol=1e-4)
AssertionError:
Not equal to tolerance rtol=0.0001, atol=0
(mismatch 33.33333333333333%)
x: array([1, 2, 3])
y: array([1, 0, 3])
the problem by this code is with big array:
(mismatch 0.0015104228617559556%)
x: array([ 0.440088, 0.35994 , 0.308225, ..., 0.199546, 0.226758, 0.2312 ])
y: array([ 0.44009, 0.35994, 0.30822, ..., 0.19955, 0.22676, 0.2312 ])
I can not find what values are mismatched. how can see them ?
Just use
~np.isclose(x, y, rtol=1e-4) # array([False, True, False], dtype=bool)
e.g.
d = ~np.isclose(x, y, rtol=1e-4)
print(x[d]) # [2]
print(y[d]) # [0]
or, to get the indices
np.where(d) # (array([1]),)