In the help of numpy.broadcst-array, an idiom is introduced.
However, the idiom give exactly the same output as original command.
Waht is the meaning of "getting contiguous copies instead of non-contiguous views."?
https://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast_arrays.html
x = np.array([[1,2,3]])
y = np.array([[1],[2],[3]])
np.broadcast_arrays(x, y)
[array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]), array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])]
Here is a useful idiom for getting contiguous copies instead of non-contiguous views.
[np.array(a) for a in np.broadcast_arrays(x, y)]
[array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]), array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])]
To understand the difference try writing into the new arrays:
Let's begin with the contiguous copies.
>>> import numpy as np
>>> x = np.array([[1,2,3]])
>>> y = np.array([[1],[2],[3]])
>>>
>>> xc, yc = [np.array(a) for a in np.broadcast_arrays(x, y)]
>>> xc
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
We can modify an element and nothing unexpected will happen.
>>> xc[0, 0] = 0
>>> xc
array([[0, 2, 3],
[1, 2, 3],
[1, 2, 3]])
>>> x
array([[1, 2, 3]])
Now, let's try the same with the broadcasted arrays:
>>> xb, yb = np.broadcast_arrays(x, y)
>>> xb
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
Although we only write to the top left element ...
>>> xb[0, 0] = 0
... the entire left column will change ...
>>> xb
array([[0, 2, 3],
[0, 2, 3],
[0, 2, 3]])
... and also the input array.
>>> x
array([[0, 2, 3]])
It means that broadcast_arrays function doesn't create entirely new object. It creates views from original arrays which means the elements of it's results have memory addresses as those arrays which may or may not be contiguous. But when you create a list you're creating new copies within a list which guarantees that its items are stored contiguous in memory.
You can check this like following:
arr = np.broadcast_arrays(x, y)
In [144]: arr
Out[144]:
[array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]), array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])]
In [145]: x
Out[145]: array([[1, 2, 3]])
In [146]: arr[0][0] = 0
In [147]: arr
Out[147]:
[array([[0, 0, 0],
[0, 0, 0],
[0, 0, 0]]), array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])]
In [148]: x
Out[148]: array([[0, 0, 0]])
As you can see, changing the arr's elements is changing both its elements and the original x array.
Related
I have a huge numpy array with 15413 rows and 70 columns. The first column represents the weight (so if the first element in a row is n, that row should be repeated n times.
I tried with numpy.repeat but I don’t think it’s giving me the correct answer because np.sum(chain[:,0]) is not equal to len(ACT_chain)
ACT_chain = []
for i in range(len(chain[:,0])):
chain_row = chain[i]
ACT_chain.append(chain_row)
if int(chain[:,0][i]) > 1:
chain_row = np.repeat(chain[i], chain[:, 0][i], axis=0)
ACT_chain.append(chain_row)
For example, running this code with this sample array
chain = np.array([[1, 5, 3], [2, 2, 1], [3, 0, 1]])
gives
[array([1, 5, 3]), array([2, 2, 1]), array([[2, 2, 1],
[2, 2, 1]]), array([3, 0, 1]), array([[3, 0, 1],
[3, 0, 1],
[3, 0, 1]])]
but the output I expect is
array([[1, 5, 3], [2, 2, 1], [2, 2, 1], [3, 0, 1], [3, 0, 1], [3, 0, 1]])
You can use repeat here, without the iteration.
np.repeat(chain, chain[:, 0], axis=0)
array([[1, 5, 3],
[2, 2, 1],
[2, 2, 1],
[3, 0, 1],
[3, 0, 1],
[3, 0, 1]])
I solved the typeError by converting the whole array to int
chain = chain.astype(int)
For example, if I have a tensor A = [[1,1,1], [2,2,2], [3,3,3]], and B = [1,2,3]. How do I get C = [[1,1,1], [2,2,2], [2,2,2], [3,3,3], [3,3,3], [3,3,3]], and doing this batch-wise?
My current element-wise solution btw (takes forever...):
def get_char_context(valid_embeds, words_lens):
chars_contexts = []
for ve, wl in zip(valid_embeds, words_lens):
for idx, (e, l) in enumerate(zip(ve, wl)):
if idx ==0:
chars_context = e.view(1,-1).repeat(l, 1)
else:
chars_context = torch.cat((chars_context, e.view(1,-1).repeat(l, 1)),0)
chars_contexts.append(chars_context)
return chars_contexts
I'm doing this to add bert word embedding to a char level seq2seq task...
Use this:
import torch
# A is your tensor
B = torch.tensor([1, 2, 3])
C = A.repeat_interleave(B, dim = 0)
EDIT:
The above works fine if A is a single 2D tensor. To repeat all (2D) tensors in a batch in the same manner, this is a simple workaround:
A = torch.tensor([[[1, 1, 1], [2, 2, 2], [3, 3, 3]],
[[1, 2, 3], [4, 5, 6], [2,2,2]]]) # A has 2 tensors each of shape (3, 3)
B = torch.tensor([1, 2, 3]) # Rep. of each row of every tensor in the batch
A1 = A.reshape(1, -1, A.shape[2]).squeeze()
B1 = B.repeat(A.shape[0])
C = A1.repeat_interleave(B1, dim = 0).reshape(A.shape[0], -1, A.shape[2])
C is:
tensor([[[1, 1, 1],
[2, 2, 2],
[2, 2, 2],
[3, 3, 3],
[3, 3, 3],
[3, 3, 3]],
[[1, 2, 3],
[4, 5, 6],
[4, 5, 6],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2]]])
As you can see each inside tensor in the batch is repeated in the same manner.
Tensor a:
tensor([[1, 2, 3],
[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
Tensor b:
tensor([4,4,4,4])
Question 1:
How to merge two tensors and get result c:
tensor([[1, 2, 3, 4],
[1, 2, 3, 4],
[1, 2, 3, 4],
[1, 2, 3, 4]])
Question 2: How to divide tensor c and get original a and b.
Question 1: Merge two tensors -
torch.cat((a, b.unsqueeze(1)), 1)
>>> tensor([[1, 2, 3, 4],
[1, 2, 3, 4],
[1, 2, 3, 4],
[1, 2, 3, 4]])
First, we use torch.unsqueeze to add single dim in b tensor to match a dim to be concanate. Then use torch.cat Concatenates tensors a and b.
Question 2:
a = c[:][:,:-1]
a
>>> tensor([[1, 2, 3],
[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
b = c[:][:,-1:].squeeze(1)
b
>>> tensor([4, 4, 4, 4])
You have to slightly modify tensor b:
a = torch.tensor([[1, 2, 3],
[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
b = torch.tensor([4,4,4,4])
b = b.reshape(1, 4)
Then you get your "joined" tensor:
c = torch.cat((a, torch.t(b)), 1)
And backward:
a1 = c[:,:-1]
b1 = torch.t(c[:,-1:])
Can anyone help me. This is what i want to do.
x = [[1,2,3,4,5],[6,7,8,9,10]]
y= [0,1]
desired output = [
[[1,2,3,4,5],[0,1]],
[[6,7,8,9,10],[0,1]]
]
I try putting it in a for loop
>>> x = [[1,2,3,4,5],[6,7,8,9,10]]
>>> for value in x:
... a = []
... a += ([x,y])
... print(a)
...
[[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]], [0, 1]]
[[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]], [0, 1]]
I also tried doing this
>>> for value in x:
... a = []
... a += ([x,y])
... print(a)
...
[[1, 2, 3, 4, 5], [0, 1]]
[[1, 2, 3, 4, 5], [0, 1]]
[[1, 2, 3, 4, 5], [0, 1]]
[[1, 2, 3, 4, 5], [0, 1]]
[[1, 2, 3, 4, 5], [0, 1]]
Thank you for helping. I need it for putting label on my data for neural networks.
You can use a list comprehension, and iterate over each sublist in x. Since you're inserting y into different sublists, you might want to insert a copy of the list, not the original.
[[i, y[:]] for i in x]
Or,
[[i, y.copy()] for i in x]
[[[1, 2, 3, 4, 5], [0, 1]], [[6, 7, 8, 9, 10], [0, 1]]]
The copy is done as a safety precaution. To understand why, consider an example,
z = [[i, y] for i in x] # inserting y (reference copy)
y[0] = 12345
print(z)
[[[1, 2, 3, 4, 5], [12345, 1]], [[6, 7, 8, 9, 10], [12345, 1]]] # oops
Modifying the original y or the y in any other sublist will reflect changes across all sublists. You can prevent that by inserting a copy instead, which is what I've done at the top.
Try this:
for i in range(len(x)):
z[i] = [x[i],y];
Suppose I have the following numpy.array:
In[]: x
Out[]:
array([[1, 2, 3, 4, 5],
[5, 2, 4, 1, 5],
[6, 7, 2, 5, 1]], dtype=int16)
In[]: y
Out[]:
array([[-3, -4],
[-4, -1]], dtype=int16)
I want to replace a sub array of x by y and tried the following:
In[]: x[[0,2]][:,[1,3]]= y
Ideally, I wanted this to happen:
In[]: x
Out[]:
array([[1, -3, 3, -4, 5],
[5, 2, 4, 1, 5],
[6, -4, 2, -1, 1]], dtype=int16)
The assignment line doesn't give me any error, but when I check the output of x
In[]: x
I find that x hasn't changed, i.e. the assignment didn't happen.
How can I make that assignment? Why did the assignment didn't happen?
The the "fancy indexing" x[[0,2]][:,[1,3]] returns a copy of the data. Indexing with slices returns a view. The assignment does happen, but to a copy (actually a copy of a copy of...) of x.
Here we see that the indexing returns a copy:
>>> x[[0,2]]
array([[1, 2, 3, 4, 5],
[6, 7, 2, 5, 1]], dtype=int16)
>>> x[[0,2]].base is x
False
>>> x[[0,2]][:, [1, 3]].base is x
False
>>>
Now you can use fancy indexing to set array values, but not when you nest the indexing.
You can use np.ix_ to generate the indices and perform the assignment:
>>> x[np.ix_([0, 2], [1, 3])]
array([[2, 4],
[7, 5]], dtype=int16)
>>> np.ix_([0, 2], [1, 3])
(array([[0],
[2]]), array([[1, 3]]))
>>> x[np.ix_([0, 2], [1, 3])] = y
>>> x
array([[ 1, -3, 3, -4, 5],
[ 5, 2, 4, 1, 5],
[ 6, -4, 2, -1, 1]], dtype=int16)
>>>
You can also make it work with broadcasted fancy indexing (if that's even the term) but it's not pretty
>>> x[[0, 2], np.array([1, 3])[..., None]] = y
>>> x
array([[ 1, -3, 3, -4, 5],
[ 5, 2, 4, 1, 5],
[ 6, -4, 2, -1, 1]], dtype=int16)
By the way, there is some interesting discussion at the moment on the NumPy Discussion mailing list on better support for "orthogonal" indexing so this may become easier in the future.