I am trying to create the data for my CNN network. Desired input shape for my CNN network is 36 X 36 X 2. This means, I have two different 2D matrices with the size of 36 X 36.
Using these two matrices, I want to get a output of 36 X 36 X 2.
I have tried above code.
arr1 = np.random.rand(36,36)
arr2 = np.random.rand(36,36)
res = np.stack((arr1, arr2), axis=2)
Output should look like as matrices in the image:
I want my input shape as described in the picture. first matrix should be arr1, second matrix should be arr2 and both matrix should be placed one after other.
However, I am quite confused with the result I got from res. It shows me the shape (36, 36, 2), but when I print the res, then I am not able to see my first matrix and second matrix properly. I see elements from my first matrix arr1 is inside other matrix.
I am not sure if this process gives me a correct output or I am doing anything wrong.
I'm converting some odd code over to be Numba compatible using parallel=True. It has a problematic array assignment that I can't quite figure out how to rewrite in a way numba can handle. I try to decode what the error means, but I get quite lost. The only thing clear is it does not like the line: Averaging_price_3D[leg, :, expired_loc] = last_non_expired_values.T The error is pretty long, included for reference here:
TypingError: No implementation of function Function(<built-in function setitem>) found for signature:
setitem(array(float64, 3d, C), Tuple(int64, slice<a:b>, array(int64, 1d, C)), array(float64, 2d, F))
There are 16 candidate implementations:
- Of which 14 did not match due to:
Overload of function 'setitem': File: <numerous>: Line N/A.
With argument(s): '(array(float64, 3d, C), Tuple(int64, slice<a:b>, array(int64, 1d, C)), array(float64, 2d, F))':
No match.
- Of which 2 did not match due to:
Overload in function 'SetItemBuffer.generic': File: numba\core\typing\arraydecl.py: Line 176.
With argument(s): '(array(float64, 3d, C), Tuple(int64, slice<a:b>, array(int64, 1d, C)), array(float64, 2d, F))':
Rejected as the implementation raised a specific error:
NumbaNotImplementedError: only one advanced index supported
And here is a short code segment to reproduce the error:
import numpy as np
import numba as nb
#nb.jit(nopython=True, parallel=True, nogil=True)
def main(Averaging_price_3D, expired_loc, last_non_expired_values):
for leg in range(Averaging_price_3D.shape[0]):
# line below causes the numba error:
Averaging_price_3D[leg, :, expired_loc] = last_non_expired_values.T
return Averaging_price_3D
if __name__ == "__main__":
Averaging_price_3D=np.random.rand(2,8192,11)*100 # shape (2,8192,11) 3D array float64
expired_loc=np.arange(4,10).astype(np.int64) # shape (6,) 1D array int64
last_non_expired_values = Averaging_price_3D[1,:,0:expired_loc.shape[0]].copy() # shape (8192,6) 2D array float64
result = main(Averaging_price_3D, expired_loc, last_non_expired_values)
Now the best I can interpret this error is that "numba doesn't know how to set values in a 3D matrix using array indexing with values from a 2D array." But I searched online quite a bit and can't find another way to accomplish the same thing, without numba crashing on it.
In other cases like this I resorted to flattening the arrays with a .reshape(-1) before indexing, but I'm having issues with figuring out how to do that in this specific case (that was easy with a 3D array indexed with another 3D array, as they both would flatten in the same order)... Any help is appreciated!
Well interesting enough, I looked at the indexes passed to the 3D array (since the error said "only one advanced index supported," I chose to examine my indexing):
3Darray[int, :, 1Darray]
Seeing numba is quite picky, I tried rewriting it a little bit, so that a 1D array wasn't used as an index (apparently, this is an "advanced index", so use an int index instead). Reading numba errors and solutions, they tend to add loops, so I tried that here. So instead of passing a 1D array as an index, I looped over the elements of the 1D array:
import numpy as np
import numba as nb
#nb.jit(cache=True, nopython=True, parallel=True, nogil=True)
def main(Averaging_price_3D, expired_loc, last_non_expired_values):
for leg in nb.prange(Averaging_price_3D.shape[0]):
# change the indexing for numba to get rid of a 1D array index
for i in nb.prange(expired_loc.shape[0]):
# now we assign values 3Darray[int,:,int] = 1Darray
Averaging_price_3D[leg, :, expired_loc[i]] = last_non_expired_values[:,i].T
return Averaging_price_3D
if __name__ == "__main__":
Averaging_price_3D=np.random.rand(2,8192,11)*100 # shape (2,8192,11) 3D array float64
expired_loc=np.arange(4,10).astype(np.int64) # shape (6,) 1D array int64
last_non_expired_values = Averaging_price_3D[1,:,0:expired_loc.shape[0]] # shape (8192,6) 2D array float64
result = main(Averaging_price_3D, expired_loc, last_non_expired_values)
Now it works no problem at all. So it appears to me if you want to access elements from a 3D array with numba, you should do it with either ints or :. It appears to not like 1D array indexing, so replace it with a loop and it should run in parallel.
I have to create a sparse matrix in python using a function similar to the Matlab function
S = sparse(i,j,v,m,n) where i, j, and v such that S(i(k),j(k)) = v(k) and the size of S is specified as m-by-n.
I have chosen the function scipy.sparse.csr_matrix to do this. My code is something like the following.
arg_shape=np.array([ndof,ndof])
K = csr_matrix((arg_data,(arg_x,arg_y)),shape=arg_shape)
here ndof=786432 and arg_data, arg_x, arg_y are numpy arrays and all of the same shape.i.e. (150994944,).
when I run this code, I get the following error:
ValueError: row index exceeds matrix dimensions
In Matlab the code looks like this and works:
K = sparse(arg_x,arg_y,arg_data, ndof, ndof);
Could anyone please help me with the following points:
1). Is scipy.sparse.csr_matrix a good replacement for the Matlab spare function.
2). If yes, what is the mistake I am making in the code?
Thank you very much.
I have an array as below.
X1=np.array([[0,0],[0,0]])
X1[:,0].shape gives me (2,).
How do I convert X1[:,0] to a shape of (2,1).
thanks for asking. what you have is a two by two matrix, so turning one part of the array to a two dimensional array will cause an error. I think you should create a new array from the sub array and then reshape it. you can do it like this new_x = X[:,0]
new_x.reshape(2,1). I hope this works
I'm currently trying to convert a list of values into a PyTorch tensor and am facing some difficulties.
The exact code that's causing the error is:
input_tensor = torch.cuda.FloatTensor(data)
Here, data is a list with two elements: The first element is another list of NumPy arrays and the second element is a list of tuples. The sizes of both lists differ, and I believe this is causing the following error:
*** ValueError: expected sequence of length x at dim 2 (got y)
Usually y is larger than x. I've tried playing around with an IPython terminal to see what's wrong, and it appears that trying to convert data of this format directly into PyTorch tensors doesn't work. Taking each individual element of the data list and converting those into tensors works, though.
Does anybody know why this doesn't work and perhaps also be able to provide some feedback on how to achieve my original goal? Thanks in advance.
Let's say that the first sublist of data contains n 1D arrays, each of size m, and the second sublist contains k tuples, each of size p.
When calling torch.FloatTensor(data) each sublist is converted to a 2D tensor, of shape (n, m) and of shape (k, p) respectively; then they are stack together to form a 3D tensor. This is possible only if n=k and m=p -- think of a 3D tensor as a cuboid.
This is quite obvious I think, so I guess you have m = p and want to create a 2D tensor of shape (n+k, m) by simply concatenating the two sublists:
torch.FloatTensor(np.concatenate(data))