I'm trying to save numpy array to csv file but there is a problem,
I use two different solution but they did not work
my numpy array looks like,
In[39]: arr[0]
Out[39]:
array([ array([[ 30, 29, 198, ..., 149, 149, 149],
[ 29, 29, 197, ..., 149, 149, 149],
[ 29, 29, 197, ..., 149, 149, 149],
...,
[ 63, 63, 96, ..., 105, 104, 104],
[ 63, 63, 96, ..., 106, 105, 105],
[ 77, 77, 217, ..., 217, 217, 217]], dtype=uint8),
list([0, 0, 0, 0, 0, 0, 0, 0, 0])], dtype=object)
Its shape is (1200, 2) numpy array and I want to save it to csv file,
with np.savetxt function
In[40]: np.savetxt("numpy_array.csv", arr, delimiter=',')
Traceback (most recent call last):
File "D:\Program files\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 1254, in savetxt
fh.write(asbytes(format % tuple(row) + newline))
TypeError: only length-1 arrays can be converted to Python scalars
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Program files\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-41-673bcc1d77a6>", line 1, in <module>
np.savetxt("numpy_array.csv", arr, delimiter=',')
File "D:\Program files\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 1258, in savetxt
% (str(X.dtype), format))
TypeError: Mismatch between array dtype ('object') and format specifier ('%.18e,%.18e')
with pandas
In[42]: df = pd.DataFrame(arr)
In[43]: df[:5]
Out[43]:
0 \
0 [[30, 29, 198, 198, 197, 197, 197, 197, 197, 1...
1 [[29, 29, 197, 197, 196, 196, 197, 197, 197, 1...
2 [[29, 29, 196, 196, 196, 196, 196, 196, 196, 1...
3 [[29, 29, 196, 196, 196, 196, 196, 196, 196, 1...
4 [[29, 29, 196, 196, 196, 196, 196, 196, 197, 1...
1
0 [0, 0, 0, 0, 0, 0, 0, 0, 0]
1 [1, 0, 0, 0, 0, 0, 0, 0, 0]
2 [1, 0, 0, 0, 0, 0, 0, 0, 0]
3 [1, 0, 0, 0, 0, 0, 0, 0, 0]
4 [1, 0, 0, 0, 0, 0, 0, 0, 0]
In[44]: df.to_csv("h.csv", index=False)
In[45]: a = pd.read_csv("h.csv", header=None,names =['input', 'output'])
In[46]: a[:5]
Out[46]:
input \
0 0
1 [[ 30 29 198 ..., 149 149 149]\r\n [ 29 29 1...
2 [[ 29 29 197 ..., 149 149 149]\r\n [ 29 29 1...
3 [[ 29 29 196 ..., 149 149 149]\r\n [ 29 29 1...
4 [[ 29 29 196 ..., 149 149 149]\r\n [ 29 29 1...
output
0 1
1 [0, 0, 0, 0, 0, 0, 0, 0, 0]
2 [1, 0, 0, 0, 0, 0, 0, 0, 0]
3 [1, 0, 0, 0, 0, 0, 0, 0, 0]
4 [1, 0, 0, 0, 0, 0, 0, 0, 0]
when I print "df[:5]", everything looks great, but after I saved it to csv then read it from csv, it looks awful, there are not commas between numbers and there are '\r\n' between list.
I want to see like "df[:5]" 's output after read csv file, how can I do it, what is the problem?
Your array is 2d, (1200, 2) with object dtype. Evidently the first column contains 2d arrays, and the 2nd column lists.
arr[0,0] is a 2d array
array([[ 30, 29, 198, ..., 149, 149, 149],
[ 29, 29, 197, ..., 149, 149, 149],
[ 29, 29, 197, ..., 149, 149, 149],
...,
[ 63, 63, 96, ..., 105, 104, 104],
[ 63, 63, 96, ..., 106, 105, 105],
[ 77, 77, 217, ..., 217, 217, 217]], dtype=uint8)
You could easily write in a csv format. For example:
In [342]: arr = np.array([[ 30, 29, 198, 149, 149, 149],
...: [ 29, 29, 197, 149, 149, 149],
...: [ 29, 29, 197, 149, 149, 149],
...: [ 63, 63, 96, 105, 104, 104],
...: [ 63, 63, 96, 106, 105, 105],
...: [ 77, 77, 217, 217, 217, 217]], dtype=np.uint8)
...:
...:
In [343]: np.savetxt('arr.txt', arr, delimiter=',', fmt='%4d')
produces a file that looks like:
In [344]: cat arr.txt
30, 29, 198, 149, 149, 149
29, 29, 197, 149, 149, 149
29, 29, 197, 149, 149, 149
63, 63, 96, 105, 104, 104
63, 63, 96, 106, 105, 105
77, 77, 217, 217, 217, 217
Read savetxt for more details on fmt.
But the full array is not compatible with the simple 2d layout of a csv file. Sure you could write something more complicated, but you couldn't load it with a csv reader like np.genfromtxt or np.loadtxt. Those expect the neat row and column layout with a well defined delimiter.
In [346]: data = np.genfromtxt('arr.txt',delimiter=',',dtype=None)
In [347]: data
Out[347]:
array([[ 30, 29, 198, 149, 149, 149],
[ 29, 29, 197, 149, 149, 149],
[ 29, 29, 197, 149, 149, 149],
[ 63, 63, 96, 105, 104, 104],
[ 63, 63, 96, 106, 105, 105],
[ 77, 77, 217, 217, 217, 217]])
The pandas df shows two columns, one with the arrays, the other with the lists. But in a column 0 appears to contain string representations of the 2d arrays, as indicated by the newline characters. Did you look at the h.csv file? Part of the reason for using csv is so people can read it, and other programs (like excel) can read it.
Make an array like your big one
In [349]: barr = np.empty((3,2), object)
In [350]: barr[:,0]=[arr,arr,arr]
In [351]: barr[:,1]=[[0,0,0] for _ in range(3)]
In [352]: barr
Out[352]:
array([[array([[ 30, 29, 198, 149, 149, 149],
[ 29, 29, 197, 149, 149, 149],
[ 29, 29, 197, 149, 149, 149],
[ 63, 63, 96, 105, 104, 104],
[ 63, 63, 96, 106, 105, 105],
[ 77, 77, 217, 217, 217, 217]], dtype=uint8),
list([0, 0, 0])],
[array([[ 30, 29, 198, 149, 149, 149],
...
[ 77, 77, 217, 217, 217, 217]], dtype=uint8),
list([0, 0, 0])]], dtype=object)
Write it %s format, the only one that will work with objects like this:
In [354]: np.savetxt('barr.txt',barr, delimiter=',',fmt='%s')
In [355]: cat barr.txt
[[ 30 29 198 149 149 149]
[ 29 29 197 149 149 149]
[ 29 29 197 149 149 149]
[ 63 63 96 105 104 104]
[ 63 63 96 106 105 105]
[ 77 77 217 217 217 217]],[0, 0, 0]
[[ 30 29 198 149 149 149]
[ 29 29 197 149 149 149]
[ 29 29 197 149 149 149]
[ 63 63 96 105 104 104]
[ 63 63 96 106 105 105]
[ 77 77 217 217 217 217]],[0, 0, 0]
[[ 30 29 198 149 149 149]
[ 29 29 197 149 149 149]
[ 29 29 197 149 149 149]
[ 63 63 96 105 104 104]
[ 63 63 96 106 105 105]
[ 77 77 217 217 217 217]],[0, 0, 0]
That is not a valid csv file. It is text, but with [] and varying line lengths, none of the standard csv file readers can handle it.
Saving that array as you did with pandas, I get:
In [364]: cat pdbarr.txt
0,1
"[[ 30 29 198 149 149 149]
[ 29 29 197 149 149 149]
[ 29 29 197 149 149 149]
[ 63 63 96 105 104 104]
[ 63 63 96 106 105 105]
[ 77 77 217 217 217 217]]","[0, 0, 0]"
"[[ 30 29 198 149 149 149]
[ 29 29 197 149 149 149]
[ 29 29 197 149 149 149]
[ 63 63 96 105 104 104]
[ 63 63 96 106 105 105]
[ 77 77 217 217 217 217]]","[0, 0, 0]"
"[[ 30 29 198 149 149 149]
[ 29 29 197 149 149 149]
[ 29 29 197 149 149 149]
[ 63 63 96 105 104 104]
[ 63 63 96 106 105 105]
[ 77 77 217 217 217 217]]","[0, 0, 0]"
Notice all the quotes - it's writing those component arrays and lists as strings. Again, not a valid csv.
Numpy itself has no 'save as csv'-function. Normally you save it through another package (like pandas or pickle).
What you see 'it looks awful' is the pandas format. Add arr = np.array(a)
and you have you numpy format again.
Related
So I have a chess board represented as an array of size 64 with top-left square being 0 and bottom-right square being 63. I have this function which gives all possible moves of King.
current_pos = i
arr = np.array([i-9, i-8, i-7, i-1, i+1, i+7, i+8, 1+9])
return arr
.
.
.
if selected position is in arr:
move king
Where i is the number of the square on which king currently is.
This works if the king is not on edges of the chessboard.
BUT if king is on the bottom-right square, that is number 63, the function gives bottom-left square that is number 56 as a valid position for a king to move.
Is there any efficient way to know that the king is going to the other edge and is not a valid move?
I'm having same problems with almost all my pieces where the function will allow piece to go on the other side of board but i figured king's movement was the simplest to ask.
A 1D list is way faster than a 2D 8x8 list so I like that you are using this approach.
The way this is handled is to use a 10x12 board where you have an extra 2 rows on bottom and top, and an extra column on the left and right:
Then in your generate move function you simple check if the square you are looking at is within the board. If it is not, you skip to the next square in your loop.
Please read more about it on https://www.chessprogramming.org/10x12_Board. It is also a great site for information about chess programming in general.
Here is one approach using table lookup.
Code
piece_offsets = {
'n': [-17, -15, -10, -6, 6, 10, 15, 17],
'b': [ -9, -7, 9, 7],
'r': [ -8, -1, 8, 1],
'q': [ -9, -8, -7, -1, 9, 8, 7, 1],
'k': [ -9, -8, -7, -1, 9, 8, 7, 1]
}
sqdist = [[0 for x in range(64)] for y in range(64)]
pseudo_legal = {
'n': [[] for y in range(64)],
'b': [[] for y in range(64)],
'r': [[] for y in range(64)],
'q': [[] for y in range(64)],
'k': [[] for y in range(64)],
}
def distance(sq1, sq2):
file1 = sq1 & 7
file2 = sq2 & 7
rank1 = sq1 >> 3
rank2 = sq2 >> 3
rank_distance = abs(rank2 - rank1)
file_distance = abs(file2 - file1)
return max(rank_distance, file_distance)
def print_board():
for i in range(64):
print(f'{i:02d} ', end='')
if (i+1)%8 == 0:
print()
def on_board(s):
return s >= 0 and s < 64
def init_board():
for sq1 in range(64):
for sq2 in range(64):
sqdist[sq1][sq2] = distance(sq1, sq2)
for pt in ['n', 'b', 'r', 'q', 'k']:
for s in range(64):
for offset in piece_offsets[pt]:
to = s + offset
if pt in ['k', 'n']:
if on_board(to) and sqdist[s][to] < 4:
pseudo_legal[pt][s].append(to)
else: # sliders
s1 = s
while True:
to1 = s1 + offset
if on_board(to1) and sqdist[s1][to1] < 4:
pseudo_legal[pt][s].append(to1)
s1 = to1
else:
break
def main():
init_board() # build sqdist and pseudo_legal_to tables
print_board()
print()
for pt in ['n', 'b', 'r', 'q', 'k']:
for s in [0, 63, 36]:
print(f'pt: {pt}, from: {s}: to: {pseudo_legal[pt][s]}')
print()
# pseudo_legal_sq = pseudo_legal['b'][61]
# print(pseudo_legal_sq)
main()
Output
00 01 02 03 04 05 06 07
08 09 10 11 12 13 14 15
16 17 18 19 20 21 22 23
24 25 26 27 28 29 30 31
32 33 34 35 36 37 38 39
40 41 42 43 44 45 46 47
48 49 50 51 52 53 54 55
56 57 58 59 60 61 62 63
pt: n, from: 0: to: [10, 17]
pt: n, from: 63: to: [46, 53]
pt: n, from: 36: to: [19, 21, 26, 30, 42, 46, 51, 53]
pt: b, from: 0: to: [9, 18, 27, 36, 45, 54, 63]
pt: b, from: 63: to: [54, 45, 36, 27, 18, 9, 0]
pt: b, from: 36: to: [27, 18, 9, 0, 29, 22, 15, 45, 54, 63, 43, 50, 57]
pt: r, from: 0: to: [8, 16, 24, 32, 40, 48, 56, 1, 2, 3, 4, 5, 6, 7]
pt: r, from: 63: to: [55, 47, 39, 31, 23, 15, 7, 62, 61, 60, 59, 58, 57, 56]
pt: r, from: 36: to: [28, 20, 12, 4, 35, 34, 33, 32, 44, 52, 60, 37, 38, 39]
pt: q, from: 0: to: [9, 18, 27, 36, 45, 54, 63, 8, 16, 24, 32, 40, 48, 56, 1, 2, 3, 4, 5, 6, 7]
pt: q, from: 63: to: [54, 45, 36, 27, 18, 9, 0, 55, 47, 39, 31, 23, 15, 7, 62, 61, 60, 59, 58, 57, 56]
pt: q, from: 36: to: [27, 18, 9, 0, 28, 20, 12, 4, 29, 22, 15, 35, 34, 33, 32, 45, 54, 63, 44, 52, 60, 43, 50, 57, 37, 38, 39]
pt: k, from: 0: to: [9, 8, 1]
pt: k, from: 63: to: [54, 55, 62]
pt: k, from: 36: to: [27, 28, 29, 35, 45, 44, 43, 37]
I'm trying to build an Electron React App that captures short videos of the screen but am unable to store videos (as part of a larger "Payload" object:
{video_buffer: xxx, time_stamp: 3456345 ...}
The videos are created in the renderer process via MediaStream and MediaRecorder API's then converted to Blob. Here is some sample code:-
let mediaRecorder = new MediaRecorder(videoStream);
mediaRecorder.start()
let chunks = []
mediaRecorder.ondataavailable = (event) => {chunks.push(event.data)}
mediaRecorder.onstop = async (event) => {
let blob = new Blob(chunks, {type: 'video/mp4'})
chunks = []
const buffer = await blob.arrayBuffer()
The buffer works when I convert to a URL and set it to the Video element src
let videoURL = URL.createObjectURL(blob)
The videos are short and occupy about 2.5MB max.
I can only send the buffer (or the URL) via IPC (not the Blob).
In the main process, I'm using Mongoose with the schema:
new payloadSchema = {video_buffer: {type: Buffer}}
but this doesn't work, presumably because buffer is an ArrayBuffer and not a Data View or TypedArray.
So I tried converting it to a Uint8Array TypedArray object:
const new_buffer = new Uint8Array(buffer)
but i get this error on saving:
video_buffer: CastError: Cast to Buffer failed for value "Uint8Array(2041510) [
[electron] 26, 69, 223, 163, 159, 66, 134, 129, 1, 66, 247, 129,
[electron] 1, 66, 242, 129, 4, 66, 243, 129, 8, 66, 130, 132,
[electron] 119, 101, 98, 109, 66, 135, 129, 4, 66, 133, 129, 2,
[electron] 24, 83, 128, 103, 1, 255, 255, 255, 255, 255, 255, 255,
[electron] 21, 73, 169, 102, 153, 42, 215, 177, 131, 15, 66, 64,
[electron] 77, 128, 134, 67, 104, 114, 111, 109, 101, 87, 65, 134,
[electron] 67, 104, 114, 111, 109, 101, 22, 84, 174, 107, 171, 174,
[electron] 169, 215, 129, 1, 115, 197, 135, 155, 112, 30, 237, 62,
[electron] 162, 245, 131, 129,
[electron] ... 2041410 more items
[electron] ]" at path "video_buffer"
I then tried to convert the Uint8Array into a hex encoded string then storing that as a buffer .
On querying MongoDB in Terminal it has stored the string:
const buff_to_hex_string = new_buffer.toString('hex')
Then save it via Mongoose as before : (Schema {type: Buffer})
This seemed to work with following output from MongoDB:
The output of the saved Payload Schema is:
video_buffer: Binary {
[electron] _bsontype: 'Binary',
[electron] sub_type: 0,
[electron] position: 9514868,
[electron] buffer: <Buffer 32 36 2c 36 39 2c 32 32 33 2c 31 36 33 2c 31 35 39 2c 36 36 2c 31 33 34 2c 31 32 39 2c 31 2c 36 36 2c 32 34 37 2c 31 32 39 2c 31 2c 36 36 2c 32 34 32 ... 9514818 more bytes>
Question: On retrieving the stored buffer object, what is the easiest way to convert it back into an ArrayBuffer so that I can play it in a video element ?
Do I need to convert backwards --> hex string --> Uint8Array --> ArrayBuffer ---> Blob ---> URL ?
I've tried lots of methods provided but none work.
Any help would be much appreciated.
Thanks.
I have a .mat file and want to convert it into a CV image format such that I can use it for a CNN model.
I am trying to obtain an RGB/ other colored image and not gray.
I tried doing the following(below) but I get a grayscale image, but when I plot the actual mat file using matplotlib it is not grayscale. Also, the .mat file has a px_spacing array apart from the image array. I am not sure how this is helpful.
def mat_to_image(mat_image):
f = loadmat(mat_image,appendmat=True)
image = np.array(f.get('I')).astype(np.float32)
mean = image.mean()
std= image.std()
print(mean, std)
hi = np.max(image)
lo = np.min(image)
image = (((image - lo)/(hi-lo))*255).astype(np.uint8)
im = Image.fromarray(image,mode='RGB')
return im
images=mat_to_image(dir/filename)
cv_img = cv2.cvtColor(np.array(images), cv2.COLOR_GRAY2RGB)
Normally plotting the .mat file fetches a non-grayscale(RGB image)
imgplot= plt.imshow(loadmat(img,appendmat=True).get('I'))
plt.show()
Here is how the mat file looks after print(loadmat('filename'))
{'__header__': b'MATLAB 5.0 MAT-file, Platform: PCWIN64, Created on: Mon Sep 9 11:32:54 2019',
'__version__': '1.0',
'__globals__': [],
'I': array([
[ 81, 75, 74, 75, -11, 14, 49, 37, 29, -24, -183, -349, -581, -740],
[ 51, 33, 67, 36, 1, 42, 30, 49, 47, 42, 14, -85, -465, -727],
[ 23, 31, 36, 20, 54, 70, 44, 56, 56, 79, 62, 19, -204, -595],
[ 7, 12, 36, 47, 59, 68, 74, 56, 59, 100, 74, 34, -3, -353],
[ 23, 19, 51, 87, 86, 79, 91, 76, 96, 95, 52, 51, 74, -141],
[ 18, 51, 54, 97, 93, 94, 98, 83, 119, 71, 36, 69, 50, -16],
[ -10, 5, 53, 92, 69, 87, 103, 114, 118, 77, 51, 68, 30, 0],
[ -24, 11, 74, 80, 49, 68, 106, 129, 107, 63, 57, 70, 39, -1],
[ -45, 43, 83, 69, 43, 64, 98, 108, 90, 35, 27, 55, 31, -13],
[ -9, 32, 83, 78, 66, 106, 89, 85, 58, 43, 31, 39, 28, 7],
[ 45, 35, 76, 45, 51, 84, 55, 66, 49, 41, 39, 28, 13, -7],
[ 85, 67, 61, 45, 69, 53, 23, 32, 31, -12, -34, -182, -376, -425],
[ 136, 93, 71, 54, 30, 39, 17, -21, -29, -43, -101, -514, -792, -816]
], dtype=int16),
'px_spacing': array([[0.78125]])}
I have a numpy array like:
[[ 90 -51 -90]
[ 67 47 -43]
[ -27 27 100]
[ 55 -1 -50]
[ -95 -55 69]]
Is there a way I can, in numpy, remove rows with opposite elements, like [ 90 -51 -90] or [ -27 27 100], without using a For loop?
Thanks !
Here's one approach -
In [20]: a
Out[20]:
array([[ 90, -51, -90],
[ 67, 47, -43],
[-27, 27, 100],
[ 55, -1, -50],
[-95, -55, 69]])
In [21]: p = np.take_along_axis(a,np.abs(a).argsort(1),axis=1)
In [22]: a[~((p[:,:-1]+p[:,1:])==0).any(1)]
Out[22]:
array([[ 67, 47, -43],
[ 55, -1, -50],
[-95, -55, 69]])
One-liner with broadcasting -
In [44]: a[~(((a[:,None,:]+a[:,:,None])==0).any(2)).any(1)]
Out[44]:
array([[ 67, 47, -43],
[ 55, -1, -50],
[-95, -55, 69]])
I have a set of known destination (background) colors and known output colors for some blending function for some constant source color, but I don't know the source color or the function.
As you can see in the image below, there doesn't seem to be full correlation between hues. Destinations on the left are sorted in order of hue, output colors are clearly not. This suggests that the R,G,B channels are not handled independently.
I know that all these destination-output pairs are using the same function and source color, so how would I go about solving? Using a color palette is not suitable as the given set is only a close approximation.
The blend function will be implemented programmatically, but the solution from these unknowns can be either manual or programmatic for my purposes.
Image of given set, destination on the left, output on the right
RGB values:
Source Output
255, 145, 139 237, 139, 131
236, 197, 189 218, 133, 149
217, 195, 42 197, 131, 93
89, 175, 141 58, 135, 90
98, 167, 195 106, 115, 143
111, 169, 193 58, 128, 91
72, 132, 161 76, 115, 139
121, 130, 196 120, 125, 158
99, 104, 167 100, 124, 149
132, 133, 147 131, 126, 135
99, 100, 116 109, 128, 127
66, 63, 79 85, 131, 121