I want to use np.random.choice inside a multiprocessing pool, but I get the IndexError: list index out of range. I don't get any error when I use the choice function inside a for loop (in series then, not parallel). Any ideas on how to overcome this? This is just a small part of my routine but would surely improve a lot its speed. My code is like below. I declare X before anything else in the routine so it works as a global variable, but it's dynamically populated inside the main. I also noticed that there is some conflict with multiprocessing and the for loop. Any ideas on how to implement this?
from multiprocessing import Pool
from numpy.random import choice
import numpy as np
K = 10
X = []
def function(k):
global X
np.random.RandomState(k)
aux = [i for i in np.arange(K) if i != k]
a,b,c = choice(aux,3,replace=False)
x = X[a]+0.7*(X[b]-X[c])
return x
if __name__ == '__main__':
X = np.arange(K)
for n in range(K):
pool = Pool(K)
w = pool.map(function,np.arange(K))
pool.close()
print(w)
Child processes do not share the memory space of parent processes. Since you populate X inside the if __name__ ... clause, the child processes only have access to the X defined at the top module, i.e X = []
A quick solution would be to shift the line X = np.arange(K) outside the clause like below:
from multiprocessing import Pool
from numpy.random import choice
import numpy as np
K = 10
X = []
X = np.arange(K)
def function(k):
global X
np.random.RandomState(k)
aux = [i for i in np.arange(K) if i != k]
a, b, c = choice(aux, 3, replace=False)
x = X[a] + 0.7 * (X[b] - X[c])
return k, x
if __name__ == '__main__':
pool = Pool(10)
w = pool.map(function, np.arange(K))
pool.close()
print(w)
Output
[(0, 10.899999999999999), (1, 9.4), (2, 5.7), (3, 7.4), (4, 1.1000000000000005), (5, -1.0999999999999996), (6, 5.6), (7, 3.8), (8, 5.5), (9, -4.8999999999999995)]
If you do not want to initialize X for all child processes (memory constraints?), you can use a manager to store X that can be shared to processes without having to copy it for every child. To pass more than one argument to the child processes, you will also have to use pool.starmap instead. Lastly, delete that global X, it is not doing anything useful since global is only used if you are planning to modify a global variable from a local scope.
from multiprocessing import Pool, Manager
from numpy.random import choice
import numpy as np
K = 10
def function(X, k):
np.random.RandomState(k)
aux = [i for i in np.arange(K) if i != k]
a, b, c = choice(aux, 3, replace=False)
x = X[a] + 0.7 * (X[b] - X[c])
return k, x
if __name__ == '__main__':
m = Manager()
X = m.list(np.arange(K))
pool = Pool(10)
args = [(X, val) for val in np.arange(K)]
w = pool.starmap(function, args)
pool.close()
print(w)
Output
[(0, -1.5999999999999996), (1, 7.3), (2, 4.9), (3, 1.9000000000000004), (4, 5.5), (5, -1.0999999999999996), (6, 4.800000000000001), (7, 7.3), (8, 0.10000000000000053), (9, 4.7)]
I am trying to adapt an application I found here, I imagined that I just had to add an axis. The problem is that the script seems like it is stuck. Can someone tell me what I did wrong and how I can use A* with a 3d matrix (i j k)?
this is the portion of the A* function I changed
for new_position in [(0, 0, 0), (0, 0, 1), (0, 1, 0), (0, 1, 1), (1, 0, 0), (1, 0, 1), (1, 1, 0), (1, 1, 1)]: # Adjacent squares, (-1, -1), (-1, 1), (1, -1), (1, 1)]: # Adjacent squares removed to ignor diagonal movement
# Get node position
node_position = (current_node.position[0] + new_position[0], current_node.position[1] + new_position[1], current_node.position[2] + new_position[2])
# Make sure within range
if node_position[0] > (len(maze) - 1) or node_position[0] < 0 or node_position[1] > (len(maze[len(maze)-1]) -1) or node_position[1] < 0 or node_position[2] > (len(maze) - 1) or node_position[2] < 0 or node_position[2] > (len(maze[len(maze)-1]) -1):
continue
# Make sure walkable terrain
if maze[node_position[0]][node_position[1]][node_position[2]] != 0:
continue
it was originally:
for new_position in [(0, -1), (0, 1), (-1, 0), (1, 0), (-1, -1), (-1, 1), (1, -1), (1, 1)]: # Adjacent squares
# Get node position
node_position = (current_node.position[0] + new_position[0], current_node.position[1] + new_position[1])
# Make sure within range
if node_position[0] > (len(maze) - 1) or node_position[0] < 0 or node_position[1] > (len(maze[len(maze)-1]) -1) or node_position[1] < 0:
continue
# Make sure walkable terrain
if maze[node_position[0]][node_position[1]] != 0:
continue
this is the whole script with my alterations:
import numpy as np
class Node():
"""A node class for A* Pathfinding"""
def __init__(self, parent=None, position=None):
self.parent = parent
self.position = position
self.g = 0
self.h = 0
self.f = 0
def __eq__(self, other):
return self.position == other.position
def astar(maze, start, end):
"""Returns a list of tuples as a path from the given start to the given end in the given maze"""
# Create start and end node
start_node = Node(None, start)
start_node.g = start_node.h = start_node.f = 0
end_node = Node(None, end)
end_node.g = end_node.h = end_node.f = 0
# Initialize both open and closed list
open_list = []
closed_list = []
# Add the start node
open_list.append(start_node)
# Loop until you find the end
while len(open_list) > 0:
# Get the current node
current_node = open_list[0]
current_index = 0
for index, item in enumerate(open_list):
if item.f < current_node.f:
current_node = item
current_index = index
# Pop current off open list, add to closed list
open_list.pop(current_index)
closed_list.append(current_node)
# Found the goal
if current_node == end_node:
path = []
current = current_node
while current is not None:
path.append(current.position)
current = current.parent
return path[::-1] # Return reversed path
# Generate children
children = []
for new_position in [(0, 0, 0), (0, 0, 1), (0, 1, 0), (0, 1, 1), (1, 0, 0), (1, 0, 1), (1, 1, 0), (1, 1, 1)]: # Adjacent squares, (-1, -1), (-1, 1), (1, -1), (1, 1)]: # Adjacent squares removed to ignor diagonal movement
# Get node position
node_position = (current_node.position[0] + new_position[0], current_node.position[1] + new_position[1], current_node.position[2] + new_position[2])
# Make sure within range
if node_position[0] > (len(maze) - 1) or node_position[0] < 0 or node_position[1] > (len(maze[len(maze)-1]) -1) or node_position[1] < 0 or node_position[2] > (len(maze) - 1) or node_position[2] < 0 or node_position[2] > (len(maze[len(maze)-1]) -1):
continue
# Make sure walkable terrain
if maze[node_position[0]][node_position[1]][node_position[2]] != 0:
continue
# Create new node
new_node = Node(current_node, node_position)
# Append
children.append(new_node)
# Loop through children
for child in children:
# Child is on the closed list
for closed_child in closed_list:
if child == closed_child:
continue
# Create the f, g, and h values
child.g = current_node.g + 1
child.h = ((child.position[0] - end_node.position[0]) ** 2) + ((child.position[1] - end_node.position[1]) ** 2)+((child.position[2] - end_node.position[2]) ** 2)
child.f = child.g + child.h
# Child is already in the open list
for open_node in open_list:
if child == open_node and child.g > open_node.g:
continue
# Add the child to the open list
open_list.append(child)
def main():
maze = np.zeros((12,12,12))
start = (10, 9, 9)
end = (1, 1, 1)
path = astar(maze, start, end)
print(path)
if __name__ == '__main__':
main()
A* can work with any number of dimensions; it's a graph traversal algorithm, and no matter how many dimensions your problem space has, connecting one position to another still produces a graph.
You have two problems with generating new nodes, however.
You included (0, 0, 0) in the list, so no change. You keep putting the current position back into the queue for consideration. That’s just busy work, because the current position is already in the closed list.
You never subtract from any of your coordinates, you only add. So your x, y and z values can only ever go up. If reaching your goal requires a path around an obstacle, then you have a problem here because all your version can ever do is move in one direction along any given axis.
In a 3 by 3 by 3 3D matrix, with the current position in the middle, there are 3 times 3 times 3 minus 1 == 26 positions you can reach with a single step. Your code reaches just 7 of those, plus one that stays put.
If you extract your tuples in the for new_position in [...] list into a separate variable and add some newlines, and re-arrange them a bit to group them by how many 1s you have in the tuple, you get the following definition. I renamed this to deltas, because it's not a new position, it's the change relative to the old position, or delta. I re-arranged your tuples to make it easier to group them logically:
deltas = [
(0, 0, 0), # no change,
(0, 0, 1), (0, 1, 0), (1, 0, 0), # add to a single axis
(0, 1, 1), (1, 0, 1), (1, 1, 0), # add to two axes
(1, 1, 1) # move in all 3 directions at once.
]
for delta in deltas:
# ...
You want to drop the first one ((0, 0, 0)), and you need to add -1 versions of the other variants. You need 26 different tuples, but writing those by hand gets to be cumbersome, really fast. You can use itertools.product() to generate these, instead:
from itertools import product
# combinations of -1, 0 and 1, filtering out the (0, 0, 0) case:
deltas = [d for d in product((-1, 0, 1), repeat=3) if any(d)]
Now, your comment next to the loop says:
# Adjacent squares removed to ignor diagonal movement
It's not entirely clear what you mean by this, because your original list of deltas includes diagonals ((0, 1, 1) moves both in the y and z directions, and you have tuples combining movement in the x plus y and the x plus z axes), and even one that moves diagonal by incrementing all 3 axes. If you only wanted to move up, down, left, right, forward, and backward, you want to restrict movement to a single axis at a time, and deltas should be:
# movement without diagonals
deltas = [
(1, 0, 0), (-1, 0, 0), # only x
(0, 1, 0), (0, -1, 0), # only y
(0, 0, 1), (0, 0, -1), # only z
]
Personally, I'd move the whole business of generating new positions to a separate method on the Node class:
def possible_moves(self, map):
x, y, z = self.x, self.y, self.z
for dx, dy, dz in DELTAS:
newx, newy, newz = x + dx, y + dy, z + dz
try:
if maze[newx][newy][newz] != 0:
yield Node(self, (newx, newy, newz))
except IndexError:
# not inside the maze anymore, ignore
pass
This method assumes that there is a DELTAS global variable that defines the possible moves, and is a generator; each time yield is reached a new Node() instance is returned to whatever is using this as an iterator (like a for loop would).
Then just use that method instead of the children = [] list you filled with the for new_position in ...: loop, so directly in the part that uses the original children list:
# Loop through children
for child in current_node.possible_moves(map):
# Child is on the closed list
for closed_child in closed_list:
if child == closed_child:
continue
# etc.
I have this snippet that 'generates' the centers of figures in my GUI... The probem is that is too slow because the solution iterates too much. So I'm wondering how to implement a generator to take exacle value I need.
from pprint import pprint
def center(i, j):
s = 50
for k, v1 in enumerate(range(int(s / 2), s * 8, s)):
for n, v2 in enumerate(range(int(s / 2), s * 8, s)):
if (i, j) == (k, n):
return (v1, v2)
pprint(center(0,0)) # -> (25, 25)
pprint(center(6,3)) # -> (325, 175)
So I made this generator:
def centerGenerator():
s = 50
for k in range(int(s / 2), s * 8, s):
for n in range(int(s / 2), s * 8, s):
yield k, n
def center(i, j):
for c in centerGenerator():
pass # how to take only c(i,j) from the Generator?
pprint(center(0,0)) # -> (25, 25)
pprint(center(6,3)) # -> (325, 175)
But idk how to implements it. Any thoughts?
I have been trying to simulate a random walk using the code below
import random
def random_walk(n):
""" Return coordiantes after 'n' block random walk"""
x, y = 0, 0
# y = 0
for i in range(n):
(dx, dy) = random.choice([(0, 1), (0, -1), (1, 0), (-1, 0)])
x = x+dx
y = y+dy
return(x, y)
for i in range(25):
walk = random_walk(10)
print(walk, "Distance from origin:",
abs(walk[0]) + abs(walk[1]))
I am always getting output as 1.No matter how much i increase number of walks or
number of blocks walked.I am not able to figure out what am i doing wrong
Just an indenting problem in the first for loop.
import random
def random_walk(n):
""" Return coordiantes after 'n' block random walk"""
x, y = 0, 0
# y = 0
for i in range(n):
(dx, dy) = random.choice([(0, 1), (0, -1), (1, 0), (-1, 0)])
x = x+dx
y = y+dy
return(x, y)
for i in range(25):
walk = random_walk(10)
print(walk, "Distance from origin:", abs(walk[0]) + abs(walk[1]))
I have N 3d arrays, with fixed sizes (n1, n2, n3)
Now I want to combine them into 4d array and as the result to have array of the dimension:
(N, n1, n2, n3)
How should I do it? Dstack is doing the same thing but only for 2D arrays making them 3D. Trying to make it for 3D array gives wrong results (n1, n2, n3*N)
EDIT 1: I need to do it in the loop, so each iteration in the loop gives new (n1, n2, n3) array (3d array) which I should to put into 4D array increasing it's first dimension: 1st iteration would give (1, n1, n2, n3)
then 2nd iteration would give (2, n1, n2, n3) and so on.
dstack only exists for backwards compatibility. Use numpy.stack instead, which is more generic and future proof:
import numpy as np
a = np.ones((4, 5, 6))
b = np.zeros((4, 5, 6))
c = np.stack([a, b])
print(c.shape) # (2, 4, 5, 6)
d = np.stack([a, b], axis=2) # You can stack along any axis
print(d.shape) # (4, 5, 2, 6)
Loopy example:
result = []
for i in range(n):
x = np.random.randn(n1, n2, n3)
result.append(x)
result = np.stack(result)
Alternative (slower):
result = np.empty((0, n1, n2, n3))
for i in range(n):
x = np.random.randn(n1, n2, n3)
result = np.concatenate([result, x[np.newaxis]])
You can also preallocate if you know N in advance:
result = np.empty((n, n1, n2, n3))
for i in range(n):
x = np.random.randn(n1, n2, n3)
result[i] = x