List values lost in Python Multiprocessing - python-3.x

Question

I would like to parallelise a series of tasks operating on a list of NetworkX graphs. For parallelisation I use the Manager and Process objects from the Multiprocessing module. In the minimal example below there is only one process calculating the adjacency matrix of each NetworkX graph. The set of graphs is stored in a list Gt. Each particular graph from this set is called Gk. Similarly, the adjacency matrices are stored in a list called At, while each particular matrix corresponding to a graph Gk is called Ak. I use keyword arguments to pass the global list of adjacency matrices At and an index k to a function adj_mtrx(). My problem is that I cannot obtain the calculated adjacency values in the main body of the program. At[k] are all zeroes. If possible, would you please take a look at the minimal example below and direct me to my mistake?
#!/usr/bin/env python3
import networkx as nx
import numpy as np
import random as rnd
from multiprocessing import Manager, Process
# Generates random graph
def gen_rnd_graph(nv, ne):
# Create random list of sources
Vsrc = [rnd.randint(0,nv-1) for iter in range(ne)]
# Create random list of sinks
Vsnk = [rnd.randint(0,nv-1) for iter in range(ne)]
# Create random list of edge weights
U = [rnd.random() for iter in range(ne)]
# Create list of tuples {Vsrc, Vsnk, U}
T = list(zip(Vsrc,Vsnk,U))
# Create graph
G = nx.Graph()
# Create list of vertices
V = list(range(nv))
# Add nodes to graph
G.add_nodes_from(V)
# Add edges between random vertices with random edge weights
G.add_weighted_edges_from(T)
return G
# Generates random time-varying graph
def gen_time_graph(nv, ne, ng):
# Initialise list of graphs
l = []
for i in range(ng):
gi = gen_rnd_graph(nv, ne)
l.append(gi)
return l
# Computes adjacency matrix for snaphot of time-varying graph
def adj_mtrx(Gk, **kwargs):
At = kwargs.get("At", None)
k = kwargs.get("k", None)
print("in adj_mtrx id(At):", id(At))
# no. of vertices
n = Gk.number_of_nodes()
# adjacency matrix
Ak = np.zeros([n,n])
# for each vertex
for i in range(n):
for j in range(n):
if Gk.has_edge(i,j):
Ak[i,j] = 1
print("func Ak[{0:d},{1:d}]: {2:f}".format(i, j, Ak[i,j]))
# Store new At[k] values
if At != None and k != None:
At[k][i,j] = Ak[i,j]
if At[k][i,j] > 0.0:
print("func At[{0:d}][{1:d},{2:d}]: {3:f}".format(k, i, j, At[k][i,j]))
return Ak
def main():
with Manager() as manager:
#-----------------------------------------------------------------------
# Specify constants
#-----------------------------------------------------------------------
NV = 10 # no. of vertices
NE = 15 # no. of edges
NG = 3 # no. of snapshot graphs
#-----------------------------------------------------------------------
# Generate random time-varying graph
#-----------------------------------------------------------------------
Gt = manager.list()
Gt = gen_time_graph(NV, NE, NG)
# Snapshot index
k = 0
# Initialise list of temporal adjacency matrics
At = manager.list()
At =[np.zeros([NV,NV])]*NG
print("create id(At):", id(At))
# for each snapshot graph
for Gk in Gt:
print("k: {0:d}".format(k))
processes = []
# Temporal adjacency matrix
print("pre adj_mtrx id(At):", id(At))
p1 = Process( target=adj_mtrx, args=(Gk,), kwargs={"At": At, "k": k} )
print("post adj_mtrx id(At):", id(At))
p1.start()
processes.append(p1)
# Wait for process 1
p1.join()
# #test
[m,n] = np.shape(At[k])
for i in range(m):
for j in range(n):
if At[k][i,j] > 0.0:
print("body At[{0:d}][{1:d},{2:d}]: {3:f}".format(k, i, j, At[k][i,j]))
k += 1
if __name__ == '__main__':
main()

Answer 1

multiprocessing supports two types of communication channel between processes: multiprocessing.Queue and multiprocessing.Pipe.
In your function 'adj_mtrx' you are modifying 'At' variable, which in your case is a deep copy of the 'At' variable you are passing to Process(…) in the 'main' function. All modifications to the 'At' variable will be local to the 'adj_mtrx'.

Answer 2

You can use the Multiprocessing.Pool class.
Instantiate a pool with the number of processes you want to run in parallel and use its map function to iterate over the list of graphs and calculate the adj matrix for each graph using your function.
In fact networkx provides it's own function for calculating adj matrix (nx.to_numpy_array)
see the code below :
import networkx as nx
import numpy as np
import random as rnd
import multiprocessing
from multiprocessing import Pool
# Generates random graph
def gen_rnd_graph(nv, ne):
G = nx.gnm_random_graph(nv, ne, seed=None, directed=False)
for s, t in G.edges():
G[s][t]['weight'] = rnd.random()
return G
# Generates random time-varying graph
def gen_time_graph(nv, ne, ng):
# Initialise list of graphs
l = []
for i in range(ng):
gi = gen_rnd_graph(nv, ne)
l.append(gi)
return l
# Computes adjacency matrix for snaphot of time-varying graph
def adj_mtrx(Gk):
Ak = nx.to_numpy_array(Gk, weight=1) # weight parameter make sure adj is 1 instead of actual weight
return Ak
def main():
num_of_processes = multiprocessing.cpu_count() // 2
print('num_of_process={}'.format(num_of_processes))
# -----------------------------------------------------------------------
# Specify constants
# -----------------------------------------------------------------------
NV = 10 # no. of vertices
NE = 15 # no. of edges
NG = 3 # no. of snapshot graphs
# -----------------------------------------------------------------------
# Generate random time-varying graph
# -----------------------------------------------------------------------
Gt = gen_time_graph(NV, NE, NG)
with Pool(num_of_processes) as p:
At = p.map(adj_mtrx, Gt)
for k in range(NG):
print(Gt[k].edges())
print(At[k])
print('==========')
if __name__ == '__main__':
main()
I've also used networkx random graph generation method to make the code more compact, it differs slightly from your implementation as it wont have self edges and it guarantees that NE edges will be produced (no duplicate edges)