I want to get the result_1 and result_2 arrays with the following code:
import multiprocessing as mp
import numpy as np
result_1=[]
result_2=[]
a=np.random.rand(10,10)
b=np.random.rand(7,7)
def inv_1(x):
result_1.append(np.linalg.inv(x))
def inv_2(y):
result_2.append(np.linalg.inv(y))
if __name__ == "__main__":
p1 = mp.Process(target=inv_1, args=(a),)
p2 = mp.Process(target=inv_2, args=(b),)
p1.start()
p2.start()
p1.join()
p2.join()
print(result_1, result_2)
However, when I run the code, I get the following output:
[] []
How can I solve this problem?
Unlike threads, you can't share arbitrary variables between processes. To do what you're trying to do, you can create shared lists using a multiprocessing.Manager object, e.g:
import multiprocessing as mp
import numpy as np
a=np.random.rand(10,10)
b=np.random.rand(7,7)
def inv_1(x, target):
target.append(np.linalg.inv(x))
def inv_2(y, target):
target.append(np.linalg.inv(y))
if __name__ == "__main__":
mgr = mp.Manager()
result_1 = mgr.list()
result_2 = mgr.list()
q = mp.Queue()
p1 = mp.Process(target=inv_1, args=(a, result_1),)
p2 = mp.Process(target=inv_2, args=(b, result_2),)
p1.start()
p2.start()
p1.join()
p2.join()
print('RESULT 1:', result_1)
print('RESULT 2:', result_2)
This does what you're trying to do, although it's not clear to me why you're doing it this way -- both result_1 and result_2 only have a single value (because you're just appending an item to an empty list), so it's not clear why you need a list in the first place.
More broadly, you might want to redesign your code so that it doesn't rely on shared variables. A common solution is to use a queue to pass data from your workers back to the main process.
Related
The following code is of a shop that has 5 items and three customers each demanding one item.
import multiprocessing as mp
class Shop:
def __init__(self, stock=5):
self.stock = stock
def get_item(self, l, x):
l.acquire()
if self.stock >= x:
self.stock -= x
print(f"{self.stock} = remaining")
l.release()
if __name__ == "__main__":
l = mp.Lock()
obj = Shop()
p1 = mp.Process(target=obj.get_item, args=(l, 1))
p2 = mp.Process(target=obj.get_item, args=(l, 1))
p3 = mp.Process(target=obj.get_item, args=(l, 1))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock)
The output that I got is as follows
4 = remaining
4 = remaining
4 = remaining
Final: 5
However, since I'm using Lock I was expecting it to be
4 = remaining
3 = remaining
2 = remaining
Final: 2
Question: How to achieve the above output just with Locks(and no process communication i.e without Pipe/Queue)?
The reason this code is not working as you expect it to is because multiprocessing does not share its state with child processes. This means that each of the process you start, p1, p2 and p3, get a copy of the object of class Shop. It is NOT the same object. There are two ways you can fix this, share the instance attribute stock with the processes, or share the whole object itself. The second way is probably better for your larger use case if the shop object holds other data that needs to be shared between the processes to.
Method 1:
To share the value of only the stock instance variable, you can use multiprocessing.Value. The way to create shared integers using this and also access their value is here:
shared_int = multiprocessing.Value('i', 5)
print(f'Value is {shared_int.value}') # 5
Adapting to your use case, the code will then become:
import multiprocessing
class Shop:
def __init__(self, stock=5):
self.stock = multiprocessing.Value('i', stock)
def get_item(self, l, x):
l.acquire()
if self.stock.value >= x:
self.stock.value -= x
print(f"{self.stock.value} = remaining")
l.release()
if __name__ == "__main__":
l = multiprocessing.Lock()
obj = Shop()
p1 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p2 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p3 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock.value)
Output
4 = remaining
3 = remaining
2 = remaining
Final: 2
Method 2
Sharing the whole complex object is a more involved process. I had recently answered a similar question in detail about sharing complex objects (like the object of class Shop in this case), which also covered the reasoning behind the code provided below. I recommend that you give it a read since it explains the logic behind the code provided at the bottom in greater detail. The only major difference for this use-case is that you will want to use multiprocess, a fork of multiprocessing, instead of multiprocessing. This library works identically to the built-in multiprocessing except for the fact that it offers better pickling support which we will need.
Basically, you will want to use multiprocessing.Managers to share the state, and a suitable proxy to access the state. The ObjProxy provided in below code is one such proxy which shares the namespace as well as instance methods (apart from protected/private attributes). Once you have these, you just need to create the objects of class Shop using the manager and the proxy. This is done using the newly added create method of class Shop. This is a class constructor and all objects of Shop should be created using this method only rather than directly calling the constructor. Full code:
import multiprocess
from multiprocess import Manager, Process
from multiprocess.managers import NamespaceProxy, BaseManager
import types
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
def wrapper(*args, **kwargs):
return self._callmethod(name, args, kwargs)
return wrapper
return result
class Shop:
def __init__(self, stock=5):
self.stock = stock
#classmethod
def create(cls, *args, **kwargs):
# Register class
class_str = cls.__name__
BaseManager.register(class_str, cls, ObjProxy, exposed=tuple(dir(cls)))
# Start a manager process
manager = BaseManager()
manager.start()
# Create and return this proxy instance. Using this proxy allows sharing of state between processes.
inst = eval("manager.{}(*args, **kwargs)".format(class_str))
return inst
def get_item(self, l, x):
with l:
if self.stock >= x:
self.stock -= x
print(f"{self.stock} = remaining")
def k(self, l, n):
pass
if __name__ == "__main__":
manager = Manager()
l = manager.Lock()
obj = Shop.create()
p1 = Process(target=obj.get_item, args=(l, 1, ))
p2 = Process(target=obj.get_item, args=(l, 1, ))
p3 = Process(target=obj.get_item, args=(l, 1, ))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock)
Output
4 = remaining
3 = remaining
2 = remaining
Final: 2
Note : Explanation for these 2 lines:
manager = Manager()
l = manager.Lock()
The reason why we didn't need to create a manager (and subsequently a proxy) for the lock before in your example is outlined here. The reason why it does not work with the above code without creating a proxy is because we are no longer creating the processes in the main process, and the lock does not exist in the current processes memory space (since creating a manager for our complex object to share its state spawned its own server process)
I am using the following code to process some pictures for my ML project and I would like to parallelize it.
import multiprocessing as mp
import concurrent.futures
def track_ids(seq):
'''The func is so big I can not put it here'''
ood = {}
for i in seq:
# I load around 500 images and process them
ood[i] = some Value
return ood
seqs = []
for seq in range(1, 10):# len(seqs)+1):
seq = txt+str(seq)
seqs.append(seq)
# serial call of the function
track_ids(seq)
#parallel call of the function
with concurrent.futures.ProcessPoolExecutor(max_workers=mp.cpu_count()) as ex:
ood_id = ex.map(track_ids, seqs)
if I run the code serially it takes 3.0 minutes but for parallel with concurrent, it takes 3.5 minutes.
can someone please explain why is that? and present a way to solve the problem.
btw, I have 12 cores.
Thanks
Here's a brief example of how one might go about profiling multiprocessing code vs serial execution:
from multiprocessing import Pool
from cProfile import Profile
from pstats import Stats
import concurrent.futures
def track_ids(seq):
'''The func is so big I can not put it here'''
ood = {}
for i in seq:
# I load around 500 images and process them
ood[i] = some Value
return ood
def profile_seq():
p = Profile() #one and only profiler instance
p.enable()
seqs = []
for seq in range(1, 10):# len(seqs)+1):
seq = txt+str(seq)
seqs.append(seq)
# serial call of the function
track_ids(seq)
p.disable()
return Stats(p), seqs
def track_ids_pr(seq):
p = Profile() #profile the child tasks
p.enable()
retval = track_ids(seq)
p.disable()
return (Stats(p, stream="dummy"), retval)
def profile_parallel():
p = Profile() #profile stuff in the main process
p.enable()
with concurrent.futures.ProcessPoolExecutor(max_workers=mp.cpu_count()) as ex:
retvals = ex.map(track_ids_pr, seqs)
p.disable()
s = Stats(p)
out = []
for ret in retvals:
s.add(ret[0])
out.append(ret[1])
return s, out
if __name__ == "__main__":
stat, retval = profile_parallel()
stat.print_stats()
EDIT: Unfortunately I found out that pstat.Stats objects cannot be used normally with multiprocessing.Queue because it is not pickleable (which is needed for the operation of concurrent.futures). Evidently it normally will store a reference to a file for the purpose of writing statistics to that file, and if none is given, it will by default grab a reference to sys.stdout. We don't actually need that reference however until we actually want to print out the statistics, so we can just give it a temporary value to prevent the pickle error, and then restore an appropriate value later. The following example should be copy-paste-able and run just fine rather than the pseudocode-ish example above.
from multiprocessing import Queue, Process
from cProfile import Profile
from pstats import Stats
import sys
def isprime(x):
for d in range(2, int(x**.5)):
if x % d == 0:
return False
return True
def foo(retq):
p = Profile()
p.enable()
primes = []
max_n = 2**20
for n in range(3, max_n):
if isprime(n):
primes.append(n)
p.disable()
retq.put(Stats(p, stream="dummy")) #Dirty hack: set `stream` to something picklable then override later
if __name__ == "__main__":
q = Queue()
p1 = Process(target=foo, args=(q,))
p1.start()
p2 = Process(target=foo, args=(q,))
p2.start()
s1 = q.get()
s1.stream = sys.stdout #restore original file
s2 = q.get()
# s2.stream #if we are just adding this `Stats` object to another the `stream` just gets thrown away anyway.
s1.add(s2) #add up the stats from both child processes.
s1.print_stats() #s1.stream gets used here, but not before. If you provide a file to write to instead of sys.stdout, it will write to that file)
p1.join()
p2.join()
I want to solve multiple cplex models simultaneously using python multiprocessing. I understand that the basic example of multiprocessing in python is something like:
from multiprocessing import Process
def func1():
'''some code'''
def func2():
'''some code'''
if __name__=='__main__':
p1 = Process(target = func1)
p1.start()
p2 = Process(target = func2)
p2.start()
p1.join()
p2.join()
The structure of my script is like:
Model1(args**):
'''cplex model written with docplex'''
return model
Model2(args**):
'''cplex model written with docplex'''
return model
Generate_pool1(args**):
cpx = mdl.get_cplex()
cpx.parameters.parallel.set(1)
cpx.parameters.threads.set(5)
cpx.parameters.emphasis.mip.set(4)
cpx.parameters.simplex.tolerances.markowitz.set(0.999)
cpx.parameters.simplex.tolerances.optimality.set(1e-9)
cpx.parameters.simplex.tolerances.feasibility.set(1e-9)
cpx.parameters.mip.pool.intensity.set(4)
cpx.parameters.mip.pool.absgap.set(1e75)
cpx.parameters.mip.pool.relgap.set(1e75)
cpx.populatelim=50
numsol = cpx.solution.pool.get_num()
return numsol
Generate_pool2(args**):
cpx = mdl.get_cplex()
cpx.parameters.parallel.set(1)
cpx.parameters.threads.set(5)
cpx.parameters.emphasis.mip.set(4)
cpx.parameters.simplex.tolerances.markowitz.set(0.999)
cpx.parameters.simplex.tolerances.optimality.set(1e-9)
cpx.parameters.simplex.tolerances.feasibility.set(1e-9)
cpx.parameters.mip.pool.intensity.set(4)
cpx.parameters.mip.pool.absgap.set(1e75)
cpx.parameters.mip.pool.relgap.set(1e75)
cpx.populatelim=50
numsol = cpx.solution.pool.get_num()
return numsol
main():
for i in range(len(data)-1):
m1=Model1(data[i])
m2=Model2(data[i+1])
p1 = Process(target = Generate_pool1,(m1,i),)
p1.start()
p2 = Process(target = Generate_pool2,(m2,i+1),)
p2.start()
p1.join()
p2.join()
When I run this code the cplex part doesn't work. The console keeps running but nothing happens and the process does not finish by itself, I have to keyboard interrupt it everytime. My engine has 32 virtual cores and it's runnig on spyder -windows 10.
with docplex you may find an example in https://www.linkedin.com/pulse/making-optimization-simple-python-alex-fleischer/
https://github.com/PhilippeCouronne/docplex_contribs/blob/master/docplex_contribs/src/zoomontecarlo2.py
which uses
https://github.com/PhilippeCouronne/docplex_contribs/blob/master/docplex_contribs/src/process_pool.py
that relies on
import concurrent.futures
from concurrent.futures import ProcessPoolExecutor
This example relies on docplex
I am following the principles laid down in this post to safely output the results which will eventually be written to a file. Unfortunately, the code only print 1 and 2, and not 3 to 6.
import os
import argparse
import pandas as pd
import multiprocessing
from multiprocessing import Process, Queue
from time import sleep
def feed(queue, parlist):
for par in parlist:
queue.put(par)
print("Queue size", queue.qsize())
def calc(queueIn, queueOut):
while True:
try:
par=queueIn.get(block=False)
res=doCalculation(par)
queueOut.put((res))
queueIn.task_done()
except:
break
def doCalculation(par):
return par
def write(queue):
while True:
try:
par=queue.get(block=False)
print("response:",par)
except:
break
if __name__ == "__main__":
nthreads = 2
workerQueue = Queue()
writerQueue = Queue()
considerperiod=[1,2,3,4,5,6]
feedProc = Process(target=feed, args=(workerQueue, considerperiod))
calcProc = [Process(target=calc, args=(workerQueue, writerQueue)) for i in range(nthreads)]
writProc = Process(target=write, args=(writerQueue,))
feedProc.start()
feedProc.join()
for p in calcProc:
p.start()
for p in calcProc:
p.join()
writProc.start()
writProc.join()
On running the code it prints,
$ python3 tst.py
Queue size 6
response: 1
response: 2
Also, is it possible to ensure that the write function always outputs 1,2,3,4,5,6 i.e. in the same order in which the data is fed into the feed queue?
The error is somehow with the task_done() call. If you remove that one, then it works, don't ask me why (IMO that's a bug). But the way it works then is that the queueIn.get(block=False) call throws an exception because the queue is empty. This might be just enough for your use case, a better way though would be to use sentinels (as suggested in the multiprocessing docs, see last example). Here's a little rewrite so your program uses sentinels:
import os
import argparse
import multiprocessing
from multiprocessing import Process, Queue
from time import sleep
def feed(queue, parlist, nthreads):
for par in parlist:
queue.put(par)
for i in range(nthreads):
queue.put(None)
print("Queue size", queue.qsize())
def calc(queueIn, queueOut):
while True:
par=queueIn.get()
if par is None:
break
res=doCalculation(par)
queueOut.put((res))
def doCalculation(par):
return par
def write(queue):
while not queue.empty():
par=queue.get()
print("response:",par)
if __name__ == "__main__":
nthreads = 2
workerQueue = Queue()
writerQueue = Queue()
considerperiod=[1,2,3,4,5,6]
feedProc = Process(target=feed, args=(workerQueue, considerperiod, nthreads))
calcProc = [Process(target=calc, args=(workerQueue, writerQueue)) for i in range(nthreads)]
writProc = Process(target=write, args=(writerQueue,))
feedProc.start()
feedProc.join()
for p in calcProc:
p.start()
for p in calcProc:
p.join()
writProc.start()
writProc.join()
A few things to note:
the sentinel is putting a None into the queue. Note that you need one sentinel for every worker process.
for the write function you don't need to do the sentinel handling as there's only one process and you don't need to handle concurrency (if you would do the empty() and then get() thingie in your calc function you would run into a problem if e.g. there's only one item left in the queue and both workers check empty() at the same time and then both want to do get() and then one of them is locked forever)
you don't need to put feed and write into processes, just put them into your main function as you don't want to run it in parallel anyway.
how can I have the same order in output as in input? [...] I guess multiprocessing.map can do this
Yes map keeps the order. Rewriting your program into something simpler (as you don't need the workerQueue and writerQueue and adding random sleeps to prove that the output is still in order:
from multiprocessing import Pool
import time
import random
def calc(val):
time.sleep(random.random())
return val
if __name__ == "__main__":
considerperiod=[1,2,3,4,5,6]
with Pool(processes=2) as pool:
print(pool.map(calc, considerperiod))
i want to send parameters between two function, but it doesn't work properly.
from multiprocessing import Process
a=False
def func1():
a=True
def func2():
if a:
print("hi")
if __name__ == '__main__':
p1 = Process(target=func1)
p1.start()
p2 = Process(target=func2)
p2.start()
p1.join()
p2.join()
Any suggestion would be appreciated.