Why Multiprocessing's Lock not blocking the object use by other processes? - python-3.x

Question

The following code is of a shop that has 5 items and three customers each demanding one item.
import multiprocessing as mp
class Shop:
def __init__(self, stock=5):
self.stock = stock
def get_item(self, l, x):
l.acquire()
if self.stock >= x:
self.stock -= x
print(f"{self.stock} = remaining")
l.release()
if __name__ == "__main__":
l = mp.Lock()
obj = Shop()
p1 = mp.Process(target=obj.get_item, args=(l, 1))
p2 = mp.Process(target=obj.get_item, args=(l, 1))
p3 = mp.Process(target=obj.get_item, args=(l, 1))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock)
The output that I got is as follows
4 = remaining
4 = remaining
4 = remaining
Final: 5
However, since I'm using Lock I was expecting it to be
4 = remaining
3 = remaining
2 = remaining
Final: 2
Question: How to achieve the above output just with Locks(and no process communication i.e without Pipe/Queue)?

Answer 1

The reason this code is not working as you expect it to is because multiprocessing does not share its state with child processes. This means that each of the process you start, p1, p2 and p3, get a copy of the object of class Shop. It is NOT the same object. There are two ways you can fix this, share the instance attribute stock with the processes, or share the whole object itself. The second way is probably better for your larger use case if the shop object holds other data that needs to be shared between the processes to.
Method 1:
To share the value of only the stock instance variable, you can use multiprocessing.Value. The way to create shared integers using this and also access their value is here:
shared_int = multiprocessing.Value('i', 5)
print(f'Value is {shared_int.value}') # 5
Adapting to your use case, the code will then become:
import multiprocessing
class Shop:
def __init__(self, stock=5):
self.stock = multiprocessing.Value('i', stock)
def get_item(self, l, x):
l.acquire()
if self.stock.value >= x:
self.stock.value -= x
print(f"{self.stock.value} = remaining")
l.release()
if __name__ == "__main__":
l = multiprocessing.Lock()
obj = Shop()
p1 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p2 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p3 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock.value)
Output
4 = remaining
3 = remaining
2 = remaining
Final: 2
Method 2
Sharing the whole complex object is a more involved process. I had recently answered a similar question in detail about sharing complex objects (like the object of class Shop in this case), which also covered the reasoning behind the code provided below. I recommend that you give it a read since it explains the logic behind the code provided at the bottom in greater detail. The only major difference for this use-case is that you will want to use multiprocess, a fork of multiprocessing, instead of multiprocessing. This library works identically to the built-in multiprocessing except for the fact that it offers better pickling support which we will need.
Basically, you will want to use multiprocessing.Managers to share the state, and a suitable proxy to access the state. The ObjProxy provided in below code is one such proxy which shares the namespace as well as instance methods (apart from protected/private attributes). Once you have these, you just need to create the objects of class Shop using the manager and the proxy. This is done using the newly added create method of class Shop. This is a class constructor and all objects of Shop should be created using this method only rather than directly calling the constructor. Full code:
import multiprocess
from multiprocess import Manager, Process
from multiprocess.managers import NamespaceProxy, BaseManager
import types
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
def wrapper(*args, **kwargs):
return self._callmethod(name, args, kwargs)
return wrapper
return result
class Shop:
def __init__(self, stock=5):
self.stock = stock
#classmethod
def create(cls, *args, **kwargs):
# Register class
class_str = cls.__name__
BaseManager.register(class_str, cls, ObjProxy, exposed=tuple(dir(cls)))
# Start a manager process
manager = BaseManager()
manager.start()
# Create and return this proxy instance. Using this proxy allows sharing of state between processes.
inst = eval("manager.{}(*args, **kwargs)".format(class_str))
return inst
def get_item(self, l, x):
with l:
if self.stock >= x:
self.stock -= x
print(f"{self.stock} = remaining")
def k(self, l, n):
pass
if __name__ == "__main__":
manager = Manager()
l = manager.Lock()
obj = Shop.create()
p1 = Process(target=obj.get_item, args=(l, 1, ))
p2 = Process(target=obj.get_item, args=(l, 1, ))
p3 = Process(target=obj.get_item, args=(l, 1, ))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock)
Output
4 = remaining
3 = remaining
2 = remaining
Final: 2
Note : Explanation for these 2 lines:
manager = Manager()
l = manager.Lock()
The reason why we didn't need to create a manager (and subsequently a proxy) for the lock before in your example is outlined here. The reason why it does not work with the above code without creating a proxy is because we are no longer creating the processes in the main process, and the lock does not exist in the current processes memory space (since creating a manager for our complex object to share its state spawned its own server process)