Show the running executors identifications in ProcessPoolExecutor - python-3.x

Question

This is the problem I am having. I wont share code because of condfidentiality but instead I will provide some dummy example.
Assume that we have a class as follows:
class SayHello:
def __init__(self, name, id):
self.name=name
self.id=id
#public func
def doSomething(self, arg1, arg2 ):
DoAHugeTaskWithArgument
Lets say now that in an other modules we have this:
class CallOperations:
def __init__(self):
self.dummydict={1: {"james":20, "peter":30, "victor":40, "john":45, "ali":21, "tom":41, "hector":37}, 2:{"james":23, "peter":31, "victor":44, "john":46, "ali":23, "tom":44, "hector":35} }
def runProcessors(self):
#runprocess
for _, v in self.dummydict.items():
Instances = [SayHello(g,b) for g ,b in v.items()]
with ProcessPoolExecutor(max_workers=2) as executor:
future = [executor.submit(ins.doSomething, 2, 1235) for ins in Instances]
So the problem starts here. I want to know what instances are running doSomething() funtion in their respective process. I want to set a variable = 1 when the function of that instance is running in the process and set it to zero when it is completed.
Each instance has its own name and id. Is there way to find out the name of the running instance in the process?
This problem is making me very confused and can not find a proper solution.
Thank you alot.

Answer 1

If I understand your question correctly, you want to know when an instance of SayHello is executing and when it is not. You can set a variable (1 or 0) by using a Manager - but the usefulness of this can be debated. You might want to use a lock instead.
I had to tweak your code a bit but this is a running example. It picks one of your tasks as the one to monitor in the while loop. It is a dummy loop that never exits but you'll get the idea. It will keep polling the variable of one of your instances and you can see it change when that task is running, and then revert back to zero.
from time import sleep
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import Manager
class SayHello:
def __init__(self, name, id):
self.name=name
self.id=id
self.status = Manager().Value("i",0)
#public func
def doSomething(self, arg1, arg2 ):
self.status.value = 1
sleep(5)
self.status.value = 0
class CallOperations:
def __init__(self):
self.dummydict={1: {"james":20, "peter":30, "victor":40, "john":45, "ali":21, "tom":41, "hector":37}, 2:{"james":23, "peter":31, "victor":44, "john":46, "ali":23, "tom":44, "hector":35} }
def runProcessors(self):
#runprocess
for _, v in self.dummydict.items():
Instances = [SayHello(g,b) for g ,b in v.items()]
f = Instances[3]
executor = ProcessPoolExecutor(max_workers=2)
future = [executor.submit(ins.doSomething, 2, 1235) for ins in Instances]
while True:
print(f.status.value)
# Insert break condition here
sleep(0.5)
executor.shutdown()
foo = CallOperations()
foo.runProcessors()
The problem with this is that it can lead to a race condition depending on what you do in your main program. If you want to do any operations on the instance when it is in passive state, it might progress to active just after you check the variable but before you have completed your actions in the main program.
Locks come to rescue here, as you can also create a shared lock Manager().Lock(). If your DoSomething() tries to acquire the lock and your main process does the same when operating on a passive instance, you avoid this problem. Of course your main program could then block the executor from processing if it reserves locks for lengthy operations, as then your two workers would be stuck waiting on locks if the execution processed to those instances where locks are being held by the main program. This case would not be suitable for parallel processing implemented using executors.
EDIT: if you are only interested in the running status, you can check the Future.running() status of your future objects, in this case items in your future array.