Why cannot join the processes after the work is done? - python-3.x

Question

I have a Queue, a Manager and a Lock inside a class. The class has a run function that starts 6 Process and they all wait to an exit_flag to be true in order to end their jobs. However, I cannot "end" the jobs because when I call the join methods on each job, it blocks it. The code looks as follows:
from multiprocessing import Process, Lock, Queue, Manager
class MyClass():
def __init__(self):
self.q = Queue(maxsize=50)
self.lock = Lock()
manager = Manager()
self.manager = manager.dict()
def fill_queue(self,idx):
while not self.exit():
#do something
result,result_type= self.perform_extraction()
if result_type not in self.manager():
self.manager[result_type] = []
while self.q.full() and not self.exit():
sleep(10)
if self.exit():
print('Exit filler')
break
self.lock.acquire()
self.q.put((result,result_type))
self.lock.release()
else:
print(f'queue filler {idx} ended')
def empty_queue(self,idx):
while not self.exit():
if self.q.emtpy():
continue
self.lock.aquire()
result, result_type = self.q.get()
self.lock.release()
result,id = self.perform_test(queue_value)
if result >=0 and result not in self.manager[result_type]:
self.manager[id] += [(result, result_type)]
self.insert_to_database(result, result_type) --> this inserts the value into a sqlite3 ddbb
else:
print(f'worker {idx} ended')
def run(self,n_workers):
jobs = []
for _ in range(2):
p = Process(target = self.fill_queue, args=(_,))
jobs.append(p)
for _ in range(n_workers):
p = Process(target = self.empty_queue, args=(_,))
jobs.append(p)
for job in jobs:
job.start()
for idx,job in enumerate(jobs):
print(f'joining job {idx}')
job.join()
if not job.is_alive():
print(f'closing job {idx}')
job.close()
else:
print(f'job {idx} still alive')
if __name__ == '__main__':
mc = MyClass()
mc.run(n_workers = 4)
print('RUN ENDED!')
The manager is used to communicate between processes and when the criteria is met and I have X amount of elements in the manager, the self.exit() function returns True.
When I run this code, it gets stucked printing joining job 0 and it stays there forever, I dont know why. If I add some timeout and set it to job.join(5) (5 arbitary, no real reason), it prints:
joining job 0
job 0 still alive
joining job 1
job 1 still alive
joining job 2
closing job 2
joining job 3
closing job 3
joining job 4
closing job 4
joining job 5
closing job 5
RUN ENDED!
And the code does not finish. I also tried to do job.terminate() if the job is still alive and this throwed an error telling me that some leaked folders could not be found. Does this means that I have some zombie processes?
Why this is happening? What I'm doing wrong?
EDIT: Added some logic on the interaction with manager(). I'm using the manager to add a couple of types of results and append all results of same type to a list, so the dict structure is something like {result_type:[result_values]} and the reason to use manager is to avoid storing duplicate results and check when I the algorithm mets the exit criteria.
Now, the exit criteria function looks like this:
def exit(self):
for v in self.manager.values():
if len(v) < 10:
return False
return True
So, all processes ends when I have 10 items on each list of each type. There are 3 possible amount of different types, so once each type is added as a manager key, it only needs to be filled and when all 3 have at least 10 values (there could be more), then all the job should end.
EDIT 2: Added some printing information to fill_queue and empty_queue functions. What is printed is:
worker 0 ended
worker 1 ended
queue filler 0 ended
worker 2 ended
queue filler 1 ended
joining job 0 --> *
worker 3 ended
so this usually (like always) prints before all workers print the "ended" statement, but it never joins the first job. This is actually stuck at the first try of calling join method at the for idx,job in enumerate(jobs): cycle.