Multiprocessing prime number check exhibits weird behavior - python-3.x

Question

I have the following parallelized code that checks if a number is a prime number.
import math
from multiprocessing import Pool, Manager
import time
from itertools import product
SERIAL_CHECK_CUTOFF = 21
CHECK_EVERY = 1000
FLAG_CLEAR = b'0'
FLAG_SET = b'1'
print("CHECK_EVERY", CHECK_EVERY)
def create_range(from_i, to_i, nbr_processes):
if from_i == to_i:
return from_i
else:
nbr_ranges = []
count = from_i
while(count < to_i + 1):
nbr_ranges.append(count)
count+=1
k, m = divmod(len(nbr_ranges), nbr_processes)
subranges = list((nbr_ranges[i*k+min(i, m):(i+1)*k+min(i+1, m)] for i in range(nbr_processes)))
subranges = [arr[::len(arr) - 1] if len(arr) > 1 else arr for arr in subranges]
return subranges
def check_prime_in_range(n_from_i_to_i, value):
(n, _range) = n_from_i_to_i
if len(_range) > 1:
(from_i, to_i) = _range
else:
return True
if n % 2 == 0:
return False
# check at every 1000 iterations.
# At every check, see if value.value has been set to FLAG_SET
# If so, exit the search.
# If in the search a process finds the factor, set the flag and exit the process.
# NOTE: check_every flag is suboptimal
check_every = CHECK_EVERY
for i in range(from_i, math.floor(to_i), 2):
check_every = -1
if not check_every:
if value.value == FLAG_SET:
return False
check_every = CHECK_EVERY
if n % i == 0:
value.value = FLAG_SET
return False
return True
def check_prime(n, nbr_processes, value):
# serial check to quickly check for small factors. if none are found, then a
# parallel search is started
from_i = 3
to_i = SERIAL_CHECK_CUTOFF
value.value = FLAG_CLEAR
if not check_prime_in_range((n, (from_i, to_i)), value):
print("Found small non-prime factor")
return False
# continue to check for larger factors in parallel
from_i = to_i
to_i = int(math.sqrt(n)) + 1
ranges_to_check = create_range(from_i, to_i, nbr_processes)
ranges_to_check = zip(len(ranges_to_check) * [n], ranges_to_check)
with Pool() as pool:
args = ((arg, value) for arg in product(list(ranges_to_check)))
# print(list(args)) # comment out this line and the code breaks
results = pool.map(check_prime_in_range, args)
if False in results:
return False
return True
if __name__ == "__main__":
start = time.time()
nbr_processes = 4
manager = Manager()
value = manager.Value(b'c', FLAG_CLEAR) # 1-byte character
n = 98_823_199_699_242_79
isprime = check_prime(n, nbr_processes, value)
end = time.time()
if isprime:
print(f"{n} is a prime number")
else:
print(f"{n} is not a prime number")
print(f"Duration: {end - start}s")
check_prime() finds a range of factors and tries to determine if there is a non-prime factor. Each range is sent to a process to find a non-prime factor. A multiprocessing.Manager object is used as a flag, so that if a process found a non-prime factor, it sets a flag. The flag is checked periodically. If the flag is set, all processes are terminated.
Because multiprocessing.map only accepts function with one argument, I used itertools.product to create an argument generator that contains the range and the manager object.
If I run the code as it is, I get the following error.
Traceback (most recent call last):
File "/home/briansia/projects/python/multiprocess/prime_manager.py", line 118, in <module>
isprime = check_prime(n, nbr_processes, value)
File "/home/briansia/projects/python/multiprocess/prime_manager.py", line 103, in check_prime
results = pool.map(check_prime_in_range, list(args))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
TypeError: check_prime_in_range() missing 1 required positional argument: 'value'
If I print the args generator above the map function, then the code runs correctly. How exactly did the print statement modify the generator such that it works with the map function?

Answer 1

To illustrate the problem, change the with Pool() block in your program to this:
with Pool() as pool:
args = ((arg, value) for arg in product(list(ranges_to_check)))
x = list(args)
# print(x)
results = pool.map(check_prime_in_range, x)
This will always crash whether you comment out the print statement or not.
The print statement in your code is not the issue: it is list(args), which causes the generator expression to run. Generator expressions run once and once only. After you've done list(args), the generator still exists and it's still named args, but it is now exhausted. When you run your program with the line containing list(args), you have already used up the generator; therefore you effectively pass an empty iterator to Pool.map. Your program doesn't actually work. It doesn't produce a traceback because it doesn't really do anything.
The problem with your code is this function:
def check_prime_in_range(n_from_i_to_i, value)
It takes two arguments. The first time you call it, you indeed pass two arguments:
if not check_prime_in_range((n, (from_i, to_i)), value):
But when you call it indirectly through Pool.map, it is called with only one argument. As you stated yourself, Pool.map only passes a single argument to its function. It's true that your generator has made two objects into a tuple, but that tuple is only one object and therefore it's the only argument that gets passed to check_prime_in_range. There is no second argument, as the traceback indicates.
I don't know how you want to fix the problem, but it might be a good idea to define check_prime_in_range as a function of one argument, and call it consistently. You can unpack the single argument inside the function, for example:
def check_prime_in_range(x):
n_from_i_to_i, value = x
(n, _range) = n_from_i_to_i
# etc.
Your first call would now be:
if not check_prime_in_range(((n, (from_i, to_i)), value)):
That's rather clumsy so I would consider defining a little class to hold all the variables in a single object. But that's a style issue.