Multiprocessing does not see global variables? - python-3.x

Question

I have run into a strange behaviour at multiprocessing.
When i try to use a global variable in a function which is called from multiprocessing it does not see a global variable.
Example:
import multiprocessing
def func(useless_variable):
print(variable)
useless_list = [1,2,3,4,5,6]
p = multiprocessing.Pool(processes=multiprocessing.cpu_count())
variable = "asd"
func(useless_list)
for x in p.imap_unordered(func, useless_list):
pass
Output:
asd
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "pywork/asd.py", line 4, in func
print(variable)
NameError: name 'variable' is not defined
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "pywork/asd.py", line 11, in <module>
for x in p.imap_unordered(func, useless_list):
File "/usr/lib/python3.4/multiprocessing/pool.py", line 689, in next
raise value
NameError: name 'variable' is not defined
As you see the first time i just simply call func it print asd as expected. However when i call the very same function with multiprocessing it says the variable variable does not exists, even after i clearly printed it just before.
Does multiprocessing ignore global variables? How can i work this around?

Answer 1

multiprocessing Pools fork (or spawn in a way intended to mimic forking on Windows) its worker processes at the moment the Pool is created. forking maps the parent memory as copy-on-write in the children, but it doesn't create a persistent tie between them; after the fork, changes made in the parent are not visible in the children, and vice versa. You can't use any variables defined after the Pool was created, and changes made to variables from before the Pool was created will not be reflected in the workers.
Typically, with a Pool, you want to avoid mutable global state entirely; have all the data needed passed to the function you're imap-ing (or whatever) as arguments (which are serialized and sent to the children, so the state is correct), and have the function return any new data instead of mutating globals, which serializes it and sends it back to the parent process to use as it sees fit.
Managerss are an option, but not usually the correct option with Pools; you usually want to stick to workers only looking at read only globals from before the Pool was created, or working with arguments and returning new values, not using global state at all.

Answer 2

When you spam a process all context is copyed, you need to get use of managers for exachanging objects between them, check the official documentations, for managing state check this.