I was working on a larger piece of code and I kept getting an error when the program just seemed to end without doing anything. I narrowed down the problem and reproduced it below. As far as I understand the code should print the sentence and it seems to work on other online IDE's while failing on mine. Feel like I'm missing something super simple.
Failing On: IDLE Python 3.8.3 32-bit from python.org
Works On: onlinegdb.com
Code:
import multiprocessing
def x():
print("This is x func")
if __name__ == '__main__':
multiprocessing.freeze_support()
p = multiprocessing.Process(target=x)
p.start()
p.join()
Output:
>>>
I think the issue is IDLE just doesn't output stuff from stuff outside the main process. Need to use consoles which would output everything from the main and all other processes. Reference : Python multiprocessing example not working
Related
I'm trying to run the following simple code using multiprocessing on Jupyter notebook (Windows)
from multiprocessing import Process
def worker():
"""worker function"""
print('Worker')
return
p = Process(target=worker)
p.start()
I could not get it working. As suggested in a few other posts, I tried to call the worker function from another file and it still didn't work.
I came across a few posts related to this but I could not find any working solution.
Am I doing something wrong here? The same code works fine on a Linux distribution.
I'm so curious about this and need some advise about how can this happen? Yesterday I've tried to implement multiprocessing in Python script which is running on Spyder in Window PC. Here is the code I've first tried.
import multiprocessing
import time
start = time.perf_counter()
def do_something():
print('Sleeping 1 second...')
time.sleep(1)
print('Done sleeping')
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'Finished in {round(finish-start,2)} second(s)')
It's return an error.
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)
Then I search for survival from this problem and also my boss. And found this suggestion
Python's multiprocessing doesn't work in Spyder IDE
So I've followed it and installed Pycharm and try to run the code on PyCharm and it's seem to be work I didn't get AttributeError, however I got this new one instead of
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
I've googled again then finally I got this
RuntimeError on windows trying python multiprocessing
what I have to do is adding this one line
if __name__ == '__main__':
before starting multiprocessing.
import multiprocessing
import time
start = time.perf_counter()
def do_something():
print('Sleeping 1 second...')
time.sleep(1)
print('Done sleeping')
if __name__ == '__main__':
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'Finished in {round(finish-start,2)} second(s)')
And it's work now moreover, it's not working only on PyCharm, now I can run this code on Spyder too. So that is why I have so curious? how come Spyder also work? This is quite persist because I'm also run this code on my other PC which is Window server 2016 with Spyder , I'm also do something.
Anyone can help explain what happen here why it's work?
Thank you.
There's a lot to unpack here, so I'll just give a brief overview. There's also some missing information like how you have spyder/pycharm configured, and what operating system you use, so I'll have to make some assumptions...
Based on the error messages you are probably using MacOS or Windows which means the default way python creates a child process is called spawn. This means it will start a completely new process from the python executable ("python.exe" on windows for example). It will then send a message to the new process telling it what function to execute (target), and optionally what arguments to call that function with. The new process will have to import the main file to have access to that function however, so if you are running the python interpreter in interactive mode, there is no "main" file to import, and you get the first error message: AttributeError.
The second error is also related to the importing of the "main" file. When you import a file, it basically just runs the file like any other python script. If you were to create a new child process during import that child would then also create a new child when it imports the same file. You would end up recursively creating infinite child processes until the computer crashed, so python disallows creating additional child processes during the import phase of a child process hence the RuntimeError.
On my windows 10 machine (and seemingly other people's as well), Jupyter Notebook can't seem to handle some basic multiprocessing functions like pool.map(). I can't seem to figure out why this might be, even though a solution has been suggested to call the function to be mapped as a script from another file. My question, though is why does this not work? Is there a better way to do this kind of thing beyond saving in another file?
Note that the solution was suggested in a similar question here. But I'm left wondering why this bug occurs, and whether there is another easier fix. To show what goes wrong, I'm including below a very simple version that hangs on my computer where the same function runs with no problems when the built-in function map is used.
import multiprocessing as mp
# create a grid
iterable = [3, 5, 10]
def add_3(iterable):
a = iterable + 3
return a
# Below runs no problem
results = list(map(add_3, iterable))
print(results)
# multiprocessing attempt (hangs)
def main():
pool = mp.Pool(2)
results = pool.map(add_3, iterable)
return results
if __name__ == "__main__": #Required not to spawn deviant children
results = main()
Edit: I've just tried this in Spyder and I've managed to get it to work. Unsurprisingly running the following didn't work.
if __name__ == "__main__": #Required not to spawn deviant children
results = main()
print(results)
But running it as the following does work because map uses the yield command and isn't evaluated until called which gives the typical problem.
if __name__ == "__main__": #Required not to spawn deviant children
results = main()
print(results)
edit edit:
From what I've read on the issue, it turns out that the issue is largely because of the ipython shell that jupyter uses. I think there might be an issue setting name. Either way using spyder or a different ide solved the problem, as long as you're not still running the multiprocessing function in an iPython shell.
I faced a similar problem like this. I can't use multiprocessing with function on the same script. The solution that works is to put the function on different notebook file and import it use ipynb:
from ipynb.fs.full.script_name import function_name
pool = Pool()
result = pool.map(function_name,iterable_argument)
I have recently started to delve into multiprocessing, as I believe my code can be easily parallelized. Upon working through the tutorials, though, I encountered an issue: functions distributed in a pool do not seem to print.
Here's the culprit:
__spec__ = None # This line is required for Spyder and not part of the actual example
from multiprocessing import Process
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
The output I receive is the following:
main line
module name: __main__
parent process: 10812
process id: 11348*
Now it is clear that the console only seems to print the info function, but not any output of the f function (which is using multiprocessing.Process). I have encountered similar issues with other examples I found online: computations are done and returned correctly when using multiprocessing, but prints never show up in the console.
Does anybody know why, and how to address this issue?
On a possibly related note, I am using Python 3.6 in Spyder 3.2.4 . Spyder seems to have a few quirks, as the first line in the code already is a workaround required to allow multiprocessing to work at all, an issue I found already discussed here. A similar, unresolved issue was mentioned here.
I would appreciate any help, and a happy new year to everyone.
(Spyder maintainer here) Multiprocessing doesn't work well on Windows in Spyder's IPython console. However, you can run your code in an external terminal to have the results you want.
To do that, please go to
Run > Configuration per file > Execute in an external system terminal.
Update: Since our 5.2.0 version (released in November 2021), prints generated while running multiprocessing code are captured and displayed in the IPython console for all operating systems.
You can run it through Spyders' IPython console by saving the function as a different .py file and importing it into the script you're running. For example, save:
def f(name):
info('function f')
print('hello', name)
In a file called worker.py. Then in your main file, do the following:
from multiprocessing import Process
import os
import worker
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
if __name__ == '__main__':
info('main line')
p = Process(target=worker.f, args=('bob',))
p.start()
p.join()
you can use log file instead. using fp.write()
I'm hoping that someone has a quick and easy solution to this issue that I'm facing. I'm working through chapter 5 of the 4th edition of the Programming Python book by Mark Lutz and I'm having some troubles beginning on page 189. Basically, there is a very simple example:
import _thread
def action(i):
print(i ** 32)
_thread.start_new_thread(action, (2, ))
For some reason, this script will not produce any output on my pc running Ubuntu 12.04, but will on my windows 7 machine. The output when run from the terminal is:
un#homepc:~/Desktop/pp$ python3.2 thread1.py
un#homepc:~/Desktop/pp$
Any help would be much appreciated.
Thanks.
I am not an expert of threads but this is most likely the problem:
When running this code, Python has nothing to do after the last line;so it exits forcefully ending the thread (whether it completed or not).
Depending on how the threads run, the thread may complete or it may not. (unreliable behaviour)
main thread pseudo-code instruction:
create the new thread
new thread:
action(2)
main thread next instruction:
program ended;exit
Fixed code:
import _thread
def action(i):
print(i ** 32)
action_lock.release() #Now Python will exit
#locks are a part of threaded programming; if you don't know what it is, you can google it.
action_lock = _thread.allocate_lock() #set a lock for the action function thread
action_lock.acquire() #Acquire the lock
_thread.start_new_thread(action, (2, ))
action_lock.acquire()
Thanks to Ramchandra for pointing me in the right direction. The problem was that the program was ending and killing the threads. If I run this:
import time, _thread
def print_t(name, delay):
while True:
try:
time.sleep(delay)
print(name)
except: pass
_thread.start_new_thread(print_t, ("First Thread", 1,))
_thread.start_new_thread(print_t, ("Second Thread", 2,))
while True:
try: pass
except KeyboardInterrupt:
print("Ending main program")
break
... then the program executes as planned. I don't have an answer as to why the code in the original post worked on windows 7, but not on ubuntu 12.04. Oh well. Hope this helps someone, sometime.