I'm trying to run the following simple code using multiprocessing on Jupyter notebook (Windows)
from multiprocessing import Process
def worker():
"""worker function"""
print('Worker')
return
p = Process(target=worker)
p.start()
I could not get it working. As suggested in a few other posts, I tried to call the worker function from another file and it still didn't work.
I came across a few posts related to this but I could not find any working solution.
Am I doing something wrong here? The same code works fine on a Linux distribution.
Related
I was working on a larger piece of code and I kept getting an error when the program just seemed to end without doing anything. I narrowed down the problem and reproduced it below. As far as I understand the code should print the sentence and it seems to work on other online IDE's while failing on mine. Feel like I'm missing something super simple.
Failing On: IDLE Python 3.8.3 32-bit from python.org
Works On: onlinegdb.com
Code:
import multiprocessing
def x():
print("This is x func")
if __name__ == '__main__':
multiprocessing.freeze_support()
p = multiprocessing.Process(target=x)
p.start()
p.join()
Output:
>>>
I think the issue is IDLE just doesn't output stuff from stuff outside the main process. Need to use consoles which would output everything from the main and all other processes. Reference : Python multiprocessing example not working
On my windows 10 machine (and seemingly other people's as well), Jupyter Notebook can't seem to handle some basic multiprocessing functions like pool.map(). I can't seem to figure out why this might be, even though a solution has been suggested to call the function to be mapped as a script from another file. My question, though is why does this not work? Is there a better way to do this kind of thing beyond saving in another file?
Note that the solution was suggested in a similar question here. But I'm left wondering why this bug occurs, and whether there is another easier fix. To show what goes wrong, I'm including below a very simple version that hangs on my computer where the same function runs with no problems when the built-in function map is used.
import multiprocessing as mp
# create a grid
iterable = [3, 5, 10]
def add_3(iterable):
a = iterable + 3
return a
# Below runs no problem
results = list(map(add_3, iterable))
print(results)
# multiprocessing attempt (hangs)
def main():
pool = mp.Pool(2)
results = pool.map(add_3, iterable)
return results
if __name__ == "__main__": #Required not to spawn deviant children
results = main()
Edit: I've just tried this in Spyder and I've managed to get it to work. Unsurprisingly running the following didn't work.
if __name__ == "__main__": #Required not to spawn deviant children
results = main()
print(results)
But running it as the following does work because map uses the yield command and isn't evaluated until called which gives the typical problem.
if __name__ == "__main__": #Required not to spawn deviant children
results = main()
print(results)
edit edit:
From what I've read on the issue, it turns out that the issue is largely because of the ipython shell that jupyter uses. I think there might be an issue setting name. Either way using spyder or a different ide solved the problem, as long as you're not still running the multiprocessing function in an iPython shell.
I faced a similar problem like this. I can't use multiprocessing with function on the same script. The solution that works is to put the function on different notebook file and import it use ipynb:
from ipynb.fs.full.script_name import function_name
pool = Pool()
result = pool.map(function_name,iterable_argument)
I have a PyQt5 GUI class that I want to be able to create multiple instances of either from an interactive console or normal run. I need these GUIs to be non-blocking so that they can be used while subsequent code runs.
I've tried calling app.exec__() in separate threads for each GUI like this answer, but the program sometimes crashes as the comment on the answer warned it would:
Run pyQT GUI main app in seperate Thread
And now I'm trying to get the code below working which I made based on this answer:
Run Pyqt GUI main app as a separate, non-blocking process
But when I run it the windows pop and and immediately disappear
import sys
from PyQt5 import QtWidgets, QtGui, QtCore
import time
class MainWindow(QtWidgets.QWidget):
def __init__(self):
# call super class constructor
super(MainWindow, self).__init__()
# build the objects one by one
layout = QtWidgets.QVBoxLayout(self)
self.pb_load = QtWidgets.QPushButton('Load')
self.pb_clear= QtWidgets.QPushButton('Clear')
self.edit = QtWidgets.QTextEdit()
layout.addWidget(self.edit)
layout.addWidget(self.pb_load)
layout.addWidget(self.pb_clear)
# connect the callbacks to the push-buttons
self.pb_load.clicked.connect(self.callback_pb_load)
self.pb_clear.clicked.connect(self.callback_pb_clear)
def callback_pb_load(self):
self.edit.append('hello world')
def callback_pb_clear(self):
self.edit.clear()
def show():
app = QtWidgets.QApplication.instance()
if not app:
app = QtWidgets.QApplication(sys.argv)
win = MainWindow()
win.show()
if __name__ == '__main__':
show()
show()
EDIT - I don't see how this question is a duplicate. The 'duplicate' questions are only slightly related and don't provide solutions to my problem at all.
I want to be able to create multiple instances of a GUI (MainWindow in my example) by calling the show() function from either an interactive session or script, and I want those windows to stay on my screen while subsequent code is running.
EDIT2 - When I run the code as a script I can do what I want by using multiprocessing, see this demo:
https://www.screencast.com/t/5WvJNVSLm9OR
However I still need help because I want it to also work in interactive Python console sessions, and multiprocessing does not work in that case.
It isn't necessary to use separate threads or processes for this. You just need a way to maintain a reference to each new window when importing the script in a python interactive session. A simple list can be used for this. It is only necessary to explictly start an event-loop when running the script from the command-line; in an interactive session, it will be handled automatically by PyQt.
Here is an implementation of this approach:
...
_cache = []
def show(title=''):
if QtWidgets.QApplication.instance() is None:
_cache.append(QtWidgets.QApplication(sys.argv))
win = MainWindow()
win.setWindowTitle(title)
win.setAttribute(QtCore.Qt.WA_DeleteOnClose)
win.destroyed.connect(lambda: _cache.remove(win))
_cache.append(win)
win.show()
if __name__ == '__main__':
show('Foo')
show('Bar')
sys.exit(QtWidgets.QApplication.instance().exec_())
This is a minor addendum to #ekhumoro's answer. I don't have enough reputation to only add a comment so I had to write this as an answer.
#ekhumoro's answer almost fully answers #Esostack's question, but doesn't work in the Ipython console. After many hours of searching for the answer to this question myself, I came across a comment from #titusjan in a three year old thread (here) also responding to a good answer from #ekhumoro.
The missing part to #ekhumoro's answer which results in the gui windows freezing for Ipython specifically is that Ipython should be set to use the qt gui at launch or once running.
To make this work with Ipython:
Launch Ipython with ipython --gui=qt5
In a running Ipython console run the magic command %gui qt5
To fix it from a Python script you can run this function
def fix_ipython():
from IPython import get_ipython
ipython = get_ipython()
if ipython is not None:
ipython.magic("gui qt5")
Strange title I know, but it is exactly what I see. I am trying to run a requests (2.13.0) command from within a forked process (Mac OSX) using the multiprocessing module. I also happen to use numpy in my code (1.15.1) running on python 3.7. Here are my observations (see code below):
1) Without importing numpy: All works fine
2) Once I import numpy: Code crashes on starting of the forked process. Message given is:
objc[45539]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[45539]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
3) I could make it work again by calling a requests call from within the main process once before starting the new process (see commented section in code
4) On python 2.7, all seems to work fine in all cases above.
Sample minimal code to reproduce:
from multiprocessing import Process
import requests
import numpy # remove this import and it works fine on 3.7
def _worker():
full_url = "http://www.google.com"
result = requests.get(full_url)
print(result.text)
return 0
def run():
p=Process(target=_worker)
p.start()
p.join()
# Add these lines and the code works in 3.7 even with numpy imported
#try:
# requests.get('http://www.google.com')
#except:
# pass
run()
print('I am done')
I don't understand why I get this threading error by just using the threading module. It works fine without the threading module.
I searched on internet, but I don't get the appropriate answer.
Add the following line before the try statement:
connSkt = None
Also modify the close part:
if connSkt is not None:
connSkt.close()