PySide QtCore.QThreadPool and QApplication.quit() causes hangs? - multithreading

I want to use Qt's QThreadPool, but it seems to be hanging my application if the workers in the queue do not finish before calling QApplication.quit(). Can anyone tell me if i'm doing something wrong in the reduced testcase below?
import logging
log = logging.getLogger(__name__)
import sys
from PySide import QtCore
import time
class SomeWork(QtCore.QRunnable):
def __init__(self, sleepTime=1):
super(SomeWork, self).__init__()
self.sleepTime = sleepTime
def run(self):
time.sleep(self.sleepTime)
print "work", QtCore.QThread.currentThreadId()
def _test(argv):
logging.basicConfig(level=logging.NOTSET)
app = QtCore.QCoreApplication(argv)
pool = QtCore.QThreadPool.globalInstance()
TASK_COUNT = int(argv[1]) if len(argv) > 1 else 1
mainThread = QtCore.QThread.currentThreadId()
print "Main thread: %s"%(mainThread)
print "Max thread count: %s"%(pool.maxThreadCount())
print "Work count: %s"%(TASK_COUNT)
for i in range(TASK_COUNT):
pool.start(SomeWork(1))
def boom():
print "boom(); calling app.quit()"
app.quit()
QtCore.QTimer.singleShot(2000, boom)
#import signal
#signal.signal(signal.SIGINT, signal.SIG_DFL)
return app.exec_()
if __name__ == '__main__':
sys.exit(_test(sys.argv))
To be clear, this is the output I get:
(env)root#localhost:# python test_pool.py 1
Main thread: 3074382624
Max thread count: 1
Work count: 1
work 3061717872
boom(); calling app.quit()
(env)root#workshop:/home/workshop/workshop/workshop# python test_pool.py 20
Main thread: 3074513696
Max thread count: 1
Work count: 20
work 3060783984
boom(); calling app.quit()
And it hangs forever on the second command, but not the first.
Thanks for any help you may have.
EDIT:
To be clear, I expect that if app.quit() is called while threads are in the thread queue, they do not run. Already running threads should run to completion. Then, the application should close.
This example fails on a Windows machine as well
This example works on the same Windows machine, but using PyQt4

Adding this to _test() just before the exec() fixes the issue, although all the threads run:
def waitForThreads():
print "Waiting for thread pool"
pool.waitForDone()
app.aboutToQuit.connect(waitForThreads)

Related

Getting `BrokenProcessPool` error in a `concurrent.futures` example

The example I am running is mentioned in this PyMOTW3 link. I am reproducing the code here:
from concurrent import futures
import os
def task(n):
return (n, os.getpid())
ex = futures.ProcessPoolExecutor(max_workers=2)
results = ex.map(task, range(5, 0, -1))
for n, pid in results:
print('ran task {} in process {}'.format(n, pid))
As per source, I am supposed to get following output:
ran task 5 in process 40854
ran task 4 in process 40854
ran task 3 in process 40854
ran task 2 in process 40854
ran task 1 in process 40854
Instead, I'm getting a long message with following concluding line -
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
I am using Windows machine and running Python 9. All other examples are otherwise running fine. What is going wrong here?
I've finally been able to resolve the issue. The issue seems to be Windows specific. Following a related Stack Overflow post, I used if __name__=="__main__" idiom. The modified code is:
from concurrent import futures
import os
def task(n):
return (n, os.getpid())
def main():
ex = futures.ProcessPoolExecutor(max_workers=2)
results = ex.map(task, range(5, 0, -1))
for n, pid in results:
print('ran task {} in process {}'.format(n, pid))
if __name__ == '__main__':
main()
It worked, although I'm still not sure why this worked.

How to kill a QProcess instance using os.kill()?

Problem
Hey, recently when I'm using pyqt6's QProcess, I try to use os.kill() to kill a QProcess's instance. (The reason why I want to use os.kill() instead of QProcess().kill() is that I want to send a CTRL_C_EVENT signal when killing the process.) Even though with using correct pid (acquired by calling QProcess().processId()), it seems that a signal would be sent to all processes unexpectedly.
Code
Here's my code:
from PyQt6.QtCore import QProcess
import os
import time
import signal
process_a = QProcess()
process_a.start("python", ['./test.py'])
pid_a = process_a.processId()
print(f"pid_a = {pid_a}")
process_b = QProcess()
process_b.start("python", ['./test.py'])
pid_b = process_b.processId()
print(f"pid_b = {pid_b}")
os.kill(pid_a, signal.CTRL_C_EVENT)
try:
time.sleep(1)
except KeyboardInterrupt:
print("A KeyboardInterrupt should not be caught here.")
process_a.waitForFinished()
process_b.waitForFinished()
print(f"process_a: {process_a.readAll().data().decode('gbk')}")
print(f"process_b: {process_b.readAll().data().decode('gbk')}")
and ./test.py is simple:
import time
time.sleep(3)
print("Done")
What I'm expecting
pid_a = 19956
pid_b = 28468
process_a:
process_b: Done
What I've got
pid_a = 28040
pid_b = 23708
A KeyboardInterrupt should not be caught here.
process_a:
process_b:
Discussion
I don't know whether this is a bug or misusage. It seems that signal.CTRL_C_EVENT is sent to all processes. So, how do I kill one QProcess instance with signal CTRL_C_EVENT correctly?

How to listen to D-Bus events and the IPC channel at the same time?

I have the following simplyfied code. It listens to the D-Bus and does something when a new job is created. For that to work I need to start the GLib.MainLoop().run(), as it was presented by multiple examples I found.
While doing that, I want the program to continuously listen to the IPC bus and do something when a message is received. But obviously that doesn't work since my program is stuck at GLib.MainLoop().run().
How to implement something that let's me listen to the D-Bus and to the IPC at the same time?
#!/usr/bin/env python3.4
import asgi_ipc as asgi
from gi.repository import GLib
from pydbus import SystemBus
from systemd.daemon import notify as sd_notify
def main():
bus = SystemBus()
systemd = bus.get(".systemd1")
systemd.onJobNew = do_something_with_job()
channel_layer = asgi.IPCChannelLayer(prefix="endercp")
# Notify systemd this unit is ready
sd_notify("READY=1")
GLib.MainLoop().run()
while True:
message = channel_layer.receive(["endercp"])
if message is not (None, None):
do_something_with_message(message)
if __name__ == "__main__":
# Notify systemd this unit is starting
sd_notify("STARTING=1")
main()
# Notify systemd this unit is stopping
sd_notify("STOPPING=1")
Since IPCChannelLayer.receive() does not block, you can run it in an idle callback. Try this:
callback_id = GLib.idle_add(GLib.PRIORITY_DEFAULT_IDLE, poll_channel, data=channel_layer)
GLib.MainLoop().run()
GLib.idle_remove_by_data(channel_layer)
# ...
def poll_channel(channel_layer):
message = channel_layer.receive(["endercp"])
if message is not (None, None):
do_something_with_message(message)
return GLib.SOURCE_CONTINUE

Run Twisted reactor from a Thread

When i run reactor from thread in a synchrone python program, the twisted code is never called.
To resolve this problem, I had to put a sleep.
def _reactor_thread(self):
if not self.reactor.running:
self.reactor.run(installSignalHandlers=0)
def _start_thread( self ):
self.client_thread = Thread( target=self._reactor_thread,
name="mine" )
self.client_thread.setDaemon(True)
self.client_thread.start()
from time import sleep
sleep( 0.5 )
What is the best way to do it, instead of calling sleep?
We can do this with Crochet. Or with using addSystemEventTrigger.

pyqt thread qRegisterMetaType warning/crash

I'm writing a pyqt program utilising the Qwizard. From one screen I am running a thread and starting a command line program via Popen. However I'm getting the following warning message:
QObject::connect: Cannot queue arguments of type 'QTextCursor'
(Make sure 'QTextCursor' is registered using qRegisterMetaType().)
The program usually continues but crashes sporadically immediately after this point, leading me to feel that it is related.
As the code of the program is too much to attach, here are the relevant snippets:
class Fee(QThread):
def __init__(self, parent=None, options=None):
QThread.__init__(self, parent)
#set the given options
if options:
self.__options = options
def copy(self, devicename, outputfilename):
self.__device = devicename
self.__outputfile = outputfilename
self.start()
def run(self):
cmd = self.__getCommandLine(self.options, self.__device, self.__outputfile)
logging.info("Running Fee with command %s" % (cmd))
#start the process
output = ""
process = Popen(shlex.split(cmd), stdout= PIPE, stderr= STDOUT)
stderr = process.communicate()[1]
class FeeManager(QtCore.QObject):
def copyAndVerify(self):
self.__fee = Fee(self, self.options)
self.connect(self.__fee, QtCore.SIGNAL("finished()"), self._setCopyFinished)
self.connect(self.__fee, QtCore.SIGNAL("progressUpdated(QString, int, QString)"), self._setCopyProgress)
devicename = self.device.blockdevice
self.__image_outputfilename = "output_file.img")
self.__fee.copy(devicename, self.__image_outputfilename)
class FeeWizardPage(QtGui.QWizardPage):
def initializePage(self):
#already earlier in another wizard page is the following: self.feemanager = FeeManager(self, info)
self.connect(self.wizard().feemanager, QtCore.SIGNAL("copyProgressUpdated(QString, int, QString)"), self.updateCopyProgress)
self.connect(self.wizard().feemanager, QtCore.SIGNAL("verifyProgressUpdated(QString, int, QString)"), self.updateVerifyProgress)
self.connect(self.wizard().feemanager, QtCore.SIGNAL("finished(PyQt_PyObject)"), self.setDone)
self.wizard().feemanager.copyAndVerify()
What am I doing wrong? How can I avoid this message and hopefully bring some stability to the program. I've searched the internet and this forum, and while I've tried a number of suggestions for others, none have worked for this problem. Even when I comment out all signals and connects, I still get the same warning message.
Can someone help?
Thanks a lot.

Resources