In my PyQt4-based program, QSliders (with signals sliderMoved and sliderReleased connected to callables) sometimes "freeze", i.e. they don't move anymore when trying to drag them with the mouse, even though sliderMoved and sliderReleased are still emitted.
This behaviour happens seemingly randomly, sometimes after running the program for hours -- making it more or less impossible to reproduce and test.
Any help to solve this issue would be welcome.
EDIT: This is with PyQt 4.10.4 on Python 3.4 and Windows 7.
After some debugging I am pretty sure that this was due to calling a GUI slot from a separate thread, which (I knew) is forbidden. Fixing this to use a proper signal-slot approach seems to have fixed the issue.
After calling the patch function defined below, all slot calls are wrapped by a wrapper that checks that they are called only from the GUI thread -- a warning is printed otherwise. This is how I found the culprit.
import functools
import sys
import threading
import traceback
from PyQt4.QtCore import QMetaMethod
from PyQt4.QtGui import QWidget
SLOT_CACHE = {}
def patch():
"""Check for calls to widget slots outside of the main thread.
"""
qwidget_getattribute = QWidget.__getattribute__
def getattribute(obj, name):
attr = qwidget_getattribute(obj, name)
if type(obj) not in SLOT_CACHE:
meta = qwidget_getattribute(obj, "metaObject")()
SLOT_CACHE[type(obj)] = [
method.signature().split("(", 1)[0]
for method in map(meta.method, range(meta.methodCount()))
if method.methodType() == QMetaMethod.Slot]
if (isinstance(attr, type(print)) and # Wrap builtin functions only.
attr.__name__ in SLOT_CACHE[type(obj)]):
#functools.wraps(
attr, assigned=functools.WRAPPER_ASSIGNMENTS + ("__self__",))
def wrapper(*args, **kwargs):
if threading.current_thread() is not threading.main_thread():
print("{}.{} was called out of main thread:".format(
type(obj), name), file=sys.stderr)
traceback.print_stack()
return attr(*args, **kwargs)
return wrapper
else:
return attr
QWidget.__getattribute__ = getattribute
Related
I wanted to use the ready() hook in my AppConfig to start django-rq scheduler job. However it does so multiple times, every times I start the server. I imagine that's due to threading however I can't seem to find a suitable workaround. This is my AppConfig:
class AnalyticsConfig(AppConfig):
name = 'analytics'
def ready(self):
print("Init scheduler")
from analytics.services import save_hits
scheduler = django_rq.get_scheduler('analytics')
scheduler.schedule(datetime.utcnow(), save_hits, interval=5)
Now when I do runserver, Init scheduler is displayed 3 times. I've done some digging and according to this question I started the server with --noreload which didn't help (I still got Init scheduler x3). I also tried putting
import os
if os.environ.get('RUN_MAIN', None) != 'true':
default_app_config = 'analytics.apps.AnalyticsConfig'
in my __init__.py however RUN_MAIN appears to be None every time.
Afterwards I created a FileLock class, to skip configuration after the first initialization, which looks like this:
class FileLock:
def __get__(self, instance, owner):
return os.access(f"{instance.__class__.__name__}.lock", os.F_OK)
def __set__(self, instance, value):
if not isinstance(value, bool):
raise AttributeError
if value:
f = open(f"{instance.__class__.__name__}.lock", 'w+')
f.close()
else:
os.remove(f"{instance.__class__.__name__}.lock")
def __delete__(self, obj):
raise AttributeError
class AnalyticsConfig(AppConfig):
name = 'analytics'
locked = FileLock()
def ready(self):
from analytics.services import save_hits
if not self.locked:
print("Init scheduler")
scheduler = django_rq.get_scheduler('analytics')
scheduler.schedule(datetime.utcnow(), save_hits, interval=5)
self.locked = True
This does work, however the lock is not destroyed after the app quits. I tried removing the .lock files in settings.py but it also runs multiple times, making this pointless.
My question is: How can I prevent django from calling ready() multiple times, or how otherwise can I teardown the .lock files after django exits or right after it boots?
I'm using python 3.8 and django 3.1.5
I have a python3 program that starts a second thread (besides the main thread) for handling some events asynchronously. Ideally, my program works without a flaw and never has an unhandled exceptions. But stuff happens. When/if there is an exception, I want the whole interpreter to exit with an error code as if it had been a single thread. Is that possible?
Right now, if an exception occurs on the spawned thread, it prints out the usual error information, but doesn't exit. The main thread just keeps going.
Example
import threading
import time
def countdown(initial):
while True:
print(initial[0])
initial = initial[1:]
time.sleep(1)
if __name__ == '__main__':
helper = threading.Thread(target=countdown, args=['failsoon'])
helper.start()
time.sleep(0.5)
#countdown('THISWILLTAKELONGERTOFAILBECAUSEITSMOREDATA')
countdown('FAST')
The countdown will eventually fail to access [0] from the string because it's been emptied causing an IndexError: string index out of range error. The goal is that whether the main or helper dies first, the whole program dies alltogether, but the stack trace info is still output.
Solutions Tried
After some digging, my thought was to use sys.excepthook. I added the following:
def killAll(etype, value, tb):
print('KILL ALL')
traceback.print_exception(etype, value, tb)
os.kill(os.getpid(), signal.SIGKILL)
sys.excepthook = killAll
This works if the main thread is the one that dies first. But in the other case it does not. This seems to be a known issue (https://bugs.python.org/issue1230540). I will try some of the workarounds there.
While the example shows a main thread and a helper thread which I created, I'm interested in the general case where I may be running someone else's library that launches a thread.
Well, you could simply raise an error in your thread and have the main thread handle and report that error. From there you could even terminate the program.
For example on your worker thread:
try:
self.result = self.do_something_dangerous()
except Exception as e:
import sys
self.exc_info = sys.exc_info()
and on main thread:
if self.exc_info:
raise self.exc_info[1].with_traceback(self.exc_info[2])
return self.result
So to give you a more complete picture, your code might look like this:
import threading
class ExcThread(threading.Thread):
def excRun(self):
pass
#Where your core program will run
def run(self):
self.exc = None
try:
# Possibly throws an exception
self.excRun()
except:
import sys
self.exc = sys.exc_info()
# Save details of the exception thrown
# DON'T rethrow,
# just complete the function such as storing
# variables or states as needed
def join(self):
threading.Thread.join(self)
if self.exc:
msg = "Thread '%s' threw an exception: %s" % (self.getName(), self.exc[1])
new_exc = Exception(msg)
raise new_exc.with_traceback(self.exc[2])
(I added an extra line to keep track of which thread is causing the error in case you have multiple threads, it's also good practice to name them)
My solution ended up being a happy marriage between the solution posted here and the SIGKILL solution piece from above. I added the following killall.py submodule to my package:
import threading
import sys
import traceback
import os
import signal
def sendKillSignal(etype, value, tb):
print('KILL ALL')
traceback.print_exception(etype, value, tb)
os.kill(os.getpid(), signal.SIGKILL)
original_init = threading.Thread.__init__
def patched_init(self, *args, **kwargs):
print("thread init'ed")
original_init(self, *args, **kwargs)
original_run = self.run
def patched_run(*args, **kw):
try:
original_run(*args, **kw)
except:
sys.excepthook(*sys.exc_info())
self.run = patched_run
def install():
sys.excepthook = sendKillSignal
threading.Thread.__init__ = patched_init
And then ran the install right away before any other threads are launched (of my own creation or from other imported libraries).
Just wanted to share my simple solution.
In my case I wanted the exception to display as normal but then immediately stop the program. I was able to accomplish this by starting a timer thread with a small delay to call os._exit before raising the exception.
import os
import threading
def raise_and_exit(args):
threading.Timer(0.01, os._exit, args=(1,)).start()
raise args[0]
threading.excepthook = raise_and_exit
Python 3.8 added threading.excepthook which makes it possible to handle this more cleanly.
I wrote the package "unhandled_exit" to do just that. It basically adds os._exit(1) to after the default handler. This means you get the normal backtrace before the process exits.
Package is published to pypi here: https://pypi.org/project/unhandled_exit/
Code is here: https://github.com/rfjakob/unhandled_exit/blob/master/unhandled_exit/\_\_init__.py
Usage is simply:
import unhandled_exit
unhandled_exit.activate()
I wrote a PyQt5 GUI (Python 3.5 on Win7). While it is running, its GUI is not responsive. To still be able to use the GUI, I tried to use QThread from this answer: https://stackoverflow.com/a/6789205/5877928
The module is now designed like this:
class MeasureThread(QThread):
def __init(self):
super().__init__()
def get_data(self):
data = auto.data
def run(self):
time.sleep(600)
# measure some things
with open(outputfile) as f:
write(things)
class Automation(QMainWindow):
[...]
def measure(self):
thread = MeasureThread()
thread.finished.connect(app.exit)
for line in open(inputfile):
thread.get_data()
thread.start()
measure() gets called once per measurement but starts the thread once per line in inputfile. The module now exits almost immediately after starting it (I guess it runs all thread at once and does not sleep) but I only want it to do all the measurements in another single thread so the GUI can still be accessed.
I also tried to apply this to my module, but could not connect the methods used there to my methods: http://www.xyzlang.com/python/PyQT5/pyqt_multithreading.html
The way I used to use the module:
class Automation(QMainWindow):
[...]
def measure(self):
param1, param2 = (1,2)
for line in open(inputfile):
self.measureValues(param1, param2)
def measureValues(self, param1, param2):
time.sleep(600)
# measure some things
with open(outputfile) as f:
write(things)
But that obviously used only one thread. Can you help me to find the right method to use here( QThread, QRunnable) or to map the example of the second link to my module?
Thanks!
I am trying to determine which parts of my python code are running the slowest, so that I have a better understanding on what I need to fix. I recently discovered cProfile and gprof2dot which have been extremely helpful. My problem is that I'm not seeing any information about functions that I'm using as callbacks, which I believe might be running very slowly. From what I understand from this answer is that cProfile only works by default in the main thread, and I'm guessing that callbacks use a separate thread. That answer showed a way to get things working if you are using the threading library, but I couldn't get it to work for my case.
Here is roughly what my code looks like:
import rospy
import cv
import cProfile
from numpy import *
from collections import deque
class Bla():
def __init__( self ):
self.image_data = deque()
self.do_some_stuff()
def vis_callback( self, data ):
cv_im = self.bridge.imgmsg_to_cv( data, "mono8" )
im = asarray( cv_im )
self.play_with_data( im )
self.image_data.append( im )
def run( self ):
rospy.init_node( 'bla', anonymous=True )
sub_vis = rospy.Subscriber('navbot/camera/image',Image,self.vis_callback)
while not rospy.is_shutdown():
if len( self.image_data ) > 0:
im = self.image_data.popleft()
self.do_some_things( im )
def main():
bla = Bla()
bla.run()
if __name__ == "__main__":
cProfile.run( 'main()', 'pstats.out' ) # this could go here, or just run python with -m cProfile
#main()
Any ideas on how to get cProfile info on the vis_callback function? Either by modifying the script itself, or even better using python -m cProfile or some other command line method?
I'm guessing it is either the reading of images or appending them to the queue which slow. My gut feeling is that storing them on a queue is a horrible thing to do, but I need to display them with matplotlib, which refuses to work if it isn't in the main thread, so I want to see exactly how bad the damage is with this workaround
I have attempted to purloin some of the code shipped with IP in Action and, following issues, I have even gone to the lengths of reading the book!
I am getting the error 'expect Delegate, got Function' when I use the following code. FYI I am passing in a reference to a WPF textBox so I should have a dispatcher on my UI element
I have removed all of the threading pipe reading stuff just to leave 'test' code:
import System
import System.IO
import Avacta.Optim.Server.WebServices
import Avacta.Optim.Server.DataModel
import sys
import clr
import time
from System import Console
from System.Threading import Thread, ThreadStart
def SetDispatcher(ui_element):
global dispatcher # needed else "Exception: 'NoneType' object has no attribute 'BeginInvoke'"
dispatcher = ui_element.Dispatcher
def Dispatch(function, *args):
dispatcher.BeginInvoke(lambda *_: function(*args))
def GetDispatchFunction(function):
return lambda *args: Dispatch(function, *args)
class ListOutput:
def __init__(self, textbox):
self.textbox = textbox
def write(self, string):
Dispatch(self.addText, string) # error: "expect Delegate, got Function"
#self.addText(string) # ok works fine w-w/o dispatcher stuff
def addText(self, string):
textbox.AppendText(string)
if textbox != None:
listout = ListOutput(textbox)
sys.stdout = listout
SetDispatcher(textbox)
print "Define running"
#running = True
Thread.Sleep(0)
time.sleep(2)
print "Start The Comms Thread..."
#comms_t = Thread(ThreadStart(run_comms))
#comms_t.Start()
Thread.Sleep(0)
time.sleep(2)
Any clues appreciated.
AndyF.
Thanks to Dino Viehland
Changing my dispatcher code to call the dispatcher directly fixes this issue.
dispatcher.BeginInvoke(System.Action(lambda *_: function(*args)))
Unfortunately I no longer get real-time output from my print statments to my 'console' - it all appears when the script completes. Remove the dispatcher and it reverts to real-time...
There is a set of dispatcher static methods (extension methods) are provided by way of DispatcherExtensions which take an Action as the parameter.
The code sample below demonstrates the usage of the WPF dispatcher. More information is available here http://msdn.microsoft.com/en-us/library/cc647497.aspx
import clr
clr.AddReference('WindowsBase')
clr.AddReference('System.Windows.Presentation')
from System import Action
from System.Windows.Threading import DispatcherExtensions, Dispatcher
dispatcher = Dispatcher.CurrentDispatcher
def workCallBack():
print 'working'
DispatcherExtensions.BeginInvoke(dispatcher, Action(workCallBack))