How can I query asynchronously via a QSqlQueryModel? - multithreading

I wish to query a SQL database via QSqlQueryModel (PyqQt 5/Qt 5.2) asynchronously, so that the GUI doesn't block. How can this be accomplished? Maybe through multithreading? Please provide code of how to do this. If using QSqlQueryModel asynchronously isn't practical, feel free to provide alternatives (should be usable with QTableView though).
My (synchronous) code currently looks as shown beneath. The main script bin/app.py loads gui/__init__.py and executes its main method. That in turn uses gui.models.Table to load data from the database. The problem is that gui.models.Table queries the database synchronously and locks up the GUI in the meantime.
bin/app.py:
import os.path
import sys
sys.path.insert(0, os.path.abspath(os.path.join(
os.path.dirname(__file__), "..")))
import gui
if __name__ == "__main__":
gui.main()
gui/__init__.py:
import sys
import os.path
from PyQt5 import uic
from PyQt5 import QtCore, QtWidgets
from gui import models
class MainWindow(QtWidgets.QMainWindow):
def __init__(self):
super(MainWindow, self).__init__()
uic.loadUi(os.path.join(os.path.dirname(__file__), 'app.ui'), self)
self.tableView.setModel(models.Table(self))
def main():
app = QtWidgets.QApplication(sys.argv)
w = MainWindow()
w.show()
app.exec_()
gui/models.py:
import os.path
from PyQt5.QtCore import *
from PyQt5.QtSql import *
class Table(QSqlQueryModel):
def __init__(self, parent=None):
super(Table, self).__init__(parent)
pth = os.path.abspath(os.path.join(os.path.dirname(__file__), "..",
"test.sqlite"))
db = QSqlDatabase.addDatabase("QSQLITE")
db.setDatabaseName(pth)
if not db.open():
raise Exception("Couldn't open database '{}'".format(pth))
try:
self.setQuery("select * from Test")
finally:
db.close()

Unfortunately, a typical database driver that Qt (or anyone else, really) uses is synchronous. Qt views unfortunately don't know how to deal with models in foreign threads.
The solution thus requires a shim proxy model, subclassing QIdentityProxyModel. The first step in the implementation is to shim all of the source model's method calls with blocking QMetaObject::invokeMethod calls. This is needed just to be correct, if not asynchronous just yet. It' just to expose a safe interface to a model that lives in another thread.
The next step is to provide an asynchronous veneer over some of the functionality. Suppose that you want to make the data method asynchronous. What you do is:
For each role, have a cache of variant values keyed by the model index.
On the dataChanged signal from the source model, cache all the values that were changed, across all roles. The data call needs to be queued in the model's thread - more on that later.
In data, if there's a cache hit, return it. Otherwise return a null variant and queue the data call in the model's thread.
Your proxy should have a private method called cacheData that will be called from the queued calls. In another answer, I've detailed how to queue functor calls in another thread. Leveraging that, your data call queuing method can look like:
void ThreadsafeProxyModel::queueDataCall(const QModelIndex & index, int role) {
int row = index.row();
int column = index.column();
void * data = index.internalPointer();
postMetacall(sourceModel()->thread(), [this, row, column, data, role]{
QVariant data = sourceModel()->data(createIndex(row, column, data), role);
QMetaObject::invoke(this, "cacheData",
Q_ARG(QVariant, data), Q_ARG(int, role),
Q_ARG(int, row), Q_ARG(int, column), Q_ARG(void*, data));
});
}
This is just a sketch. It'd be fairly involved, but certainly doable, and still maintaining the semantics of a real model.

Related

Swift: How to put worker into separate thread, show results in main view?

I wonder if there is an easy way to do this: Put a long-running worker into a separate thread so as not to block the UI. And display the results in the main view. Under SwiftUI or UIKit. What I found on the web was all very complex. Or is there a completely different approach in Swift?
I made a minimalistic Python program to show what I want to do. It displays the WiFi signal strength in MacOS.
import time
import sys
import subprocess
from PyQt5.QtWidgets import QApplication, QWidget, QLabel, QHBoxLayout
from PyQt5.QtCore import QObject, pyqtSignal, QThread
class Worker(QObject):
send_output = pyqtSignal(str)
def __init__(self):
super().__init__()
def worker(self):
while True:
_out = subprocess.check_output(
["/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport",
"-I"])\
.decode("utf-8")
self.send_output.emit("RSSI: " + _out.splitlines()[0][-3:] + " dBm")
time.sleep(2)
class MainWindow(QWidget):
do_work = pyqtSignal(object)
def __init__(self):
super().__init__()
self.label = QLabel()
layout = QHBoxLayout()
self.setLayout(layout)
layout.addWidget(self.label)
self.show()
self.app_thread = QThread()
self.app = Worker()
self.app.moveToThread(self.app_thread)
self.app.send_output.connect(self.output_label)
self.app_thread.started.connect(self.app.worker)
self.app_thread.start()
def output_label(self, _str):
self.label.setText(_str)
if __name__ == '__main__':
application = QApplication(sys.argv)
mainwindow = MainWindow()
sys.exit(application.exec())
I'm trying to find my way into Swift right now. It's exciting, but really a big thing. Thanks in advance!
This is a very broad question, so I'm going to answer it rather broadly as well.
Yes, you can run tasks in separate threads and then return data to the main thread. There are number of ways to do this, a common one being DispatchQueue.
Let's take an over-simplified example (definitely not meant to be real-world code):
struct ContentView : View {
#State var result = ""
var body: some View {
Text("Result: \(result)")
.onAppear {
DispatchQueue.global(qos: .background).async {
//do your long-running code
var innerResult = 0
for i in 0...1000000 {
innerResult += 5
print(i)
//use DispatchQueue.main.async here if you want to update during the task
}
//update at the end:
DispatchQueue.main.async {
result = "Done! \(innerResult)"
}
}
}
}
}
In this example, when the view appears, a task is run in a background thread, via DispatchQueue. In this case, I'm just taking advantage of the fact that printing to the console is a rather expensive operation if done a million times.
When it finishes, it dispatches back to the main thread and updates the results in the state variable.
DispatchQueue is not specific to UIKit or SwiftUI -- it was just easiest to put the demo together using SwiftUI.
If you were to truly start writing code to do this, you'd want to do some research about how the DispatchQueues work, including which queue to use, whether to create your own, whether you want tasks done serially, etc, but for the purposes of the broad question of can this be done (and how easily), this at least shows the basics.

pyqt threading, appropriate pattern?

I working on a gui application which plots real time data as it comes in. There is a main class which is a plot window. The window shows the plot of the data itself as well as a number of buttons, checkboxes, textboxes, etc. that a user can interact with to modify the plot output. All of these widgets configure the "settings" for the plot. The plotting of new data itself is resource intensive. The loading and plotting of the data may take around a second. I want the user to be able to modify the settings in the GUI without worrying about the lag of loading and plotting so I want to execute the loading and plotting in a separate thread.
I've come up with, what seems to me, to be a weird way to accomplish this but which works. It is something like this:
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtCore import QThread, pyqtSignal
class PlotWorker(QtCore.QObject):
def __init__(self, plotwindow):
self.plotwindow = plotwindow
self.plot_thread = QThread()
self.plot_thread.start()
self.moveToThread(self.plot_thread)
def worker_update:
self.plotwindow.updating = True
self.plotwindow.plot()
self.plotwindow.updating = False
class PlotWindow(Ui_PlotWindow, QtWidgets.QMainWindow):
update_signal = pyqtSignal()
def __init__(self):
self.setupUi(self)
self.settings = self.settings_lineEdit.text()
self.worker = PlotWorker(self)
self.update_signal.connect(self.worker.worker_update)
self.update_settings_pushButton.clicked.connect(self.update_settings)
self.updating = False
self.refresh_timer = QtCore.QTimer(self)
self.refresh_timer.timeout.connect(self.update)
self.update_settings()
self.plot()
self.refresh_time.start(1)
def update_setting(self):
self.settings = self.settings_lineEdit.text()
def update(self)
if self.updating:
print('Already updating, cannot update until previous update complete')
elif not self.updating:
self.update_signal.emit()
def plot(self):
# Plot the data, slow. Uses self.settings
self.load()
...
def load(self):
# Load the data from the appropriate directory/file, slow
...
This gets the rough gist of what I am going for. Ui_PlotWindow implements setupUi. In practices there self.settings stands for a long list of settings variables which can be manipulated by the user. The check on self.updating ensures that requests to update the plot which arrive while the plot is currently updating are terminated rather than added to a queue in the worker thread event loop.
Though my code works, I feel like the pattern I am using with the worker thread is a bit strange. Basically ALL of the information needed is in the PlotWindow class, but since I want to call one of the methods of the PlotWindow class in a separate thread I feel I need a different QObject which can live in a different QThread to house the slot which will run the expensive method.
It just feels a bit roundabout to have a whole seperate class and object just to call a function which already exists in the first class.. However, I am new to threading applications and perhaps this is a normal pattern and not to be worried about?
I would appreciate any advice for how I might be able to make the flow of this function more clear.
The answer is to use a different pattern for threading. Instead of the self.thread = and self.moveToThread for the worker the worker should simply BE a QThread. The thread can be "started" in which case its run method is initiated in a new thread. If a request is made in the main thread to "start" the worker thread again while it is already running nothing will happen. Below is a revision of the above code which implements this pattern.
WARNING: This code is still a bad idea because the plotting involving redrawing GUI objects (the plots). This should not happen in a thread different than the main thread. For posterity I'll state that code in the format in the original question did in fact run but I think that was by luck that the plotting worked in the worker thread. The best pattern would be to have any data loading and processing happen in the worker thread but to then finally emit the data to a plotting method of the main thread to update the GUI with the new data.
class PlotWorker(QtCore.QThread):
def __init__(self, plotwindow):
self.plotwindow = plotwindow
def run(self):
self.plotwindow.plot()
class PlotWindow(Ui_PlotWindow, QtWidgets.QMainWindow):
update_signal = pyqtSignal()
def __init__(self):
self.setupUi(self)
self.settings = self.settings_lineEdit.text()
self.worker = PlotWorker(self)
self.update_settings_pushButton.clicked.connect(self.update_settings)
self.refresh_timer = QtCore.QTimer(self)
self.refresh_timer.timeout.connect(self.update)
self.update_settings()
self.plot()
self.refresh_time.start(1)
def update_setting(self):
self.settings = self.settings_lineEdit.text()
def update(self)
if self.worker.started():
print('Already updating, cannot update until previous update complete')
elif not self.worker.started():
self.worker.start()
def plot(self):
# Plot the data, slow. Uses self.settings
self.load()
...
def load(self):
# Load the data from the appropriate directory/file, slow
...

No event loop is runing on this thread

I am creating a new thread to separate the UI interface from the data processing logic., but for reasons completely unfathomable to me, when I build and run my app , the app promptly crashes with the exception:
File "capnp/lib/capnp.pyx", line 2150, in capnp.lib.capnp._DynamicCapabilityClient._send
capnp.lib.capnp.KjException: src/kj/async.c++:53: failed: expected loop != nullptr; No event loop is running on this thread.
stack: 0x7f9bd4774489 0x7f9bd477845d 0x7f9bd4c0aacc 0x7f9bd49a81cb 0x7f9bd49a8207 0x7f9bd49a8304 0x7f9bd54d0341 0x7f9bd4edb93e 0x7f9bd4edba96 0x7f9bd4ec305e 0x7f9bd4ec43e9 0x7f9bd4eb0651 0x7f9bd54b5ad8 0x7f9bd5485844 0x55b92dc620c5 0x55b92dd55107
I'm not sure what in the build system could cause this runtime error,here is my code ,My best guess is some kind of arcane linking quirk, but I have no idea what or how to fix it.
import sys
import os
from PyQt5.QtCore import *
from PyQt5.QtGui import QGuiApplication
from cnc import Cnc
cnc = Cnc(os.path.join(os.getenv("HOME"), "cnc"), "192.168.7.98")
class workthread(QThread):
def __init__(self):
super(workthread, self).__init__()
def run(self):
self.cnc_axis_names = {axis.id: axis.name.absolute for axis in cnc.axes}
self.coors = {self.cnc_axis_names[id_]: coor for id_, coor in cnc.axes.coors().items()}
data = list(self.coors.values())
print(data)
self.sleep(1)
self.run()
self.exic_()
if __name__ == '__main__':
thread1 = workthread()
thread1.start()
app = QGuiApplication(sys.argv)
sys.exit(app.exec_())
Any advice on how to troubleshoot this is welcome.
I don't know what cnc is, but my guess is that it uses Cap'n Proto RPC. Cap'n Proto RPC uses a single-threaded event loop concurrency model, so RPC objects originally created on one thread cannot be manipulated on another thread.
In your program, you are creating the cnc object at startup, in the main thread. You then create a new thread and try to access the object from there. This won't work. You need to construct and use the object on a single thread.

Running Infinite loops in PyQT [duplicate]

I have a GUI in PyQt with a function addImage(image_path). Easy to imagine, it is called when a new image should be added into a QListWidget. For the detection of new images in a folder, I use a threading.Thread with watchdog to detect file changes in the folder, and this thread then calls addImage directly.
This yields the warning that QPixmap shouldn't be called outside the gui thread, for reasons of thread safety.
What is the best and most simple way to make this threadsafe? QThread? Signal / Slot? QMetaObject.invokeMethod? I only need to pass a string from the thread to addImage.
You should use the built in QThread provided by Qt. You can place your file monitoring code inside a worker class that inherits from QObject so that it can use the Qt Signal/Slot system to pass messages between threads.
class FileMonitor(QObject):
image_signal = QtCore.pyqtSignal(str)
#QtCore.pyqtSlot()
def monitor_images(self):
# I'm guessing this is an infinite while loop that monitors files
while True:
if file_has_changed:
self.image_signal.emit('/path/to/image/file.jpg')
class MyWidget(QtGui.QWidget):
def __init__(self, ...)
...
self.file_monitor = FileMonitor()
self.thread = QtCore.QThread(self)
self.file_monitor.image_signal.connect(self.image_callback)
self.file_monitor.moveToThread(self.thread)
self.thread.started.connect(self.file_monitor.monitor_images)
self.thread.start()
#QtCore.pyqtSlot(str)
def image_callback(self, filepath):
pixmap = QtGui.QPixmap(filepath)
...
I believe the best approach is using the signal/slot mechanism. Here is an example. (Note: see the EDIT below that points out a possible weakness in my approach).
from PyQt4 import QtGui
from PyQt4 import QtCore
# Create the class 'Communicate'. The instance
# from this class shall be used later on for the
# signal/slot mechanism.
class Communicate(QtCore.QObject):
myGUI_signal = QtCore.pyqtSignal(str)
''' End class '''
# Define the function 'myThread'. This function is the so-called
# 'target function' when you create and start your new Thread.
# In other words, this is the function that will run in your new thread.
# 'myThread' expects one argument: the callback function name. That should
# be a function inside your GUI.
def myThread(callbackFunc):
# Setup the signal-slot mechanism.
mySrc = Communicate()
mySrc.myGUI_signal.connect(callbackFunc)
# Endless loop. You typically want the thread
# to run forever.
while(True):
# Do something useful here.
msgForGui = 'This is a message to send to the GUI'
mySrc.myGUI_signal.emit(msgForGui)
# So now the 'callbackFunc' is called, and is fed with 'msgForGui'
# as parameter. That is what you want. You just sent a message to
# your GUI application! - Note: I suppose here that 'callbackFunc'
# is one of the functions in your GUI.
# This procedure is thread safe.
''' End while '''
''' End myThread '''
In your GUI application code, you should create the new Thread, give it the right callback function, and make it run.
from PyQt4 import QtGui
from PyQt4 import QtCore
import sys
import os
# This is the main window from my GUI
class CustomMainWindow(QtGui.QMainWindow):
def __init__(self):
super(CustomMainWindow, self).__init__()
self.setGeometry(300, 300, 2500, 1500)
self.setWindowTitle("my first window")
# ...
self.startTheThread()
''''''
def theCallbackFunc(self, msg):
print('the thread has sent this message to the GUI:')
print(msg)
print('---------')
''''''
def startTheThread(self):
# Create the new thread. The target function is 'myThread'. The
# function we created in the beginning.
t = threading.Thread(name = 'myThread', target = myThread, args = (self.theCallbackFunc))
t.start()
''''''
''' End CustomMainWindow '''
# This is the startup code.
if __name__== '__main__':
app = QtGui.QApplication(sys.argv)
QtGui.QApplication.setStyle(QtGui.QStyleFactory.create('Plastique'))
myGUI = CustomMainWindow()
sys.exit(app.exec_())
''' End Main '''
EDIT
Mr. three_pineapples and Mr. Brendan Abel pointed out a weakness in my approach. Indeed, the approach works fine for this particular case, because you generate / emit the signal directly. When you deal with built-in Qt signals on buttons and widgets, you should take another approach (as specified in the answer of Mr. Brendan Abel).
Mr. three_pineapples adviced me to start a new topic in StackOverflow to make a comparison between the several approaches of thread-safe communication with a GUI. I will dig into the matter, and do that tomorrow :-)

QSlider stuck though still emitting sliderMoved

In my PyQt4-based program, QSliders (with signals sliderMoved and sliderReleased connected to callables) sometimes "freeze", i.e. they don't move anymore when trying to drag them with the mouse, even though sliderMoved and sliderReleased are still emitted.
This behaviour happens seemingly randomly, sometimes after running the program for hours -- making it more or less impossible to reproduce and test.
Any help to solve this issue would be welcome.
EDIT: This is with PyQt 4.10.4 on Python 3.4 and Windows 7.
After some debugging I am pretty sure that this was due to calling a GUI slot from a separate thread, which (I knew) is forbidden. Fixing this to use a proper signal-slot approach seems to have fixed the issue.
After calling the patch function defined below, all slot calls are wrapped by a wrapper that checks that they are called only from the GUI thread -- a warning is printed otherwise. This is how I found the culprit.
import functools
import sys
import threading
import traceback
from PyQt4.QtCore import QMetaMethod
from PyQt4.QtGui import QWidget
SLOT_CACHE = {}
def patch():
"""Check for calls to widget slots outside of the main thread.
"""
qwidget_getattribute = QWidget.__getattribute__
def getattribute(obj, name):
attr = qwidget_getattribute(obj, name)
if type(obj) not in SLOT_CACHE:
meta = qwidget_getattribute(obj, "metaObject")()
SLOT_CACHE[type(obj)] = [
method.signature().split("(", 1)[0]
for method in map(meta.method, range(meta.methodCount()))
if method.methodType() == QMetaMethod.Slot]
if (isinstance(attr, type(print)) and # Wrap builtin functions only.
attr.__name__ in SLOT_CACHE[type(obj)]):
#functools.wraps(
attr, assigned=functools.WRAPPER_ASSIGNMENTS + ("__self__",))
def wrapper(*args, **kwargs):
if threading.current_thread() is not threading.main_thread():
print("{}.{} was called out of main thread:".format(
type(obj), name), file=sys.stderr)
traceback.print_stack()
return attr(*args, **kwargs)
return wrapper
else:
return attr
QWidget.__getattribute__ = getattribute

Resources