I have read a lot on threads, but I really need help with this one:
I have a PyQt Main GUI that runs an optimization with scipy.minimize...
As I cannot not make an example of this I use a "placeholder" process
to show what my problem is.
I want to let the Main GUI remain stoppable by the User, if the Process takes too long to give
a result.
My working example is this one, where I use an integration with sympy
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
import time, sys
from PyQt4.QtCore import *
from PyQt4.QtGui import *
class IntegrationRunner(QObject):
'Object managing the integration'
def __init__(self):
super(IntegrationRunner, self).__init__()
self._isRunning = True
def longRunning(self):
# reset
if not self._isRunning:
self._isRunning = True
print("preparing Integration")
#this is known to be time consuming and returning not the right integral
#this is a placeholder for a "time consuming" operation that sould
#be handled by a thread outside the main GUI
#imagine an optimization process or any other time consuming operation
#that would freeze the Main GUI
t=sp.symbols('t')
exp=sp.sqrt((3*t+1)/t)
sol=sp.integrate(exp,t)
print(sol)
print('finished...')
def stop(self):
self._isRunning = False
#this is displayed when the "stop button" is clicked
#but actually the integration process won´t stop
print("Integration too long - User terminated")
class SimulationUi(QDialog):
'PyQt interface'
def __init__(self):
super(SimulationUi, self).__init__()
self.goButton = QPushButton('Run Integration')
self.stopButton = QPushButton('Stop if too long')
self.layout = QHBoxLayout()
self.layout.addWidget(self.goButton)
self.layout.addWidget(self.stopButton)
self.setLayout(self.layout)
self.simulThread = QThread()
self.simulThread.start()
self.simulIntegration = IntegrationRunner()
self.simulIntegration.moveToThread(self.simulThread)
#self.simulIntegration.stepIncreased.connect(self.currentStep.setValue)
# call stop on Integr.Runner from this (main) thread on click
self.stopButton.clicked.connect(lambda: self.simulIntegration.stop())
self.goButton.clicked.connect(self.simulIntegration.longRunning)
if __name__ == '__main__':
app = QApplication(sys.argv)
simul = SimulationUi()
simul.show()
sys.exit(app.exec_())
After clicking the "start" button and the "stop" button before the integration stops automatically
i get this output:
>>preparing Integration
>>Integration too long - User terminated
>>Integral(sqrt((3*t + 1)/t), t)
>>finished...
This isn´t exactly my problem but I want to understand how I can use a thread where I can put
time consuming calculations and stop them in order to try other "parameters" or "initial guesses"
when using something like Scipy Minimize
Is it even possible to stop an iterative function from "outside" and restart it without getting into "non responding"?
Any help to improve here is appreciated.
I took these example here as a guideline
how-to-signal-from-a-running-qthread-back-to-...
and pastebin
Related
Please bear with me I am just getting into PyQt6 multithreading so this question might seem too simple and very well might have been answered previously, however I am at an impass and none of the answers or posts about the topic really make anything clear to me. I am looking to be able to click a button it then starts a thread for each task and as each one finishes it updates the progress bar. Here is what I have so far:
from PyQt6.QtWidgets import QApplication, QMainWindow, QPushButton,QVBoxLayout, QWidget, QProgressBar
import sys
from time import sleep
from random import random
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
self.fi = [1,2,3,4,5,6,7,8,9,10] * 100
self.p = None
# add the widgets
self.btn = QPushButton("Execute")
self.btn.pressed.connect(self.start_threads)
self.progress = QProgressBar()
self.progress.setRange(0, len(self.fi))
l = QVBoxLayout()
l.addWidget(self.btn)
l.addWidget(self.progress)
w = QWidget()
w.setLayout(l)
self.setCentralWidget(w)
def task(self,num):
sleep(random())
# where i will run the task using what is passed through
def start_threads(self):
with ThreadPoolExecutor(len(self.fi)) as executor:
futures = []
for i in self.fi:
futures.append(executor.submit(lambda:self.task(i)))
for future in as_completed(futures):
value = self.progress.value()
self.progress.setValue(value + 1)
print(value)
app = QApplication(sys.argv)
w = MainWindow()
w.show()
app.exec()
Whilst this does nearly all my criteria the it only runs 998 of the 1000 tasks it is supposed to. It also makes the GUI freeze which is exactly what I'm trying to avoid by multithreading. I am aware that Qt workers exist however my test for them did not result in any success. I am also aware that this is definetly not how this should be done and thats why I am asking for a more acceptable solution.
I am currently programming a GUI with tkinter in Python 3.6 for another Program, which is called through subprocess. I would like to represent the progress of the Program called, with a Progressbar, popping up in another window as soon as I hit the 'Run Program' Button. The value of the progressbar is set by arguments passed from the subprocess script, where I calculate the percentage of progress in a function. What I need:
1. How do I properly use the Threading-Module for spawning the subprocess?
2. How do I import a function (and therefore its arguments) of a script, that runs with command line parameters?
I was looking for a answer in other posts the whole day, but couldn't really connect them to my problem.
I tried to spawn the subprocess by simply calling its function with threading.Thread(target=self.runProgram()).start()
Apparently the GUI-program still waits until the subprocess/Thread is finished and then continues with creating the progressbar, but I expected it to happen at the same time. I then tried to import the percentage value of the subprocess-Program, which didn't work due to missing parameters. Also I don't think that this would really work the way I want it to, because I want the arguments passed during the runtime, when the subprocess is called. It's very hard to properly explain what I am looking for, so here is some code:
GUI-Program:
import tkinter as tk
from tkinter import ttk
from tkinter import *
from tkinter.filedialog import askdirectory,askopenfilename
import os,subprocess
import threading
class eodApplication(tk.Tk):
def __init__(self):
#different entryboxes are created, not necessary for understanding
self.runButton=''
self.progress=''
self.window = tk.Tk()
self.window.geometry("500x350")
self.window.configure(background="white")
self.window.title("End of Day Application")
self.create_widgets()
def create_widgets(self):
#creating entryboxes initiated in __init__-method, containing paths one can browse for and a submit/'Run Program'-Button
self.submit()
def submit(self):
#Button calls function when pressed
self.runButton=ttk.Button(self.window,text='Run Program',command=lambda:self.bar_init())
self.runButton.grid(row=7, columnspan=2)
def bar_init(self):
barWindow = tk.Tk()
barWindow.geometry("250x150")
barWindow.configure(background="white")
barWindow.title("Progress")
self.progress=ttk.Progressbar(barWindow, orient="horizontal", length=200, mode="determinate",value=0)
self.progress.grid(row=0, column=0, sticky=tk.W)
threading.Thread(target=self.runProgram()).start()
from PythonProgrammeDWH import get_percentage
self.progress["value"] = percentage
def runProgram(self):
devnull = open(os.devnull, 'w')
subprocess.call(f"{self.entryboxPyExe.get()} {self.entryboxPyCode.get()} {self.entrybox64.get()} {self.entrybox66.get()} {self.entryboxXsd.get()} {self.entryboxOutput.get()} {self.entryboxXmlExe.get()}",cwd=os.getcwd(),stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,shell=True)
eodApplication=eodApplication()
eodApplication.window.mainloop()
PythonProgrammeDWH (subprocess):
def main(sys.argv[1],sys.argv[2],sys.argv[3],sys.argv[4]):
~ a lot of stuff going on ~
sumFiles=len(os.listdir(path))
checkedFiles=0
for file in os.listdir(path):
checkedFiles+=1
get_percentage(sumFiles, checkedFiles)
def get_percentage(sumFiles, checkedFiles):
percentage= round((checkedFiles/sumFiles)*100)
return percentage
#percentage is the argument I want to pass to the other script for updating the progressbar
I wrote simple program which has pyQt interface with 2 buttons (start and cancel). Start button runs some calculations in the background (by starting update function) and thanks to threading I can still use UI.
But the application crashes after 10sec - 2 minutes. UI just dissapears, program shutdown.
when I use pythonw to run app without console thread crashes after ~25 sec but gui still works.
#!/usr/bin/python
import threading
import sys
from PyQt4 import QtGui, QtCore
import time
import os
class Class(QtGui.QWidget):
def __init__(self):
#Some init variables
self.initUI()
def initUI(self):
#some UI
self.show()
def update(self,stop_event):
while True and not stop_event.isSet():
self.updateSpeed()
self.updateDistance()
self.printLogs()
self.saveCSV()
self.guiUpdate()
time.sleep(1)
#gui button function
def initiate(self):
self.stop_event = threading.Event()
self.c_thread = threading.Thread(target = self.update, args=(self.stop_event,))
self.c_thread.start()
#Also gui button function
def cancelTracking(self):
self.stop_event.set()
self.close()
def main():
app = QtGui.QApplication(sys.argv)
ex = Class()
sys.exit(app.exec_())
ex.update()
if __name__ == '__main__':
main()
I dont know if I'm doing threading right. I found example like this on stack. I'm quite new to python and I'm using threading for the first time.
It is most likely due to calling a GUI function in your separate thread. PyQt GUI calls like setText() on a QLineEdit are not allowed from a thread. Anything that has PyQt painting outside of the main thread will not work. One way to get around this is to have your thread emit a signal to update the GUI when data is ready. The other way is to have a timer periodically checking for new data and updating the paintEvent after a certain time.
========== EDIT ==========
To Fix this issue I created a library named qt_thread_updater. https://github.com/justengel/qt_thread_updater This works by continuously running a QTimer. When you call call_latest the QTimer will run the function in the main thread.
from qt_thread_updater import get_updater
lbl = QtWidgets.QLabel('Value: 1')
counter = {'a': 1}
def run_thread():
while True:
text = 'Value: {}'.format(counter['a'])
get_updater().call_latest(lbl.setText, text)
counter['a'] += 1
time.sleep(0.1)
th = threading.Thread(target=run_thread)
th.start()
========== END EDIT ==========
#!/usr/bin/python
import threading
import sys
from PyQt4 import QtGui, QtCore
import time
import os
class Class(QtGui.QWidget):
display_update = QtCore.pyqtSignal() # ADDED
def __init__(self):
#Some init variables
self.initUI()
def initUI(self):
#some UI
self.display_update.connect(self.guiUpdate) # ADDED
self.show()
def update(self):
while True and not self.stop_event.isSet():
self.updateSpeed()
self.updateDistance()
self.printLogs()
self.saveCSV()
# self.guiUpdate()
self.display_update.emit() # ADDED
time.sleep(1)
#gui button function
def initiate(self):
self.stop_event = threading.Event()
self.c_thread = threading.Thread(target = self.update)
self.c_thread.start()
#Also gui button function
def cancelTracking(self):
self.stop_event.set()
self.close()
def main():
app = QtGui.QApplication(sys.argv)
ex = Class()
sys.exit(app.exec_())
# ex.update() # - this does nothing
if __name__ == '__main__':
main()
The other thing that could be happening is deadlock from two threads trying to access the same variable. I've read that this shouldn't be possible in python, but I have experienced it from the combination of PySide and other Python C extension libraries.
May also want to join the thread on close or use the QtGui.QApplication.aboutToQuit signal to join the thread before the program closes.
The Qt documentation for QThreads provides two popular patterns for using threading. You can either subclass QThread (the old way), or you can use the Worker Model, where you create a custom QObject with your worker functions and run them in a separate QThread.
In either case, you can't directly update the GUI from the background thread, so in your update function, the guiUpdate call will most likely crash Qt if it tries to change any of the GUI elements.
The proper way to run background processes is to use one of the two QThread patterns and communicate with the main GUI thread via Signals and Slots.
Also, in the following bit of code,
app = QtGui.QApplication(sys.argv)
ex = Class()
sys.exit(app.exec_())
ex.update()
app.exec_ starts the event loop and will block until Qt exits. Python won't run the ex.update() command until Qt has exited and the ex window has already been deleted, so you should just delete that command.
Well, I am familiar with Qt, but when using PyQt, the syntax of signal/slot really confused me.
When using C++/Qt, the compiler will give you a hint where you are wrong about the signal/slot, but the PyQt default configuration doesn't give a hint about error. Is there a ways or such as debug trigger mode to enable PyQt to display more information?
the Code is as following:
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import time
class workThread(QThread):
def __init__(self,parent = None):
super(workThread,self).__init__(parent)
self.mWorkDoneSignal = pyqtSignal() ## some people say this should be defined as clas member, however, I defined it as class member and still fails.
def run(self):
print "workThread start"
time.sleep(1)
print "workThread stop"
print self.emit(SIGNAL("mWorkDoneSignal"))
class MainWidget(QWidget):
def __init__(self , parent = None):
super(MainWidget,self).__init__(parent)
#pyqtSlot()
def display(self):
print "dispaly"
if __name__ == "__main__":
import sys
app = QApplication(sys.argv)
c = workThread()
d = MainWidget()
##In Qt, when using QObject::connect or such things, the return value will show the
## signal/slot binding is success or failed
print QObject.connect(c,SIGNAL('mWorkDoneSignal()'),d,SLOT('display()'))
c.start()
d.show()
app.exec_()
In C++,the QObject::connect return value will show the signal/slot binding is success or not. In PyQt, the return value is True, but it doesn't trigger the slot.
My Question:
1) Is the signal shoud be a class member or instance member?
2) If QObject.connect 's return value can't give the hint of the binding is success or not, is there other ways to detect it?
I want to bind the signal/slot outside the signal sender and slot receiver, so I perfer to use the QObject.connect ways. But how can I write this correct, I tried the following ways,both fail.
QObject.connect(c,SIGNAL('mWorkDoneSignal'),d,SLOT('display'))
QObject.connect(c,SIGNAL('mWorkDoneSignal()'),d,SLOT('display()'))
First, you should really use new style signals with pyqt. In fact, QObject.connect and QObject.emit will not even be there anymore in PyQt5.
def __init__(self,parent = None):
super(workThread,self).__init__(parent)
self.mWorkDoneSignal = pyqtSignal()
This creates an unbound signal and assigns it to a instance variable mWorkDoneSignal, wich dosn't really have an effect. If you want to create an signal, then you really have to declare it on the class.
So if you didn't really create a signal here, then why did this call succeed:
QObject.connect(c,SIGNAL('mWorkDoneSignal()'),d,SLOT('display()'))
The answer lies in the handling of old style signals by PyQt4:
The act of emitting a PyQt4 signal implicitly defines it.
For that reason when you connect a signal to a slot, only the existence of the slot is checked. The signal itself doesn't really need to exist at that point, so the call will always succeed unless the slot doesn't exist.
I tried the following ways,both fail.
QObject.connect(c,SIGNAL('mWorkDoneSignal'),d,SLOT('display'))
QObject.connect(c,SIGNAL('mWorkDoneSignal()'),d,SLOT('display()'))
The first fails because display (without parenthesis) isn't a valid slot.
The second succeeds. The reason it doesn't work is because you emit mWorkDoneSignal, but what you actually need to emit is:
self.emit(SIGNAL("mWorkDoneSignal()"))
Using new style signals, there's no way to mess things like this up:
from utils import sigint
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import time
class workThread(QThread):
mWorkDoneSignal = pyqtSignal()
def __init__(self,parent = None):
super(workThread,self).__init__(parent)
def run(self):
print "workThread start"
time.sleep(1)
print "workThread stop"
self.mWorkDoneSignal.emit()
class MainWidget(QWidget):
def __init__(self , parent = None):
super(MainWidget,self).__init__(parent)
#pyqtSlot()
def display(self):
print "dispaly"
if __name__ == "__main__":
import sys
app = QApplication(sys.argv)
c = workThread()
d = MainWidget()
c.mWorkDoneSignal.connect(d.display)
c.start()
d.show()
app.exec_()
I've written a GUI program which do a lot of calculation in the back, at the same time, display a progress bar. After it was done, a new screen will show the result.
Then I want to make another interface before the calculation to let the user select if they want to use the last calculation result, thus skipping the calculation.
I made a screen with a button which connected to the calculation, and a combo box to select last calculation result file.
However, when I clicked the button, it did nothing. And after around 10 secs( the duration for the calculation ), the result screen pop open. Thus, it skipped the progress bar screen. Why?
This it part of the original program:
import sys
import configparser
import getpass
import telnetlib
import time
import subprocess
from datetime import *
from log_tracker import *
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from PyQt4 import QtCore
from PyQt4 import QtGui
class Task_Checker(QMainWindow):
def __init__(self):
super(Task_Checker, self).__init__()
config = configparser.RawConfigParser()
config.read('profile.cfg')
self.log_path = config.get('Config', 'log_path')
self.log_prefix = config.get('Config', 'log_prefix')
self.log_suffix = config.get('Config', 'log_suffix')
self.initUI()
def check_production(self):
self.log_tracker = Log_Tracker(self)
self.log_tracker.tick.connect(self.pbar.setValue)
self.log_tracker.parseConfig()
self.log_tracker.connectDb()
self.log_tracker.trackLog()
def initUI(self):
self.resize(1400, 768)
self.center()
self.statusBar().showMessage('Checking Production Programs')
self.wait_message = QLabel('Checking Production Programs')
self.wait_message.setAlignment(Qt.Alignment(Qt.AlignHCenter))
self.pbar = QProgressBar(self)
self.pbar.setMinimum(0)
self.pbar.setMaximum(100)
vbox = QVBoxLayout()
vbox.addWidget(self.wait_message)
vbox.addWidget(self.pbar)
tmpWidget2 = QWidget()
tmpWidget2.setLayout(vbox)
self.setCentralWidget(tmpWidget2)
self.show()
self.check_production()
self.pbar.hide()
self.statusBar().showMessage('Processing Information')
self.tabs = QTabWidget()
mua_table = self.processInfo('MUA')
bps_table = self.processInfo('BPS')
obdua_table = self.processInfo('OBDUA')
sua_table = self.processInfo('SUA')
ngr_ftp_table = self.processInfo('NGR_FTP')
bpspdfbill_table = self.processInfo('BpsPdfBill')
disk_space_table = self.processInfo('Disk_Space')
self.tabs.addTab(mua_table, 'MUA')
self.tabs.addTab(bps_table, 'BPS')
self.tabs.addTab(obdua_table, 'ODBUA')
self.tabs.addTab(sua_table, 'SUA')
self.tabs.addTab(ngr_ftp_table, 'NGR_FTP')
self.tabs.addTab(bpspdfbill_table, 'BpsPdfBill')
self.tabs.addTab(disk_space_table, 'Disk_Space')
self.setCentralWidget(self.tabs)
self.statusBar().showMessage('Ready')
self.setWindowTitle('Task Checker')
self.show()
def center(self):
qr = self.frameGeometry()
cp = QDesktopWidget().availableGeometry().center()
qr.moveCenter(cp)
self.move(qr.topLeft())
def processInfo(self, project_name):
...the processing...
In order to add a new screen before the progress bar ( originally it loads the calculation right away ), I made a fews changes to initUI() and move the calculation part to a new sub routine checkProd(), then connect it with a button:
import sys
import configparser
import getpass
import telnetlib
import time
import subprocess
from datetime import *
from log_tracker import *
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from PyQt4 import QtCore
from PyQt4 import QtGui
class Task_Checker(QMainWindow):
def __init__(self):
super(Task_Checker, self).__init__()
config = configparser.RawConfigParser()
config.read('profile.cfg')
self.log_path = config.get('Config', 'log_path')
self.log_prefix = config.get('Config', 'log_prefix')
self.log_suffix = config.get('Config', 'log_suffix')
self.initUI()
def check_production(self):
self.log_tracker = Log_Tracker(self)
self.log_tracker.tick.connect(self.pbar.setValue)
self.log_tracker.parseConfig()
self.log_tracker.connectDb()
self.log_tracker.trackLog()
def initUI(self):
self.resize(1400, 768)
self.center()
btn_check = QPushButton('Check Lastest Status', self)
btn_check.setToolTip('Click this if you want to check the lastest status in production')
combo = QComboBox()
dirlist = os.listdir(self.log_path)
for f in dirlist:
combo.addItem(f)
QtCore.QObject.connect(btn_check, QtCore.SIGNAL('clicked()'), self.checkProd)
hbox = QHBoxLayout()
hbox.addWidget(btn_check)
hbox.addWidget(combo)
tmpWidget = QWidget()
tmpWidget.setLayout(hbox)
self.setCentralWidget(tmpWidget)
self.show()
def checkProd(self):
self.statusBar().showMessage('Checking Production Programs')
self.wait_message = QLabel('Checking Production Programs')
self.wait_message.setAlignment(Qt.Alignment(Qt.AlignHCenter))
self.pbar = QProgressBar(self)
self.pbar.setMinimum(0)
self.pbar.setMaximum(100)
vbox = QVBoxLayout()
vbox.addWidget(self.wait_message)
vbox.addWidget(self.pbar)
tmpWidget2 = QWidget()
tmpWidget2.setLayout(vbox)
self.setCentralWidget(tmpWidget2)
self.show()
self.check_production()
self.pbar.hide()
self.statusBar().showMessage('Processing Information')
self.tabs = QTabWidget()
mua_table = self.processInfo('MUA')
bps_table = self.processInfo('BPS')
obdua_table = self.processInfo('OBDUA')
sua_table = self.processInfo('SUA')
ngr_ftp_table = self.processInfo('NGR_FTP')
bpspdfbill_table = self.processInfo('BpsPdfBill')
disk_space_table = self.processInfo('Disk_Space')
self.tabs.addTab(mua_table, 'MUA')
self.tabs.addTab(bps_table, 'BPS')
self.tabs.addTab(obdua_table, 'ODBUA')
self.tabs.addTab(sua_table, 'SUA')
self.tabs.addTab(ngr_ftp_table, 'NGR_FTP')
self.tabs.addTab(bpspdfbill_table, 'BpsPdfBill')
self.tabs.addTab(disk_space_table, 'Disk_Space')
self.setCentralWidget(self.tabs)
self.statusBar().showMessage('Ready')
self.setWindowTitle('Task Checker')
self.show()
def center(self):
qr = self.frameGeometry()
cp = QDesktopWidget().availableGeometry().center()
qr.moveCenter(cp)
self.move(qr.topLeft())
def processInfo(self, project_name):
...the processing...
This is a lot of unnecessary code to read, but what I have managed to parse out of it, is that you are doing heavy calculations in the main thread, while expecting another widget in the main thread to also maintain responsiveness.
This is a common pitfall when working with PyQt4 (and probably most GUI frameworks that use a main thread event loop). The issue is that when you start your app, the main event loop constantly polls for new events to process from your GUI. Its expecting that every operation should either take a very short time to complete, or, periodically relinquish control back to the eventloop. Right now, your calculation is hogging up all the availability of the main thread, and anything that your progress window is trying to do is just backing up into a queue waiting for a chance for the event loop to pick it up.
The simple answer to this is: Do any heavy calculations in a separate QThread, and communicate back to your main thread via signals. This will require a little bit of retooling on you part. You should not put anything heavy into an init of any class. But you have managed to address this in your second example by attaching the computation to a button. Good start. The button should actually start the calc in another thread, thus not blocking up the main thread.
Qt4.7 Threading Basics
GUI Thread and Worker Thread
As mentioned, each program has one thread
when it is started. This thread is called the "main thread" (also
known as the "GUI thread" in Qt applications). The Qt GUI must run in
this thread. All widgets and several related classes, for example
QPixmap, don't work in secondary threads. A secondary thread is
commonly referred to as a "worker thread" because it is used to
offload processing work from the main thread.
...
Using Threads
There are basically two use cases for threads:
1. Make
processing faster by making use of multicore processors.
2. Keep the GUI
thread or other time critical threads responsive by offloading long
lasting processing or blocking calls to other threads.
As a quick fix if you just want to see some results, you can periodically call QtGui.QApplication.processEvents() from your calculation method. This will every so often allow the event loop to flush out pending operations. Doing so will let something like your progress widget actually function. Essentially what you are doing is manually "pumping" the event loop. This isn't the best approach though. Usually its reserved for lighter weight one off stuff. If you want the best performance, move it to a thread.