Trying to implement multiprocessing within a PyQt5 GUI framework to process multiple files in a parallel process, preferably in a QThread() background. Running the code below seems to create the multiple processes but has additional issue that multiple GUI windows appear and everything seems locked out at that point.
import multiprocess as mp
from PyQt5.QtWidgets import QMainWindow
class MyApp(QMainWindow, myUiMainWindow):
def __init__(self):
super(self.__class__, self).__init()
self.setupUI(self)
self.pushButton.clicked.connect(self.doMyStuff)
def consumer(self, inQ, outQ):
val = inQ.get()
ret = self.process_single(val)
outQ.put(ret)
def process_single(self, f):
<process each file f>
<update progress bar>
return f
def doMyStuff(self):
<get file_list from GUI Widget>
n_w = len(file_list) if len(file_list) < 5 else 5
inQ = mp.Queue()
outQ = mp.Queue()
workers = [mp.Process(target=consumer, args=(inQ, outQ) for i in range(n_w)]
[w.start() for w in workers]
[inQ.put(f) for f in file_list]
[inQ.put(None) for i in range(n_w)]
completed_files = []
while len(completed_files) != len(file_list):
completed_files.append(outQ.get())
[w.join() for w in workers]
Guess what...
implement the following to build the GUI only once:
import multiprocess as mp
from PyQt5 import QtWidgets
from QtWidgets import QMainWindow
... snippet ... your script code ...
if __name__ == '__main__':
app = QtWidgets.QApplication(sys.argv)
window = MyApp()
window.setGeometry(300, 100, 600, 600) # adjustable (left,top,width,height)
window.show()
sys.exit(app.exec_())
... you probably figured this one out by now ;p
Related
Please bear with me I am just getting into PyQt6 multithreading so this question might seem too simple and very well might have been answered previously, however I am at an impass and none of the answers or posts about the topic really make anything clear to me. I am looking to be able to click a button it then starts a thread for each task and as each one finishes it updates the progress bar. Here is what I have so far:
from PyQt6.QtWidgets import QApplication, QMainWindow, QPushButton,QVBoxLayout, QWidget, QProgressBar
import sys
from time import sleep
from random import random
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
self.fi = [1,2,3,4,5,6,7,8,9,10] * 100
self.p = None
# add the widgets
self.btn = QPushButton("Execute")
self.btn.pressed.connect(self.start_threads)
self.progress = QProgressBar()
self.progress.setRange(0, len(self.fi))
l = QVBoxLayout()
l.addWidget(self.btn)
l.addWidget(self.progress)
w = QWidget()
w.setLayout(l)
self.setCentralWidget(w)
def task(self,num):
sleep(random())
# where i will run the task using what is passed through
def start_threads(self):
with ThreadPoolExecutor(len(self.fi)) as executor:
futures = []
for i in self.fi:
futures.append(executor.submit(lambda:self.task(i)))
for future in as_completed(futures):
value = self.progress.value()
self.progress.setValue(value + 1)
print(value)
app = QApplication(sys.argv)
w = MainWindow()
w.show()
app.exec()
Whilst this does nearly all my criteria the it only runs 998 of the 1000 tasks it is supposed to. It also makes the GUI freeze which is exactly what I'm trying to avoid by multithreading. I am aware that Qt workers exist however my test for them did not result in any success. I am also aware that this is definetly not how this should be done and thats why I am asking for a more acceptable solution.
I want use a parallel downloading videos from youtube, but my code ending with exception "PicklingError". Can you help guys with code, how it should be, please.
Another fixed variant:
import sys
#from pathos.multiprocessing import ProcessingPool as Pool
from multiprocessing import Pool
from pytube import YouTube
from youtubeMultiDownloader import UiMainWindow
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtWidgets import QFileDialog
class YouTubeInstance:
def __init__(self, path):
self.youtube = YouTube
self.path = path
#self.ui_obj = ui_obj
def download_file(self, url):
self.youtube(url).streams.get_highest_resolution().download(self.path)
#self.ui.ui.youtube_outputs.setText(f'Video \'{self.youtube.title}\' has been downloaded successfully!')
class YouTubeMultiDownloader(QtWidgets.QMainWindow):
def __init__(self):
super().__init__()
self.pool = Pool
self.ui = UiMainWindow()
self.ui.setup_ui(self)
self.path_to_dir = None
self.urls = None
def _get_urls_from_form(self):
self.urls = self.ui.youtube_urls.toPlainText().split('\n')
return len(self.urls)
def choose_directory(self):
self.path_to_dir = str(QFileDialog.getExistingDirectory(self, "Select Directory"))
def run_multi_downloads(self):
youtube = YouTubeInstance(self.path_to_dir)
self.pool(self._get_urls_from_form()).map(youtube.download_file, self.urls)
if __name__ == "__main__":
app = QtWidgets.QApplication([])
application = YouTubeMultiDownloader()
application.show()
sys.exit(app.exec_())
Updated:
My ui :)
Error 1 fixed:
Error 2 fixed:
Error 3 actual:
You've got the wrong side of the stick. Take a look at multiprocessing module documents. As it says, calling Pool method is for running multiple instance of same function simultaneously (in parallel). So call Pool method as many numbers you want, meanwhile your method does not any parameters, call it without any arguments:
with Pool(5) as p:
print(p.map(YouTubeMultiDownloader))
It create 5 parallel instance. You can change the code an refine your errors.
I am new to python and multiprocessing (Electronics background). I am writing an GUI application to communicate with micro controller over serial port. The interaction with user is through some buttons on the GUI, which sends some command on serial port to the MCU and displays data received from MCU in the same GUI(there can be data from MCU without any command being sent). I am using PyQt5 module and QT designer to design the UI. I converted the UI to py code using uic.compileUi. I am then calling this python file in my main code.
After reading through several documents and stack overflow posts I decided to use to QThreads for the application.
I am however facing issue with serial communication when being used inside thread (It works fine if used in a single main loop without threads).
My code is:
from PyQt5 import uic
from PyQt5.QtWidgets import QApplication,QWidget,QMainWindow #
from PyQt5.QtCore import QCoreApplication, QObject, pyqtSignal, pyqtSlot, QThread
import serial
from GUI import Ui_MainGUIWindow
import sys
serialString = ""
def ClosePort():
serialPort.close()
# print("port closed")
def OpenPort():
serialPort.open()
#print("port open")
def sendFV():
serialPort.write('fv\r'.encode())
def configSerial(port,baudRate):
global serialPort
serialPort=serial.Serial(port = port, baudrate=baudRate, bytesize=8, timeout=None, stopbits=serial.STOPBITS_ONE)
if not serialPort.isOpen():
serialPort.open()
class Worker(QObject):
finished = pyqtSignal()
dataReady = pyqtSignal(['QString'])#[int], ['Qstring']
print("workerinit")
def run(self):
print ("worker run")
while(1):
if(serialPort.in_waiting > 0):
serialString = serialPort.read()
self.dataReady.emit(str(serialString))
self.finished.emit()
class GUI(QWidget):
def __init__(self):
self.obj = Worker()
self.thread = QThread()
self.obj.dataReady.connect(self.onDataReady)
self.obj.moveToThread(self.thread)
self.obj.finished.connect(self.thread.quit)
self.thread.started.connect(self.obj.run)
self.thread.finished.connect(app.exit)
self.thread.start()
self.initUI()
def initUI(self):
#Form, Window = uic.loadUiType("TestGUI.ui")
#print(Form,Window)
#self.window = Window()
#self.form=Form()
self.window = QMainWindow()
self.form = Ui_MainGUIWindow()
self.form.setupUi(self.window)
self.form.FVButton.clicked.connect(sendFV)
print("guiinit")
self.window.show()
def onDataReady(self, serialString):
print("here")
print(serialString.decode('Ascii'))
self.form.plainTextOutput.insertPlainText((serialString.decode('Ascii')))
self.form.plainTextOutput.ensureCursorVisible()
if __name__ == '__main__':
configSerial("COM20",9600)
"""
app = QCoreApplication.instance()
if app is None:
app = QCoreApplication(sys.argv)
"""
app = QApplication(sys.argv)
form=GUI()
app.exec_()
sys.exit(ClosePort())
I get following error on line if(serialPort.in_waiting > 0):
File "C:\Users\...\Anaconda3\lib\site-packages\serial\serialwin32.py", line 259, in in_waiting
raise SerialException("ClearCommError failed ({!r})".format(ctypes.WinError()))
SerialException: ClearCommError failed (OSError(9, 'The handle is invalid.', None, 6))
I found this post talking about similar issue, but do know how exactly to implement the suggested solution in my code.
Also I get "kernel died" error multiple times when I run the code on spyder IDE. I added a check if I am creating multiple instances of QT application but that did not help.Running the code from anaconda command prompt works fine.
First of all, I'm currently migrating my source code from PyQt5 to PySide2 which requires me to change some of the syntaxes. As this site said that it only needs 3 things to do migrate from PyQt to Pyside2.
1.app.exec_. exec_ was used as exec is a Python2 keyword. Under Python3, PyQt5 allows the use of exec but not PySide2.
2.Under PyQt5 it’s QtCore.pyqtSignal and QtCore.pyqtSlot and under PySide2 it’s QtCore.Signal and QtCore.Slot .
3.loading Ui files.
But anyway later on when I tried to run my code it gave me this following error:
QThread: Destroyed while thread is still running
I had more than 2000 lines of code and I cannot determine which is the cause of this other than my last action which is trying to call QFileDialog which shouldn't be a problem (I've tested this with PyQt import and there's no problem and no warning at all). But in PySide2 it definitely might be the cause of it. I look up into this, he doesn't have the same problem as mine exactly. I'm not trying to call QFileDialog from different thread.
this is the minimal reproducible example of my working code in PyQt5:
import sys
import os
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QFileDialog, QMessageBox, QWidget, QDialog
import random
class MyWidget(QtWidgets.QWidget):
def __init__(self):
QtWidgets.QWidget.__init__(self)
self.path = os.path.abspath(os.path.dirname(sys.argv[0]))
self.button = QtWidgets.QPushButton("Open File")
self.labelFile = QtWidgets.QLabel("empty")
self.labelData = QtWidgets.QLabel("None")
self.layout = QtWidgets.QVBoxLayout()
self.layout.addWidget(self.button)
self.layout.addWidget(self.labelFile)
self.layout.addWidget(self.labelData)
self.setLayout(self.layout)
self.button.clicked.connect(self.open_file)
timer = QtCore.QTimer(self)
timer.timeout.connect(self.update_data_value)
timer.start(1000)
def open_file(self):
x = QFileDialog.getOpenFileName(self,"Pilih File CSV yang Ingin Diproses",self.path,"CSV Files (*.csv)")
self.labelFile.setText(x[0])
def update_data_value(self):
self.DataProcess = DataProcess()
self.DataProcess.progress.connect(self.update_data_label)
self.DataProcess.start()
def update_data_label(self,x):
self.labelData.setText(str(x[0]))
class DataProcess(QtCore.QThread):
progress = QtCore.pyqtSignal(object)
def __init__(self):
QtCore.QThread.__init__(self)
def run(self):
x = random.randint(1,100)
self.progress.emit([str(x)+ " from thread"])
if __name__ == "__main__":
app = QtWidgets.QApplication(sys.argv)
widget = MyWidget()
widget.show()
sys.exit(app.exec_())
and this is the non-working one in PySide2 after renaming import accordingly to PySide2 also renaming 'pyqtsignal' to 'Signal'
import sys
import os
from PySide2 import QtCore, QtGui, QtWidgets
from PySide2.QtWidgets import QMainWindow, QFileDialog, QMessageBox, QWidget, QDialog
import random
class MyWidget(QtWidgets.QWidget):
def __init__(self):
QtWidgets.QWidget.__init__(self)
self.path = os.path.abspath(os.path.dirname(sys.argv[0]))
self.button = QtWidgets.QPushButton("Open File")
self.labelFile = QtWidgets.QLabel("empty")
self.labelData = QtWidgets.QLabel("None")
self.layout = QtWidgets.QVBoxLayout()
self.layout.addWidget(self.button)
self.layout.addWidget(self.labelFile)
self.layout.addWidget(self.labelData)
self.setLayout(self.layout)
self.button.clicked.connect(self.open_file)
timer = QtCore.QTimer(self)
timer.timeout.connect(self.update_data_value)
timer.start(1000)
def open_file(self):
x = QFileDialog.getOpenFileName(self,"Pilih File CSV yang Ingin Diproses",self.path,"CSV Files (*.csv)")
self.labelFile.setText(x[0])
def update_data_value(self):
self.DataProcess = DataProcess()
self.DataProcess.progress.connect(self.update_data_label)
self.DataProcess.start()
def update_data_label(self,x):
self.labelData.setText(str(x[0]))
class DataProcess(QtCore.QThread):
progress = QtCore.Signal(object)
def __init__(self):
QtCore.QThread.__init__(self)
def run(self):
x = random.randint(1,100)
self.progress.emit([str(x)+ " from thread"])
if __name__ == "__main__":
app = QtWidgets.QApplication(sys.argv)
widget = MyWidget()
widget.show()
sys.exit(app.exec_())
so after creating this minimal example, I realized that PySide QFileDialog makes the QThread stop while PyQt QFileDialog doesn't freeze the main thread. Is there anything I could do to handle this in similar syntax architecture? (e.g not using "movetothread" or "QObject")
The problem is that you're overwriting self.DataProcess every time a new thread is created, which may cause the previous object to be garbage-collected by Python before Qt has a chance to delete it. This can result in a core-dump if Qt tries to delete an object which is no longer there. Problems of this kind are quite common in both PyQt and PySide, and are usually caused by not keeping proper references to dependant objects. The normal solution is to ensure that the affected objects are given a parent and, if necessary, explicitly delete them at an appropriate time.
Here is one way to fix your example:
class MyWidget(QtWidgets.QWidget):
...
def update_data_value(self):
# ensure the thread object has a parent
process = DataProcess(self)
process.progress.connect(self.update_data_label)
process.start()
def update_data_label(self,x):
self.labelData.setText(str(x[0]))
class DataProcess(QtCore.QThread):
progress = QtCore.Signal(object)
def __init__(self, parent):
# ensure the thread object has a parent
QtCore.QThread.__init__(self, parent)
def run(self):
x = random.randint(1,100)
self.progress.emit([str(x)+ " from thread"])
# explicitly schedule for deletion
self.deleteLater()
It's hard to say exactly why PySide behaves differently to PyQt in this particular case. It usually just comes down to low-level differences between the two implementations. Probably there are equivalent cases that affect PyQt but not PySide. However, if you manage object references and cleanup carefully, such differences can usually be eliminated.
I try to create a PyQt application. In order to run process in background and keep the PyQt5 application available for new instruction, I want to use multiprocessing.
On the Windows OS, when I call a function from the Qt MainWindow class with multiprocessing.process, I have an error about pickling this class. But it is running find on Linux.
Here is an example:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sys
import multiprocessing
#
from PyQt5.QtWidgets import *
from PyQt5.QtGui import *
from PyQt5.QtCore import *
class Frame_main(QMainWindow):
def __init__(self, parent = None):
super(Frame_main, self).__init__(parent)
self.thrd_list=[]
self.initUI()
def initUI(self):
# Button
btn_run = QPushButton('Run', self)
btn_run.clicked.connect(lambda: self.ThdSomething(self.DoRun) ) #
btn_stop = QPushButton('Stop', self)
btn_stop.clicked.connect(self.DoStop)
### TEXT Edit
self.textEdit_term = QTextEdit("terminal: \n ")
self.textEdit_term.append("")
self.textEdit_term.setStyleSheet("color: rgb(255, 255, 255); background-color: black;")
self.textEdit_term.setLineWrapMode(QTextEdit.NoWrap)
self.textEdit_term.setToolTip(' terminal message ')
self.textEdit_term.setStatusTip('textEdit1')
### LAYOUT
Wid_main = QWidget() #
grid_major = QGridLayout() #
grid_major.addWidget( btn_run, 1, 5)
grid_major.addWidget( btn_stop, 2, 5)
Wid_main.setLayout(grid_major)
self.setCentralWidget(Wid_main)
### Affichage
self.show() #
#
def DoRun(self):
print('g starts')
time_start=time.time()
time_stop=time.time()
name='bob'
n=0
while time_stop-time_start <2 :
n+=1
time_stop=time.time()
time.sleep(0.8)
print ('hola', name,n, flush=True)
print('g stops')
def DoStop(self):
''' subourtine kill all the thread '''
print('stop action detected')
while len(self.thrd_list) > 0 :
print("Terminating the job: {}".format(self.thrd[-1].pid) )
os.kill(self.thrd[-1].pid, signal.SIGTERM)
self.thrd_list[-1].terminate()
self.thrd_list.pop()
def ThdSomething(self, job):
''' subourtine to create a new thread to do the job subroutine '''
arg=''
p=multiprocessing.Process(target=job, args=(arg))
self.thrd_list.append( p )
p.start()
print("Start the job GUI: {} with PID: {}".format(str(job) ,self.thrd[-1].pid), file=sys.stdout )
def closeEvent(self, event):
''' subroutine to define what happen when closing the main frame'''
self.statusBar().showMessage('waiting for a respond')
reply = QMessageBox.question(self, 'Message',
"Are you sure to quit?", QMessageBox.Yes |
QMessageBox.No, QMessageBox.No)
if reply == QMessageBox.Yes:
event.accept()
else:
event.ignore()
# Main
if __name__ == '__main__':
# Create QApplication and Main Frame
Qapp = QApplication(sys.argv) # creation de lappli Qt
Qapp.setStyle("fusion") #
frame1 = Frame_main() #
sys.exit(Qapp.exec_()) #
EDIT:
I found several similar case, like :
Python: multiprocessing in pyqt application
but none of them helped. I think this might be linked to fact that my case is using function and attributes of the MainWindow class.
You are correct that this is due to the attempt to fork on a method within the GUI class. Unfortunately windows doesn't really have the ability to fork processes the way linux does, which is why your code works on linux and not on windows. The Python 3 documentation for the multiprocessing library has some useful information on the differences and what the default method for starting a new process is under different platforms.
To answer your question: because your target is a method associated with a GUI object, it has to send the object state to the new process but fails to pickle it because a PyQt GUI object is far too complex to pickle.
If you rewrite your code so that the target function (and args specified) are picklable, your code will work.