I have code that should show communication between tkinter widget (NOTE: not implemented yet) and another thread. As communication between those two I choose python queue. To see what is really happening print is shown in console and it's not what I would expect.
As can be seen in console output after sleep time in generate_text output from process is shown. What I expected is that as generate_text is slower then process I would see a lot more process is called then Item x, but this is not happening.
import tkinter as tk
import threading
import queue
import time
def generate_text(storage):
count = 0
while True:
message = "Item {}".format(count)
storage.put(message)
print(message)
count +=1
time.sleep(3000/1000)
def process(storage):
print("process is called")
try:
storage.get()
except queue.Empty:
print("queue empty")
# register awake function
root.after(500, process, message)
# init variables
message = queue.Queue()
root = tk.Tk()
t = threading.Thread(target=generate_text, args=(message,))
t.setDaemon(True)
t.start()
root.after(500, process, message)
root.mainloop()
Output:
Item 0
process is called
process is called
Item 1
process is called
Item 2
process is called
Item 3
process is called...
Desired output:
Item 0
process is called
process is called
process is called
process is called
process is called
process is called
Item 1
#Himal's answer is correct for the current question however you might want to amend this to use event_generate in the message generating code and have the UI respond to the events when they are raised rather than polling the queue like this. You can use root.event_generate('<<MessageQueued>>') in the generate_text fuction to place a virtual event onto Tk's event queue. This is thread safe where calling window methods directly is not. If you also add a binding to this virtual event on the UI code then the Tk message loop with call the bound function when it receives the virtual event. No more polling.
import tkinter as tk
import threading
import queue
import time
def generate_text(mainwin, storage):
count = 0
while True:
message = "Item {}".format(count)
storage.put(message)
print("Queued {0}".format(message))
count += 1
mainwin.event_generate('<<MessageGenerated>>')
time.sleep(3000/1000)
def process(storage, event):
msg = storage.get()
print("New message: {0}".format(msg))
def main():
message_queue = queue.Queue()
root = tk.Tk()
root.bind('<<MessageGenerated>>', lambda e: process(message_queue, e))
t = threading.Thread(target=generate_text, args=(root, message_queue,))
t.setDaemon(True)
t.start()
root.mainloop()
if __name__ == '__main__':
main()
storage.get() is a blocking function. it won't reach the root.after(500, process, message) call until there's an item in the queue.
You could use storage.get_nowait() or storage.get(False) to get the desired behavior.
More about Queue
Related
I can use QStatusBar to display a message by feeding it a single string, e.g.:
self.statusBar().showMessage("My message here, also show it for 1 sec", 1000)
In my event-loop, the above message will be shown on the status-bar for exactly 1000 ms. But say that instead of a string "My message here", I have a list of strings I want to display one at the time. I can do that via a loop by giving it a delay via time.sleep(1) - but that would unfortunately block the gui until the loop is over, and I don't want that. Can I separate the processes of main event loop and status bar updating, and if so, how?
The example code below has a button, which when pressed, generates three different messages stored in a list. They are then shown in the status-bar, one at a time, and the button cannot be pressed until the loop ends. The behaviour that I want is that the button is clickable (gui isn't blocked), and if it's clicked while the previous messages are showing, the showing is interrupted and new messages are shown.
---Example code below---
import sys
import time
from PyQt5 import QtWidgets
class Window(QtWidgets.QMainWindow):
"""Main Window."""
MSG_ID = 1
def __init__(self, parent=None):
"""Initializer."""
super().__init__(parent)
#stuff
self.messages = []
self.setWindowTitle('Main Window')
self.setStatusBar(QtWidgets.QStatusBar())
self.statusBar().showMessage("My message first time")
self.button = QtWidgets.QPushButton("Test",self)
self.button.clicked.connect(self.handleButton)
def handleButton(self):
self.messages = [f"message_num {msg_id}" for msg_id in range(Window.MSG_ID,Window.MSG_ID+3)]
Window.MSG_ID+=3
self.updateStatus()
print(self.messages)
def updateStatus(self):
self.statusBar().clearMessage()
for item in self.messages:
self.statusBar().showMessage(item,1000)
time.sleep(1.2)
if __name__ == '__main__':
app = QtWidgets.QApplication(sys.argv)
win = Window()
win.show()
sys.exit(app.exec_())
The above code creates a list of messages
There is no need to use timers or sleep for this, because the status-bar sends a messageChanged signal every time a temporary message changes (including when it's removed). This can be used to create a simple call-back loop that does what you want:
class Window(QtWidgets.QMainWindow):
...
def __init__(self, parent=None):
...
self.statusBar().messageChanged.connect(self.updateStatus)
def handleButton(self):
self.messages = [f"message_num {msg_id}" for msg_id in range(Window.MSG_ID,Window.MSG_ID+3)]
Window.MSG_ID+=3
self.updateStatus()
def updateStatus(self, message=None):
print('update-status:', (message, self.messages))
if not message and self.messages:
self.statusBar().showMessage(self.messages.pop(0), 1000)
This does not block the gui and allows the message sequence to be re-started at any time by clicking the button.
I am trying to figure out why this code crashes if I try to run the threads for a second time once they are completed.
The first time I click "Start 5 Threads" It runs just fine and finishes. But if I click it again. The entire program crashes and I get the QThread: Destroyed while thread is still running Error
This code was found on the web. I am trying to learn from it.
import time
import sys
from PyQt5.QtCore import QObject, QThread, pyqtSignal, pyqtSlot
from PyQt5.QtWidgets import QApplication, QPushButton, QTextEdit, QVBoxLayout, QWidget
def trap_exc_during_debug(*args):
# when app raises uncaught exception, print info
print(args)
# install exception hook: without this, uncaught exception would cause application to exit
sys.excepthook = trap_exc_during_debug
class Worker(QObject):
"""
Must derive from QObject in order to emit signals, connect slots to other signals, and operate in a QThread.
"""
sig_step = pyqtSignal(int, str) # worker id, step description: emitted every step through work() loop
sig_done = pyqtSignal(int) # worker id: emitted at end of work()
sig_msg = pyqtSignal(str) # message to be shown to user
def __init__(self, id: int):
super().__init__()
self.__id = id
self.__abort = False
#pyqtSlot()
def work(self):
"""
Pretend this worker method does work that takes a long time. During this time, the thread's
event loop is blocked, except if the application's processEvents() is called: this gives every
thread (incl. main) a chance to process events, which in this sample means processing signals
received from GUI (such as abort).
"""
thread_name = QThread.currentThread().objectName()
thread_id = int(QThread.currentThreadId()) # cast to int() is necessary
self.sig_msg.emit('Running worker #{} from thread "{}" (#{})'.format(self.__id, thread_name, thread_id))
for step in range(100):
time.sleep(0.1)
self.sig_step.emit(self.__id, 'step ' + str(step))
# check if we need to abort the loop; need to process events to receive signals;
app.processEvents() # this could cause change to self.__abort
if self.__abort:
# note that "step" value will not necessarily be same for every thread
self.sig_msg.emit('Worker #{} aborting work at step {}'.format(self.__id, step))
break
self.sig_done.emit(self.__id)
def abort(self):
self.sig_msg.emit('Worker #{} notified to abort'.format(self.__id))
self.__abort = True
class MyWidget(QWidget):
NUM_THREADS = 5
# sig_start = pyqtSignal() # needed only due to PyCharm debugger bug (!)
sig_abort_workers = pyqtSignal()
def __init__(self):
super().__init__()
self.setWindowTitle("Thread Example")
form_layout = QVBoxLayout()
self.setLayout(form_layout)
self.resize(400, 800)
self.button_start_threads = QPushButton()
self.button_start_threads.clicked.connect(self.start_threads)
self.button_start_threads.setText("Start {} threads".format(self.NUM_THREADS))
form_layout.addWidget(self.button_start_threads)
self.button_stop_threads = QPushButton()
self.button_stop_threads.clicked.connect(self.abort_workers)
self.button_stop_threads.setText("Stop threads")
self.button_stop_threads.setDisabled(True)
form_layout.addWidget(self.button_stop_threads)
self.log = QTextEdit()
form_layout.addWidget(self.log)
self.progress = QTextEdit()
form_layout.addWidget(self.progress)
QThread.currentThread().setObjectName('main') # threads can be named, useful for log output
self.__workers_done = None
self.__threads = None
def start_threads(self):
self.log.append('starting {} threads'.format(self.NUM_THREADS))
self.button_start_threads.setDisabled(True)
self.button_stop_threads.setEnabled(True)
self.__workers_done = 0
self.__threads = []
for idx in range(self.NUM_THREADS):
worker = Worker(idx)
thread = QThread()
thread.setObjectName('thread_' + str(idx))
self.__threads.append((thread, worker)) # need to store worker too otherwise will be gc'd
worker.moveToThread(thread)
# get progress messages from worker:
worker.sig_step.connect(self.on_worker_step)
worker.sig_done.connect(self.on_worker_done)
worker.sig_msg.connect(self.log.append)
# control worker:
self.sig_abort_workers.connect(worker.abort)
# get read to start worker:
# self.sig_start.connect(worker.work) # needed due to PyCharm debugger bug (!); comment out next line
thread.started.connect(worker.work)
thread.start() # this will emit 'started' and start thread's event loop
# self.sig_start.emit() # needed due to PyCharm debugger bug (!)
#pyqtSlot(int, str)
def on_worker_step(self, worker_id: int, data: str):
self.log.append('Worker #{}: {}'.format(worker_id, data))
self.progress.append('{}: {}'.format(worker_id, data))
#pyqtSlot(int)
def on_worker_done(self, worker_id):
self.log.append('worker #{} done'.format(worker_id))
self.progress.append('-- Worker {} DONE'.format(worker_id))
self.__workers_done += 1
if self.__workers_done == self.NUM_THREADS:
self.log.append('No more workers active')
self.button_start_threads.setEnabled(True)
self.button_stop_threads.setDisabled(True)
# self.__threads = None
#pyqtSlot()
def abort_workers(self):
self.sig_abort_workers.emit()
self.log.append('Asking each worker to abort')
for thread, worker in self.__threads: # note nice unpacking by Python, avoids indexing
thread.quit() # this will quit **as soon as thread event loop unblocks**
thread.wait() # <- so you need to wait for it to *actually* quit
# even though threads have exited, there may still be messages on the main thread's
# queue (messages that threads emitted before the abort):
self.log.append('All threads exited')
if __name__ == "__main__":
app = QApplication([])
form = MyWidget()
form.show()
sys.exit(app.exec_())
The problem is solved by passing him as a parent to self. You must change:
thread = QThread()
to:
thread = QThread(parent=self)
I am displaying a sensor device's measurement values using the Tkinter GUI application. The sensor sends new data every second. I started a new thread and put the result in a Queue and process the queue to display the values on GUI. Now I am facing another problem. The sensor has two modes of operations. For simplicity, I have used a random generator in the program. Users should be able to switch the modes using two buttons. Button-1 for Mode-1, Button-2 for Mode-2. (say the mode-1 operation is random.randrange(0,10) and mode-2 operation is random.randrange(100, 200). How do I control these two operations through Threading? if a user started a mode-1 operation, when he presses the Button-2, the mode-1 operation should stop (thread-1) and mode-2 operation (thread-2) should start. Does it mean do I need to kill thread-1? Or is there any way to control two modes in same thread? I am totally new into threading. Any suggestions, please.
import tkinter
import threading
import queue
import time
import random
class GuiGenerator:
def __init__(self, master, queue):
self.queue = queue
# Set up the GUI
master.geometry('800x480')
self.output = tkinter.StringVar()
output_label = tkinter.Label(master, textvariable= self.output)
output_label.place(x=300, y=200)
#I haven't shown command parts in following buttons. No idea how to use it to witch modes?
mode_1_Button = tkinter.Button(master, text = "Mode-1")
mode_1_Button.place(x=600, y=300)
mode_2_Button = tkinter.Button(master, text = "Mode-2")
mode_2_Button.place(x=600, y=400)
def processQueue(self):
while self.queue.qsize():
try:
sensorOutput = self.queue.get() #Q value
self.output.set(sensorOutput) #Display Q value on GUI
except queue.Empty:
pass
class ClientClass:
def __init__(self, master):
self.master = master
# Create the queue
self.queue = queue.Queue()
# Set up the GUI part
self.myGui = GuiGenerator(master, self.queue)
#How do I switch the modes of operations? do I need some flags setting through button press?
# Set up the thread to do asynchronous I/O
self.thread1_mode1 = threading.Thread(target=self.firstModeOperation)
self.thread1_mode1.start()
# Start the periodic call in the GUI to check if the queue contains
# anything new
self.periodicCall()
def periodicCall(self):
# Check every 1000 ms if there is something new in the queue.
self.myGui.processQueue()
self.master.after(1000, self.periodicCall)
def firstModeOperation(self):
while True: #??? how do i control here through mode selection
time.sleep(1.0)
msg_mode_1= random.randrange(0,10)
self.queue.put(msg_mode_1)
def secondModeOperation(self):
while True: #??? how do i control here through mode selection
time.sleep(1.0)
msg_mode_2= random.randrange(100,200)
self.queue.put(msg_mode_2)
#Operation part
root = tkinter.Tk()
client = ClientClass(root)
root.mainloop()
I am following the principles laid down in this post to safely output the results which will eventually be written to a file. Unfortunately, the code only print 1 and 2, and not 3 to 6.
import os
import argparse
import pandas as pd
import multiprocessing
from multiprocessing import Process, Queue
from time import sleep
def feed(queue, parlist):
for par in parlist:
queue.put(par)
print("Queue size", queue.qsize())
def calc(queueIn, queueOut):
while True:
try:
par=queueIn.get(block=False)
res=doCalculation(par)
queueOut.put((res))
queueIn.task_done()
except:
break
def doCalculation(par):
return par
def write(queue):
while True:
try:
par=queue.get(block=False)
print("response:",par)
except:
break
if __name__ == "__main__":
nthreads = 2
workerQueue = Queue()
writerQueue = Queue()
considerperiod=[1,2,3,4,5,6]
feedProc = Process(target=feed, args=(workerQueue, considerperiod))
calcProc = [Process(target=calc, args=(workerQueue, writerQueue)) for i in range(nthreads)]
writProc = Process(target=write, args=(writerQueue,))
feedProc.start()
feedProc.join()
for p in calcProc:
p.start()
for p in calcProc:
p.join()
writProc.start()
writProc.join()
On running the code it prints,
$ python3 tst.py
Queue size 6
response: 1
response: 2
Also, is it possible to ensure that the write function always outputs 1,2,3,4,5,6 i.e. in the same order in which the data is fed into the feed queue?
The error is somehow with the task_done() call. If you remove that one, then it works, don't ask me why (IMO that's a bug). But the way it works then is that the queueIn.get(block=False) call throws an exception because the queue is empty. This might be just enough for your use case, a better way though would be to use sentinels (as suggested in the multiprocessing docs, see last example). Here's a little rewrite so your program uses sentinels:
import os
import argparse
import multiprocessing
from multiprocessing import Process, Queue
from time import sleep
def feed(queue, parlist, nthreads):
for par in parlist:
queue.put(par)
for i in range(nthreads):
queue.put(None)
print("Queue size", queue.qsize())
def calc(queueIn, queueOut):
while True:
par=queueIn.get()
if par is None:
break
res=doCalculation(par)
queueOut.put((res))
def doCalculation(par):
return par
def write(queue):
while not queue.empty():
par=queue.get()
print("response:",par)
if __name__ == "__main__":
nthreads = 2
workerQueue = Queue()
writerQueue = Queue()
considerperiod=[1,2,3,4,5,6]
feedProc = Process(target=feed, args=(workerQueue, considerperiod, nthreads))
calcProc = [Process(target=calc, args=(workerQueue, writerQueue)) for i in range(nthreads)]
writProc = Process(target=write, args=(writerQueue,))
feedProc.start()
feedProc.join()
for p in calcProc:
p.start()
for p in calcProc:
p.join()
writProc.start()
writProc.join()
A few things to note:
the sentinel is putting a None into the queue. Note that you need one sentinel for every worker process.
for the write function you don't need to do the sentinel handling as there's only one process and you don't need to handle concurrency (if you would do the empty() and then get() thingie in your calc function you would run into a problem if e.g. there's only one item left in the queue and both workers check empty() at the same time and then both want to do get() and then one of them is locked forever)
you don't need to put feed and write into processes, just put them into your main function as you don't want to run it in parallel anyway.
how can I have the same order in output as in input? [...] I guess multiprocessing.map can do this
Yes map keeps the order. Rewriting your program into something simpler (as you don't need the workerQueue and writerQueue and adding random sleeps to prove that the output is still in order:
from multiprocessing import Pool
import time
import random
def calc(val):
time.sleep(random.random())
return val
if __name__ == "__main__":
considerperiod=[1,2,3,4,5,6]
with Pool(processes=2) as pool:
print(pool.map(calc, considerperiod))
I have created a program to generate data points of functions that I later plot. The program takes a class which defines the function, creates a data outputting object which when called generates the data to a text file. To make the whole process faster I put the jobs in threads, however when I do, the data generated is not always correct. I have attached a picture to show what I mean:
Here are some of the relevant bits of code:
from queue import Queue
import threading
import time
queueLock = threading.Lock()
workQueue = Queue(10)
def process_data(threadName, q, queue_window, done):
while not done.get():
queueLock.acquire() # check whether or not the queue is locked
if not workQueue.empty():
data = q.get()
# data is the Plot object to be run
queueLock.release()
data.parent_window = queue_window
data.process()
else:
queueLock.release()
time.sleep(1)
class WorkThread(threading.Thread):
def __init__(self, threadID, q, done):
threading.Thread.__init__(self)
self.ID = threadID
self.q = q
self.done = done
def get_qw(self, queue_window):
# gets the queue_window object
self.queue_window = queue_window
def run(self):
# this is called when thread.start() is called
print("Thread {0} started.".format(self.ID))
process_data(self.ID, self.q, self.queue_window, self.done)
print("Thread {0} finished.".format(self.ID))
class Application(Frame):
def __init__(self, etc):
self.threads = []
# does some things
def makeThreads(self):
for i in range(1, int(self.threadNum.get()) +1):
thread = WorkThread(i, workQueue, self.calcsDone)
self.threads.append(thread)
# more code which just processes the function etc, sorts out the gui stuff.
And in a separate class (as I'm using tkinter, so the actual code to get the threads to run is called in a different window) (self.parent is the Application class):
def run_jobs(self):
if self.running == False:
# threads are only initiated when jobs are to be run
self.running = True
self.parent.calcsDone.set(False)
self.parent.threads = [] # just to make sure that it is initially empty, we want new threads each time
self.parent.makeThreads()
self.threads = self.parent.threads
for thread in self.threads:
thread.get_qw(self)
thread.start()
# put the jobs in the workQueue
queueLock.acquire()
for job in self.job_queue:
workQueue.put(job)
queueLock.release()
else:
messagebox.showerror("Error", "Jobs already running")
This is all the code which relates to the threads.
I don't know why when I run the program with multiple threads some data points are incorrect, whilst running it with just 1 single thread the data is all perfect. I tried looking up "threadsafe" processes, but couldn't find anything.
Thanks in advance!