I have a piece of code which has to get executed every 100ms and update the GUI. When I am updating the GUI - I am pressing a button, which calls a thread and in turn it calls a target function. The target function gives back the message to the GUI thread using pub sub as follows.
wx.CallAfter(pub.sendMessage, "READ EVENT", arg1=data, arg2=status_read) # This command line is in my target function
pub.subscribe(self.ReadEvent, "READ EVENT") # This is in my GUI file - whihc calls the following function
def ReadEvent(self, arg1, arg2):
if arg2 == 0:
self.MessageBox('The program did not properly read data from MCU \n Contact the Program Developer')
return
else:
self.data = arg1
self.firmware_version_text_control.Clear()
#fwversion = '0x' + ''.join('{:02X}'.format(j) for j in reversed(fwversion))
self.firmware_version_text_control.AppendText(str(SortAndDecode(self.data, 'FwVersion')))
# Pump Model
self.pump_model_text_control.Clear()
self.pump_model_text_control.AppendText(str(SortAndDecode(self.data, 'ModelName')))
# Pump Serial Number
self.pump_serial_number_text_control.Clear()
self.pump_serial_number_text_control.AppendText(str(SortAndDecode(self.data, 'SerialNum'))[:10]) # Personal Hack to not to display the AA , AB and A0
# Pressure GAIN
self.gain_text_control.Clear()
self.gain_text_control.AppendText(str(SortAndDecode(self.data, 'PresGain')))
# Pressure OFFSET Offset
self.offset_text_control.Clear()
self.offset_text_control.AppendText(str(SortAndDecode(self.data, 'PresOffset')))
#Wagner Message:
#self.status_text.SetLabel(str(SortAndDecode(self.data, 'WelcomeMsg')))
# PUMP RUNNING OR STOPPED
if PumpState(SortAndDecode(self.data, 'PumpState')) == 1:
self.led6.SetBackgroundColour('GREEN')
elif PumpState(SortAndDecode(self.data, 'PumpState')) == 0:
self.led6.SetBackgroundColour('RED')
else:
self.status_text.SetLabel(PumpState(SortAndDecode(self.data, 'PumpState')))
# PUMP RPM
self.pump_rpm_text_control.Clear()
if not self.new_model_value.GetValue():
self.pump_rpm_text_control.AppendText("000")
else:
self.pump_rpm_text_control.AppendText(str(self.sheet_num.cell_value(self.sel+1,10)*(SortAndDecode(self.data, 'FrqQ5'))/65536))
# PUMP PRESSURE
self.pressure_text_control.Clear()
self.pressure_text_control.AppendText(str(SortAndDecode(self.data, 'PresPsi')))
# ON TIME -- HOURS AND MINUTES --- EDITING IF YOU WANT
self.on_time_text_control.Clear()
self.on_time_text_control.AppendText(str(SortAndDecode(self.data, 'OnTime')))
# JOB ON TIME - HOURS AND MINUTES - EDITING IF YOU WANT
self.job_on_time_text_control.Clear()
self.job_on_time_text_control.AppendText(str(SortAndDecode(self.data, 'JobOnTime')))
# LAST ERROR ----- RECHECK THIS AGAIN
self.last_error_text_control.Clear()
self.last_error_text_control.AppendText(str(SortAndDecode(self.data, 'LastErr')))
# LAST ERROR COUNT --- RECHECK THIS AGAIN
self.error_count_text_control.Clear()
self.error_count_text_control.AppendText("CHECK THIS")
As you can see my READEVENT is very big and it takes a while for the GUI to take enough time to successfully do all these things. My problem here is, when my GUI is updating the values of TEXTCTRL it is going unresponsive - I cannot do anything else. I cant press a button or enter data or anything else. My question is if there is a better way for me to do this, so my GUI wont go unresponsive. I dont know how I can put this in a different thread as all widgets are in the main GUI. But that also requires keep creating threads every 100ms - which is horrible. Any suggestions would be greatly helpful.
Some suggestions:
How long does SortAndDecode take? What about the str() of the result? Those may be good candidates for keeping that processing in the worker thread instead of the UI thread, and passing the values to the UI thread pre-sorted-and-decoded.
You can save a little time in each iteration by calling ChangeValue instead of Clear and AppendText. Why do two function calls for each text widget instead of just one? Function calls are relatively expensive in Python compared to other Python code.
If it's possible that the same value will be sent that was sent on the last iteration then adding checks for the new value matching the old value and skipping the update of the widget could potentially save lots of time. Updating widget values is very expensive compared to leaving them alone.
Unless there is a hard requirement for 100ms updates you may want to try 150 or 200. Fewer updates per second may be fast enough for most people, especially since it's mostly textual. How much text can you read in 100ms?
If you are still having troubles with having more updates than the UI thread can keep up with, then you may want to use a different approach than pubsub and wx.CallAfter. For example you could have the worker thread receive and process the data and then add an object to a Queue.Queue and then call wx.WakeUpIdle(). In the UI thread you can have an EVT_IDLE event handler that checks the queue and pulls the first item out of the queue, if there are any, and then updates the widgets with that data. This will give the benefit of not flooding the pending events list with events from too many wx.CallAfter calls, and you can also do things like remove items from your data queue if there are too many items in it.
Related
I'm trying to implement a GUI to assist some coworkers with image processing, but I am running into a problem with the Tkinter (ttk) progress bar, which keeps lagging while I execute work in another thread.
Basically, I want to have the progress bar run in indeterminate mode and bounce back and forth, as a kind of visual confirmation that things are still progressing (like the "working" circle in windows).
I set it up as follows:
# Frame/root set up (etc.) above here...
pbv = IntVar()
progressBar = ttk.Progressbar(consoleFrame, orient="horizontal", length=100, variable=pbv, mode="indeterminate")
progressBar.grid(column=1, row=1, sticky=N)
progressBar.start()
root.mainloop()
and it bounces along as intended, running nicely on the primary thread. Good so far.
Then, I initialize a separate thread and run a very computationally intensive function.
externalFunctionThread = threading.Thread(target=expensiveFunctionThread, args=[foo])
externalFunctionThread.deamon = True
externalFunctionThread.start()
With some function:
def expensiveFunctionThread(foo):
for i in range(a_few_iterations):
superDuperExpensiveFunction(i, foo)
# Note: Were I to comment this out and replace it with a time.sleep(n), the bar would progress normally
# Even a for loop that prints numbers up to some large integer would not cause lag
For context, superDuperExpensiveFunction is a call to a pyradiomics function, which is generating many texture features from some image.
What ends up happening is the bar will heavily lag, frequently locking up (not moving) while the superDuperExpensiveFunction is running, only moving after each iteration. Once the super duper expensive function thread finishes it will return to normal.
I have looked at other threads on this board (How to connect a progress bar to a function? , Tkinter: How to use threads to preventing main event loop from "freezing" , ttk progress bar freezing , etc.), but none of them help, as they are concerned with the progress bar freezing at the start, as opposed to lagging in its updates. The last one does not work (including progressBar.update()/progressBar.update_idletasks() does nothing). My fear is that it has to do with how Python handles threading (in that there is ultimately only 1 "real" thread that can do work at one time, and Python has to cycle which thread is allowed to work (the GIL or something?)).
Anyways, is there a workaround for this? Why might this be occurring?
Okay, so after doing some research I found out that pyradiomics has a flag that prohibits this kind of rapid task switching, and therefore threading is not a viable solution. To solve the problem, as furas recommended, I implemented multiprocessing (pooling, in fact!). This not only allowed me to have the progress bar run without any lag, but also enabled dividing up the computation across cores, speeding up my implementation greatly. The syntax is very similar to that of threading, with one main exception that caused me a lot of grief before I figured it out. Variables are not inherited by child processes! Therefore, should I want to get a value from an Entry or something, I would have to explicitly pass it in to the child process. Past that, there really aren't that many changes. Instead of threading just use the associated process thing (Process instead of Thread, multiprocessing.Queue instead of queue). No lag, and greater efficiency!
I have two threads concurrently running, speechRecognition and speakBack. Both of these threads are run in while loops (while True: #do something).
Speech recognition is constantly waiting for microphone input. Then, once it is received, it saves the text version of the verbal input to a file, which is loaded by my second thread, speakBack, and spoken through the speakers.
My issue is that when the phrase is spoken through the speakers, it is picked up by the microphone and then translated and once again saved to this file to be processed, resulting in an endless loop.
How can i make the speechRecognition thread suspend itself, wait for the speakBack thread to stop outputting sound through the speakers, and then continue listening for the next verbal input?
Im using the speechRecognition library and the pyttsx3 library for speech recognition and verbal ouput respectively.
The way to do this is to have shared state between the threads (either with global variables that the threads can store into and read from to indicate their progress, or with a mutable reference that is passed into each thread). The solution I’ll give below involves a global variable that stores a mutable reference, but you could just as easily pass the queue into both threads instead of storing it globally.
Using queues is a very standard way to pass messages between threads in python, because queues are already written in a thread-safe way that makes it so you don’t have to think about synchronization and locking. Furthermore, the blocking call to queue.get is implemented in a way that doesn’t involve repeatedly and wastefully checking a condition variable in a while loop.
Here’s how some code might look:
import queue
START_SPEAK_BACK = 0
START_SPEECH_RECOGNITION = 1
messageQueue = queue.Queue()
# thread 1
def speechRecognition():
while True:
# wait for input like you were doing before
# write to file as before
# put message on the queue for other thread to get
messageQueue.put(START_SPEAK_BACK)
# Calling `get` with no arguments makes the call be
# "blocking" in the sense that it won't return until
# there is an element on the queue to get.
messageFromOtherThread = messageQueue.get()
# logically, messageFromOtherThread can only ever be
# START_SPEECH_RECOGNITION, but you could still
# check that this is true and raise an exception if not.
# thread 2
def speakBack():
while True:
messageFromOtherThread = messageQueue.get()
# likewise, this message will only be START_SPEAK_BACK
# but you could still check.
# Here, fill in the code that speaks through the speakers.
# When that's done:
messageQueue.put(START_SPEECH_RECOGNITION)
Some comments:
This solution uses a single queue. It could just have easily used two queues, one for speakBack —> speechRecognition communication and the other for speechRecognition —> communication. This might make more sense if the two threads were generating messages concurrently.
This solution doesn’t actually involve inspecting the contents of the messages. However, if you need to pass additional information between threads, you could very easily pass objects or data as messages (instead of just constant values)
Finally, it’s not clear to me why you don’t just run all code in the same thread. It seems like there’s a very clear (serial) series of steps you want your program to follow: get audio input, write it to file, speak it back, start over. It might make more sense to write everything as a normal, serial, threadless python program.
I want to do a simple job. I have a list of n elements, and want to split the list into two smaller lists and use threading to perform a simple calculation and append them to a new list. I've written some testcode and it seems to work fine when I have a small amount of elements (say 3000). But when the element list is larger (30,000), over 12-20k tasks are being dropped and the append just doesn't go through.
I've read a lot about what constitutes threadsafe, and queueing. I believe it has something to do with that, but even after experimenting with Lock() I still seem to be unable to get a threadsafe Thread.
Can someone point me in the right direction? Cheers.
# Seperate thread workload
a_genes = genes[0:count_seperator]
b_genes = genes[count_seperator:genes_count]
class GeneThread (Thread):
def __init__(self, genelist):
Thread.__init__(self)
self.genelist = genelist
def run(self):
for gene in self.genelist:
total_reputation = 0
for local_snp in gene:
user_rsid = rsids[0]
if user_rsid is None:
continue
rep = "B"
# If multiplier is 0, don't waste time calculating
if not rep or rep == "G" or rep == "U":
continue
importance = 1
weighted_reputation = importance * mul[rep]
zygosity = "homozygous_minor"
if rep == "B":
weighted_reputation *= z_mul[zygosity]
# Now we apply the spread amplifier, we raise the score to the power of the spread number
rep_square = pow(spread, weighted_reputation)
total_reputation += rep_square
try:
with lock:
UserGeneReputation.append(total_reputation)
except:
pass
start_time = time.time()
# Create new threads
gene_thread1 = GeneThread(genelist=a_genes)
gene_thread2 = GeneThread(genelist=b_genes)
gene_thread1.daemon, gene_thread2.daemon = True, True
# Start new Threads
gene_thread1.start()
gene_thread2.start()
print(len(UserGeneReputation))
print("--- %s seconds ---" % (time.time() - start_time))
You have, broadly speaking, two choices with threads. You can have them be autonomous, do their work, and then terminate themselves quietly. Or you can have them be managed by some other thread that monitors their lifetime and knows when they're done. You have a design that absolutely requires the second option (how else will you know when you have all the results you need?), yet you've chosen the first (set them for self-termination and not monitored).
Don't make the threads daemon threads. Instead, wait for both threads to finish after you start them. That's not the most sophisticated or elegant solution, but it's the one everyone learns first.
The problem with this approach is that it forces your code to be dependent on how work is assigned to threads. This can cause performance problems as you wind up having to create and destroy a thread every time you want to know when work is done, and the only way you can know that work is done is by waiting for it. Ideally, you would treat threads as an abstraction that gets work done somehow and code that has to wait for work to be finished would wait for the work itself to be finished (through some synchronization associated with the work itself) rather than wait for the thread to be finished. That way, you can be flexible about what thread does what work and don't have to keep creating and destroying threads every time you need to assign work.
But everyone learns the create/join method. And sometimes it really is the best choice. Even when you use other methods, you likely still have an outer create/join to create the threads in the first place and, typically, ensure they cleanly finish to shut down your program in an orderly way.
I've written a PySide Windows application that uses libvlc to show a video, log keystrokes, and write aggregated information about those keystrokes to a file. I'm experiencing two bugs that are causing the application to crash (other question here -> https://stackoverflow.com/questions/18326943/pyside-qlistwidget-crash).
The application writes the keystroke file at every five minute interval on the video. Users can change the playback speed, so that five minute interval may take more or less than five minutes; it's not controlled by a timer.
The video continues playing while the file is written, so I've created an object inheriting from threading.Thread for the file creation - IntervalFile. Some information about the file to be written is passed in the constructor; IntervalFile doesn't access its parent (the main QWidget) at all. This is the only threading object I use in the app. There are no timer declared anywhere.
Intermittently, the application will crash and I'll get the following message: "QObject::killTimers: timers cannot be stopped from another thread".
The code that creates IntervalFile is (part of CustomWidget, inherited from QWidget):
def doIntervalChange(self):
...
ifile = IntervalFile(5, filepath, dbpath) # db is sqlite, with new connection created within IntervalFile
ifile.start()
#end of def
doIntervalChange is called from within QWidget using a signal.
IntervalFile is:
class IntervalFile(threading.Thread):
def __init__(self, interval, filepath, dbpath):
# declaration of variables
threading.Thread.__init__(self)
def run(self):
shutil.copy('db.local', self.dbPath) # because db is still being used in main QWidget
self.localDB = local(self.dbPath) # creates connection to sqlite db, with sql within the object to make db calls easier
# query db for keystroke data
# write file
self.localDB.close()
self.localDB = None
os.remove(self.dbPath) # don't need this copy anymore
When ifile.start() is commented out, I don't see the killTimers crash. Any suggestions? Note that the crash seems random; sometimes I can use the app (just continuely pressing the same keystroke over and over) for an hour without it crashing, sometimes it's within the first couple of intervals. Because of this difficulty reproducing the crashes, I think these lines of code are the issue, but I'm not 100% sure.
I'm pretty sure you need to hold a reference to your thread object. When your doIntervalChange() method finishes, nothing is holding a reference to the thread object (ifile) any more and so it can be garbage collected. Presumably this is why the crash happens randomly (if the thread finishes it's task before the object is garbage collected, then you don't have a problem).
Not exactly sure what is creating the QTimers, but I'm fairly certain that won't affect my proposed solution!
So in doIntervalChange() save a reference to ifile in a list, and periodically clean up the list when threads have finished execution. Have a look at this for an idea (and if a better way to clean up threads shows up in that post, implement that!): Is there a more elegant way to clean up thread references in python? Do I even have to worry about them?
I have a web application that requests a report that takes more than 10 minutes to run. Apart from improving that performance, I would for now prefer to set up a thread to run the report and mail it to the user, returning that decision message back to the user immediately.
I have been looking at cherrypy.process.plugins.Monitor, but I'm not clear if it is the correct choice (what to do with the frequency parameter?)
Monitor is not the correct choice; it's for running the same task repeatedly on a schedule. You're probably better off just calling threading.Thread(target=run_report).start(). You can then return 202 Accepted to the user, along with a URL for the client to watch the status and/or retrieve the newly-created report resource when it's ready.
The one caveat to that is that you might want your new thread to shut down gracefully when the cherrypy.engine stops. Have a look at the various plugins for examples of how to hook into the 'stop' channel on the bus. The other option would be to make your thread daemonic, if you don't care if it terminates abnormally.
Besides agreeing with fumanchu's answer, I would like to add that the frequency parameter is actually the period expressed in seconds.cherrypy.process.plugins.Monitor (the name is misleading).
Another possible solution could be having a monitor executed periodically, and a set of working computations which can be checked periodically for completion. The code would be something like
class Scheduler:
def __init__ (self):
self.lock = threading.Lock()
self.mon = Monitor(cherrypy.engine, check_computations, frequency=whatever)
self.mon.start()
self.computations = list() # on which we append stuff
def check_computations (self):
with self.lock:
for i in self.computations:
check(i) # Single check function
Caveats:
The computation time of check matters. You don't want to have workload on this perioic routine
Beware on how you use locks:
It is protecting the computations list;
If you access it (even indirectly) from with check your program gets into deadlock. This could be the case if you want to unsubscribe something from the computations list.