Progress bar lagging with computationally intensive function on separate thread tkinter - python-3.x

I'm trying to implement a GUI to assist some coworkers with image processing, but I am running into a problem with the Tkinter (ttk) progress bar, which keeps lagging while I execute work in another thread.
Basically, I want to have the progress bar run in indeterminate mode and bounce back and forth, as a kind of visual confirmation that things are still progressing (like the "working" circle in windows).
I set it up as follows:
# Frame/root set up (etc.) above here...
pbv = IntVar()
progressBar = ttk.Progressbar(consoleFrame, orient="horizontal", length=100, variable=pbv, mode="indeterminate")
progressBar.grid(column=1, row=1, sticky=N)
progressBar.start()
root.mainloop()
and it bounces along as intended, running nicely on the primary thread. Good so far.
Then, I initialize a separate thread and run a very computationally intensive function.
externalFunctionThread = threading.Thread(target=expensiveFunctionThread, args=[foo])
externalFunctionThread.deamon = True
externalFunctionThread.start()
With some function:
def expensiveFunctionThread(foo):
for i in range(a_few_iterations):
superDuperExpensiveFunction(i, foo)
# Note: Were I to comment this out and replace it with a time.sleep(n), the bar would progress normally
# Even a for loop that prints numbers up to some large integer would not cause lag
For context, superDuperExpensiveFunction is a call to a pyradiomics function, which is generating many texture features from some image.
What ends up happening is the bar will heavily lag, frequently locking up (not moving) while the superDuperExpensiveFunction is running, only moving after each iteration. Once the super duper expensive function thread finishes it will return to normal.
I have looked at other threads on this board (How to connect a progress bar to a function? , Tkinter: How to use threads to preventing main event loop from "freezing" , ttk progress bar freezing , etc.), but none of them help, as they are concerned with the progress bar freezing at the start, as opposed to lagging in its updates. The last one does not work (including progressBar.update()/progressBar.update_idletasks() does nothing). My fear is that it has to do with how Python handles threading (in that there is ultimately only 1 "real" thread that can do work at one time, and Python has to cycle which thread is allowed to work (the GIL or something?)).
Anyways, is there a workaround for this? Why might this be occurring?

Okay, so after doing some research I found out that pyradiomics has a flag that prohibits this kind of rapid task switching, and therefore threading is not a viable solution. To solve the problem, as furas recommended, I implemented multiprocessing (pooling, in fact!). This not only allowed me to have the progress bar run without any lag, but also enabled dividing up the computation across cores, speeding up my implementation greatly. The syntax is very similar to that of threading, with one main exception that caused me a lot of grief before I figured it out. Variables are not inherited by child processes! Therefore, should I want to get a value from an Entry or something, I would have to explicitly pass it in to the child process. Past that, there really aren't that many changes. Instead of threading just use the associated process thing (Process instead of Thread, multiprocessing.Queue instead of queue). No lag, and greater efficiency!

Related

Does using a context manager in a generator may lead to resources leak?

I have a function that yields from a context manager:
def producer(pathname):
with open(pathname) as f:
while True:
chunk = f.read(4)
if not chunk:
break
yield chunk
It is not a problem when the generator is entirely consumed since, during the last iteration, the generator resumes execution after the yield statement, and the loop breaks and we nicely exit the context manager.
However, if the generator is only partially consumed, and there are no more consumers to consume it entirely, will the generator remain suspended forever? In that case, we will never exit from the context manager. Would that mean the file will remain open for the rest of the program execution? Or at least until the generator is garbage collected? Is this a corner case I should take care of by myself, or can I rely on the Python runtime to close dangling context manager in time?
FWIW, I've seen Generator and context manager at the same time and How to use a python context manager inside a generator but I don't think they really answer the same question. Unless I missed something?
If you fail to consume the whole generator, the context manager won't be cleaned until the generator is garbage collected, which may take quite a while if reference cycles are involved, or you're running on a non-CPython interpreter.
You can work around this by close-ing the generator-iterator; all generator functions provide a close method on the resulting generator-iterator that raises GeneratorExit inside it; the exception bubbles out of with statements and the like to make sure they're properly cleaned deterministically.
To make it occur at a guaranteed point in time, you can use contextlib.closing to get guaranteed closing of the generator itself:
from contextlib import closing
with closing(producer(mypath)) as produced_items:
for item in produced_items:
# Do stuff, maybe break loop early
Even if you break, return, or raise an exception, the with controlling produced_items will close it, which will in turn invoke cleanup for the with statements within it.

F# / MailBoxProcessor is unresponsive to PostAndReply under nearly 100% load

I have a MailBoxProcessor, which does the following things:
Main loop (type AsyncRunner: https://github.com/kkkmail/ClmFSharp/blob/master/Clm/ContGen/AsyncRun.fs#L257 – the line number may change as I keep updating the code). It generates some "models", compiles each of them into a model specific folder, spawns them as external processes, and then each model uses WCF to "inform" AsyncRunner about its progress by calling updateProgress. A model may take several days to run. Once any of the models is completed, the runner generates / spawns more. It is designed to run at 100% processor load (but with priority: ProcessPriorityClass.BelowNormal), though I can specify a smaller number of logical cores to use (some number between 1 and Environment.ProcessorCount). Currently I "async"-ed almost everything that goes inside MailBoxProcessor by using … |> Async.Start to ensure that I "never ever" block the main loop.
I can "ask" the runner (using WCF) about its state by calling member this.getState () = messageLoop.PostAndReply GetState.
OR I can send some commands to it (again using WCF), e.g. member this.start(), member this.stop(), …
Here is where it gets interesting. Everything works! However, if I run a "monitor", which would ask for a state by effectively calling PostAndReply (exposed as this.getState ()) in an infinite loop, the after a while it sort of hangs up. I mean that it does eventually return, but with some unpredictably large delays (like a few minutes). At that same time, I can issue commands and they do return fast while getState still has not returned.
Is it possible to make it responsive at nearly 100% load? Thanks a lot!
I would suggest not asyncing anything(other than your spawning of processes) in your main program, since your code creates additional processes. Your main loop is waiting on the loop return to continue before processing the GetState() method.

Frequent Updating of GUI WxPYTHON

I have a piece of code which has to get executed every 100ms and update the GUI. When I am updating the GUI - I am pressing a button, which calls a thread and in turn it calls a target function. The target function gives back the message to the GUI thread using pub sub as follows.
wx.CallAfter(pub.sendMessage, "READ EVENT", arg1=data, arg2=status_read) # This command line is in my target function
pub.subscribe(self.ReadEvent, "READ EVENT") # This is in my GUI file - whihc calls the following function
def ReadEvent(self, arg1, arg2):
if arg2 == 0:
self.MessageBox('The program did not properly read data from MCU \n Contact the Program Developer')
return
else:
self.data = arg1
self.firmware_version_text_control.Clear()
#fwversion = '0x' + ''.join('{:02X}'.format(j) for j in reversed(fwversion))
self.firmware_version_text_control.AppendText(str(SortAndDecode(self.data, 'FwVersion')))
# Pump Model
self.pump_model_text_control.Clear()
self.pump_model_text_control.AppendText(str(SortAndDecode(self.data, 'ModelName')))
# Pump Serial Number
self.pump_serial_number_text_control.Clear()
self.pump_serial_number_text_control.AppendText(str(SortAndDecode(self.data, 'SerialNum'))[:10]) # Personal Hack to not to display the AA , AB and A0
# Pressure GAIN
self.gain_text_control.Clear()
self.gain_text_control.AppendText(str(SortAndDecode(self.data, 'PresGain')))
# Pressure OFFSET Offset
self.offset_text_control.Clear()
self.offset_text_control.AppendText(str(SortAndDecode(self.data, 'PresOffset')))
#Wagner Message:
#self.status_text.SetLabel(str(SortAndDecode(self.data, 'WelcomeMsg')))
# PUMP RUNNING OR STOPPED
if PumpState(SortAndDecode(self.data, 'PumpState')) == 1:
self.led6.SetBackgroundColour('GREEN')
elif PumpState(SortAndDecode(self.data, 'PumpState')) == 0:
self.led6.SetBackgroundColour('RED')
else:
self.status_text.SetLabel(PumpState(SortAndDecode(self.data, 'PumpState')))
# PUMP RPM
self.pump_rpm_text_control.Clear()
if not self.new_model_value.GetValue():
self.pump_rpm_text_control.AppendText("000")
else:
self.pump_rpm_text_control.AppendText(str(self.sheet_num.cell_value(self.sel+1,10)*(SortAndDecode(self.data, 'FrqQ5'))/65536))
# PUMP PRESSURE
self.pressure_text_control.Clear()
self.pressure_text_control.AppendText(str(SortAndDecode(self.data, 'PresPsi')))
# ON TIME -- HOURS AND MINUTES --- EDITING IF YOU WANT
self.on_time_text_control.Clear()
self.on_time_text_control.AppendText(str(SortAndDecode(self.data, 'OnTime')))
# JOB ON TIME - HOURS AND MINUTES - EDITING IF YOU WANT
self.job_on_time_text_control.Clear()
self.job_on_time_text_control.AppendText(str(SortAndDecode(self.data, 'JobOnTime')))
# LAST ERROR ----- RECHECK THIS AGAIN
self.last_error_text_control.Clear()
self.last_error_text_control.AppendText(str(SortAndDecode(self.data, 'LastErr')))
# LAST ERROR COUNT --- RECHECK THIS AGAIN
self.error_count_text_control.Clear()
self.error_count_text_control.AppendText("CHECK THIS")
As you can see my READEVENT is very big and it takes a while for the GUI to take enough time to successfully do all these things. My problem here is, when my GUI is updating the values of TEXTCTRL it is going unresponsive - I cannot do anything else. I cant press a button or enter data or anything else. My question is if there is a better way for me to do this, so my GUI wont go unresponsive. I dont know how I can put this in a different thread as all widgets are in the main GUI. But that also requires keep creating threads every 100ms - which is horrible. Any suggestions would be greatly helpful.
Some suggestions:
How long does SortAndDecode take? What about the str() of the result? Those may be good candidates for keeping that processing in the worker thread instead of the UI thread, and passing the values to the UI thread pre-sorted-and-decoded.
You can save a little time in each iteration by calling ChangeValue instead of Clear and AppendText. Why do two function calls for each text widget instead of just one? Function calls are relatively expensive in Python compared to other Python code.
If it's possible that the same value will be sent that was sent on the last iteration then adding checks for the new value matching the old value and skipping the update of the widget could potentially save lots of time. Updating widget values is very expensive compared to leaving them alone.
Unless there is a hard requirement for 100ms updates you may want to try 150 or 200. Fewer updates per second may be fast enough for most people, especially since it's mostly textual. How much text can you read in 100ms?
If you are still having troubles with having more updates than the UI thread can keep up with, then you may want to use a different approach than pubsub and wx.CallAfter. For example you could have the worker thread receive and process the data and then add an object to a Queue.Queue and then call wx.WakeUpIdle(). In the UI thread you can have an EVT_IDLE event handler that checks the queue and pulls the first item out of the queue, if there are any, and then updates the widgets with that data. This will give the benefit of not flooding the pending events list with events from too many wx.CallAfter calls, and you can also do things like remove items from your data queue if there are too many items in it.

IronPython: StartThread creates 'DummyThreads' that don't get cleaned up

The problem I have is that in my IronPython application threads are being created but never cleaned up, even when the method they run has exited. In my application I start threads in two ways: a) by using Python-style threads (sub-classes of threading.Thread that do something in their run() method), and b) by using the .NET 'ThreadStart' approach. The Python-style threads behave as expected, and after their 'run()' exits they get cleaned up. The .NET style threads never get cleaned up, even after they have exited. You can call del, Abort, whatever you want, and it has no effect on them.
The following IronPython script demonstrates the issue:
import System
import threading
import time
import logging
def do_beeps():
logging.debug("starting do_beeps")
t_start = time.clock()
while time.clock() - t_start < 10:
System.Console.Beep()
System.Threading.Thread.CurrentThread.Join(1000)
logging.debug("exiting do_beeps")
class PythonStyleThread(threading.Thread):
def __init__(self, thread_name="PythonStyleThread"):
super(PythonStyleThread, self).__init__(name=thread_name)
def run(self):
do_beeps()
class ThreadStarter():
def start(self):
t = System.Threading.Thread(System.Threading.ThreadStart(do_beeps))
t.IsBackground = True
t.Name = "ThreadStartStyleThread"
t.Start()
if __name__ == '__main__':
logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.DEBUG, datefmt='%H:%M:%S')
# Start some ThreadStarter threads:
for _ in range(5):
ts = ThreadStarter()
ts.start()
System.Threading.Thread.CurrentThread.Join(200)
# Start some Python-style threads:
for _ in range(5):
pt = PythonStyleThread()
pt.start()
System.Threading.Thread.CurrentThread.Join(200)
# Do something on the main thread:
for _ in range(30):
print(".")
System.Threading.Thread.CurrentThread.Join(1000)
When this is debugged in PyDev, what I see is that all the threads appear as expected in the 'debug' view as they are created:
but whereas the Python-style ones disappear after they've finished, the .NET / ThreadStart ones stay until the main thread exits.
As can be seen in the image, in the debugger the problematic threads appear with names 'Dummy-4', 'Dummy-5' etc, whereas the Pythonic ones appear with the name I've given them ('PythonStyleThread'). Looking in the threading.py file in my IronPython installation I see there is a class called "_DummyThread", a subclass of Thread, that sets its 'name' as 'name=_newname("Dummy-%d")', so it looks like by using ThreadStart I'm ending up with _DummyThreads. The comment for the class also says:
# Dummy thread class to represent threads not started here.
# These aren't garbage collected when they die, nor can they be waited for.
which would explain why I can't get rid of them.
But I don't want 'DummyThread's. I just want normal ones, that behave nicely and get garbage-collected when they've finished doing their thing.
Now, a slightly odd thing about all of this is that unless I set up the logger I don't see the DummyThread entries in the debugger at all (although they still run). This may be a funny of the PyDev debugger, or it may relevant. Is there any sane reason why logging should have any bearing on this? Can I solve my problem just by not logging in my thread?
Here, it says:
"There is the possibility that "dummy thread objects" are created. These are thread objects corresponding to "alien threads", which are threads of control started outside the threading module, such as directly from C code. Dummy thread objects have limited functionality; they are always considered alive and daemonic, and cannot be joined. They are never deleted, since it is impossible to detect the termination of alien threads."
Which makes me wonder why I've had the misfortune of ending up with them?
While I have a workaround in that I can use Python-style threading.Thread subclasses everywhere I currently use .NET 'ThreadStart' threads, I am not keen to do this as the reason I was using .NET style threads in certain places was because they give me an Abort method (whereas the Python ones don't). I know Aborting threads is a Bad Thing, but the application is a unit-testing framework, and I a) need to run unit-tests in a thread, and b) have no control over their contents (they are written by third-parties), so I have no means of periodically checking for a 'please shut me down nicely' flag on these threads, and in extremis may need to kill them rudely.
So a) why am I getting DummyThreads, b) has this got anything to do with logging and c) what can I do about it?
Thanks.

Operations in separate TThread block GUI thread

I used this tutorial http://delphi.about.com/od/kbthread/a/thread-gui.htm to create a class that asynchronously downloads a file from the internet in another thread with a TDownLoadURL. I did this because I want to download a file without blocking the UI thread so the program doesn't become unresponsive during large downloads, the progress bar can update, etc.
I am having a problem because even though I have done the download in another thread (inheriting from TThread and doing the work in the Execute method) the GUI thread seems to be blocked and does not process messages until the download is finished. Here is the code for my class: http://codepad.org/nArfOPJK (it's just 99 lines, a simple class). I am executing it by this, in the event handler for a button click:
var
frame: TTProgressFrame;
dlt: TDownloadThread;
begin
dlt := TDownloadThread.Create(True);
dlt.SetFile('C:\ohayo.zip');
dlt.SetURL('http://download.thinkbroadband.com/512MB.zip');
dlt.SetFrame(frame);
dlt.SetApp(Application);
dlt.Start;
Note: The SetApp method was for when I was manually calling app.ProcessMessages from inside the UpdateDownloadProgress method of my class TDownloadThread. This would keep the GUI from being unresponsive, but it made the progress bar behave wierdly (the moving glowing light thing of aero's progress bar moving way too fast), so I removed it. I want to fix this properly, and if I have to call ProcessMessages there's really no point in multithreading it.
Can someone help me fix this problem? Thanks.
I now have the solution for you!
Calling TDownLoadURL.Execute (your call to dl.Execute in TDownloadThread) results in the action being transferred back into the main thread which is why your UI becomes unresponsive.
Instead you should call ExecuteTarget(nil) which performs no such machinations and works as you intend: the download runs on the worker thread.

Resources