I have created a program to generate data points of functions that I later plot. The program takes a class which defines the function, creates a data outputting object which when called generates the data to a text file. To make the whole process faster I put the jobs in threads, however when I do, the data generated is not always correct. I have attached a picture to show what I mean:
Here are some of the relevant bits of code:
from queue import Queue
import threading
import time
queueLock = threading.Lock()
workQueue = Queue(10)
def process_data(threadName, q, queue_window, done):
while not done.get():
queueLock.acquire() # check whether or not the queue is locked
if not workQueue.empty():
data = q.get()
# data is the Plot object to be run
queueLock.release()
data.parent_window = queue_window
data.process()
else:
queueLock.release()
time.sleep(1)
class WorkThread(threading.Thread):
def __init__(self, threadID, q, done):
threading.Thread.__init__(self)
self.ID = threadID
self.q = q
self.done = done
def get_qw(self, queue_window):
# gets the queue_window object
self.queue_window = queue_window
def run(self):
# this is called when thread.start() is called
print("Thread {0} started.".format(self.ID))
process_data(self.ID, self.q, self.queue_window, self.done)
print("Thread {0} finished.".format(self.ID))
class Application(Frame):
def __init__(self, etc):
self.threads = []
# does some things
def makeThreads(self):
for i in range(1, int(self.threadNum.get()) +1):
thread = WorkThread(i, workQueue, self.calcsDone)
self.threads.append(thread)
# more code which just processes the function etc, sorts out the gui stuff.
And in a separate class (as I'm using tkinter, so the actual code to get the threads to run is called in a different window) (self.parent is the Application class):
def run_jobs(self):
if self.running == False:
# threads are only initiated when jobs are to be run
self.running = True
self.parent.calcsDone.set(False)
self.parent.threads = [] # just to make sure that it is initially empty, we want new threads each time
self.parent.makeThreads()
self.threads = self.parent.threads
for thread in self.threads:
thread.get_qw(self)
thread.start()
# put the jobs in the workQueue
queueLock.acquire()
for job in self.job_queue:
workQueue.put(job)
queueLock.release()
else:
messagebox.showerror("Error", "Jobs already running")
This is all the code which relates to the threads.
I don't know why when I run the program with multiple threads some data points are incorrect, whilst running it with just 1 single thread the data is all perfect. I tried looking up "threadsafe" processes, but couldn't find anything.
Thanks in advance!
Related
I am building a GUI on python and pyqt.
The GUI has a lot of pushbuttons, generated through class LED, meaning each led has 3 buttons, for an n number of leds.
In a few of the buttons, I want an effect that changes the opacity of the pushbutton, in a loop from 0 to 1 and back again, so it disappears and appears. I need only one process to manage all, so the effect starts at same time for every button and all blink at the same time.
I've managed to achieve that, through qgraphicseffect in a thread, iterating through a list.
The problem is that after a few minutes, the effect stops, although the thread is still running (print(opacity_level)). more pushbuttons with the effect makes even shorter duration. Clicking any button, even others without effect, restarts the gui animation.
My small research in threading on pyqt made me implement this thread manager, although I do not fully understand it.
class WorkerSignals(QtCore.QObject):
finished = QtCore.pyqtSignal()
error = QtCore.pyqtSignal(tuple)
result = QtCore.pyqtSignal(object)
progress = QtCore.pyqtSignal(tuple)
class Worker(QtCore.QRunnable):
'''
Worker thread
Inherits from QRunnable to handler worker thread setup, signals and wrap-up.
'''
def __init__(self, fn, *args, **kwargs):
super(Worker, self).__init__()
# Store constructor arguments (re-used for processing)
self.fn = fn
self.args = args
self.kwargs = kwargs
self.signals = WorkerSignals()
# Add the callback to our kwargs
self.kwargs['progress_callback'] = self.signals.progress
#pyqtSlot()
def run(self):
'''
Initialise the runner function with passed args, kwargs.
'''
# Retrieve args/kwargs here; and fire processing using them
try:
result = self.fn(*self.args, **self.kwargs)
except:
traceback.print_exc()
exctype, value = sys.exc_info()[:2]
self.signals.error.emit((exctype, value, traceback.format_exc()))
else:
self.signals.result.emit(result) # Return the result of the processing
finally:
self.signals.finished.emit() # Done
Next the leds class
class LEDs:
def __init__(self,name,group,frame):
self.opacity_effect = QtWidgets.QGraphicsOpacityEffect()
self.button_auto = QtWidgets.QPushButton()
self.button_auto.setObjectName("button_auto_neutral")
self.button_auto.clicked.connect(lambda state, x=self: self.AutoMode())
def AutoMode(self):
print(self.name,"Automode")
if len(settings.blink) ==0: # start thread only if no previous thread, both thread and
this reference the size of settings.blink, so not ideal.
print("start thread")
settings.ledAutomode()
settings.blink.append(self)
And finally the settings class, which has the thread with the effect performing action. There is a second thread, which handles the icon of the button, accordingly with a timetable.
class Settings:
def __init__(self):
self.blink=[]
def ledAutomode(self):
def blink(progress_callback):
print("opacity")
op_up=[x/100 for x in range(0,101,5)]
op_down=op_up[::-1]; op_down=op_down[1:-1]; opacity=op_up+op_down
while len(self.blink) !=0:
for i in opacity:
print(i)
QtCore.QThread.msleep(80)
for led in self.blink:
led.opacity_effect.setOpacity(i)
def timeCheck(progress_callback):
while len(self.blink) != 0:
QtCore.QThread.msleep(500)
for led in self.blink:
matrix = [v for v in settings.leds_config[led.group][led.name]["Timetable"]]
matrix_time=[]
...
# some code
...
if sum(led_on_time):
led.button_auto.setObjectName("button_auto_on")
led.button_auto.setStyleSheet(ex.stylesheet)
else:
led.button_auto.setObjectName("button_auto_off")
led.button_auto.setStyleSheet(ex.stylesheet)
QtCore.QThread.msleep(int(30000/len(self.blink)))
worker = Worker(blink) # Any other args, kwargs are passed to the run function
ex.threadpool.start(worker)
worker2 = Worker(timeCheck) # Any other args, kwargs are passed to the run function
ex.threadpool.start(worker2)
So, perhaps a limitation on qgraphicseffect, or some problem with the thread (although its keeps printing), or I made some error.
I've read about subclassing the qgraphicseffect but I don't know if that solves the problem.
If anyone has another implementation, always eager to learn.
Grateful for your time.
Widgets are not thread-safe.
They cannot be created nor accessed from external threads. While it "sometimes" works, doing it is wrong and usually leads to unexpected behavior, drawing artifacts and even fatal crash.
That said, you're making the whole process incredibly and unnecessarily convoluted, much more than it should be, most importantly because Qt already provides both timed events (QTimer) and animations.
class FadeButton(QtWidgets.QPushButton):
def __init__(self):
super().__init__()
self.effect = QtWidgets.QGraphicsOpacityEffect(opacity=1.0)
self.setGraphicsEffect(self.effect)
self.animation = QtCore.QPropertyAnimation(self.effect, b'opacity')
self.animation.setStartValue(1.0)
self.animation.setEndValue(0.0)
self.animation.setDuration(1500)
self.animation.finished.connect(self.checkAnimation)
self.clicked.connect(self.startAnimation)
def startAnimation(self):
self.animation.stop()
self.animation.setDirection(self.animation.Forward)
self.animation.start()
def checkAnimation(self):
if not self.animation.value():
self.animation.setDirection(self.animation.Backward)
self.animation.start()
else:
self.animation.setDirection(self.animation.Forward)
If you want to synchronize opacity amongst many widgets, there are various possibilities, but a QVariantAnimation that updates all opacities is probably the easier choice:
class LEDs(QtWidgets.QWidget):
def __init__(self, parent=None):
super().__init__(parent)
layout = QtWidgets.QHBoxLayout(self)
self.animation = QtCore.QVariantAnimation()
self.animation.setStartValue(1.0)
self.animation.setEndValue(0.0)
self.animation.setDuration(1500)
self.animation.valueChanged.connect(self.updateOpacity)
self.animation.finished.connect(self.checkAnimation)
self.buttons = []
for i in range(3):
button = QtWidgets.QPushButton()
self.buttons.append(button)
layout.addWidget(button)
effect = QtWidgets.QGraphicsOpacityEffect(opacity=1.0)
button.setGraphicsEffect(effect)
button.clicked.connect(self.startAnimation)
# ... as above ...
def updateOpacity(self, opacity):
for button in self.buttons:
button.graphicsEffect().setOpacity(opacity)
Note that you shouldn't change the object name of a widget during runtime, and doing it only because you want to update the stylesheet is wrong. You either use a different stylesheet, or you use the property selector:
QPushButton {
/* default state */
background: #ababab;
}
QPushButton[auto_on="true"] {
/* "on" state */
background: #dadada;
}
class FadeButton(QtWidgets.QPushButton):
def __init__(self):
super().__init__()
# ...
self.setProperty('auto_on', False)
def setAuto(self, state):
self.setProperty('auto_on', state)
self.setStyleSheet(self.styleSheet())
main method takes no arguments. main method should read all files and folders on the hard disk. However, it seems that there is no way to execute the main method in a thread. In the beginning, it reads well, then slows down from the middle, and the QListWidget, ScanInfo, stops and prints out at once, which is strange. I want it to print smoothly. What should I do?
InfoAV.py
def AScanButtonClicked(self):
set1 = Setting.fileSet
set2 = Setting.disinfectSet
self.iv = InfonetV('C:/', set1.checkState(), set2.checkState())
self.window = ScanForm()
self.iv.started.connect(self.window.show)
self.iv.logSignal1.connect(self.window.ScanInfo.addItem)
self.iv.logSignal1.connect(self.window.ScanInfo.scrollToBottom)
self.iv.logSignal2.connect(self.window.ResultInfo.addItem)
self.iv.scan()
IV.py
class InfonetV(QObject):
started = pyqtSignal()
finished = pyqtSignal()
logSignal1 = pyqtSignal(str)
logSignal2 = pyqtSignal(str)
def __init__(self, path, set1, set2, parent = None):
...
...
def scan(self):
threading.Thread(target=self.main).start()
...
def main(self):
self.started.emit()
logSinal1 is a signal to add the Item of ScanList which is QListWidget. Write time.sleep(0.05) before logSignal1 is executed. Then it runs fine.
time.sleep(0.05)
self.logSignal1.emit(str(fname+ " " +message))
I have a number of certain objects which need to run a specific function at specific ever-changing intervals, again and again, until they decide they are done.
For example, one object may need to wait 30 seconds, run, wait 60 seconds, run, wait 10 seconds, run... You get the point, and this could be going on for 30-120 different objects, running the exact same kind of function.
I was thinking that simply having a function that sleeps for the exact interval would solve my problem, but, correct me if I'm wrong, I remembered that thread pools can only run a certain number of threads at any given time (12 for me). How do I get around this limit?
class Thing(object):
def getCurrentPeriod(self):
return random.randint(5, 30) # Some ever changing period of time
def refresh(self):
doThings() # A long running task that is disk and network intensive
def waitRefresh(self):
period = self.getCurrentPeriod()
time.sleep(period) # Wait that period out
self.refresh()
return self.needRefresh()
# Boolean if it needs to restart - Not sure about how to reschedule,
# or specifically where to connect the worker emit when it finishes
# to make sure this *specific* Thing obj gets it's waitRefresh func called again.
class App(QMainWindow):
def __init__(self, *args, **kwargs):
super(MainWindow, self).__init__(*args, **kwargs)
self.threadpool = QThreadPool()
# Add initial objects to pool (other portions of app may add more over time)
for thing in self.acquireThings():
worker = Worker(thing.waitRefresh)
self.threadpool.start(worker)
Doesn't include the WorkerSignals class nor the QRunnable subclass, this example includes what I usually do. The example is tackling the same problem, but in a (most likely) inefficient way.
edit: New example with complete working example of how time.sleep does not pause the thread and allow others to work. I feel that async may be the only implementation, but is there a quick fix so I don't have to alter my entire app?
Here's what it looks like when you try to sleep more than 12 threads.
The ultimate solution came when I decided to actually try the QTimer class. Perhaps there are more optimized solutions, but this one seems to hit all the checkboxes, even if it's worryingly simple.
import random
import time
import traceback
from functools import partial
from PyQt5.QtCore import *
from PyQt5.QtGui import QFont
from PyQt5.QtWidgets import *
class WorkerSignals(QObject):
"""
Represents the signals a Worker can emit.
"""
finished = pyqtSignal()
starting = pyqtSignal(int) # ID of thread
result = pyqtSignal(tuple) # Tuple refresh result, result and ID
class Worker(QRunnable):
"""
A worker designed to tell when it's starting, when it's finished and the result.
Designed to work around Thread.refresh().
"""
def __init__(self, fn, thread_id, *args, **kwargs):
super(Worker, self).__init__()
# Store constructor arguments (re-used for processing)
self.fn = fn
self.id = thread_id
self.args = args
self.kwargs = kwargs
self.signals = WorkerSignals()
#pyqtSlot()
def run(self):
"""
Runs a given method, and emits the result with the Worker's coordinated ID.
"""
try:
self.signals.starting.emit(self.id) # Thread is now finally ready to work.
result = self.fn(*self.args, **self.kwargs) # Refresh Thread!
self.signals.result.emit(result) # Thread is finished, emit result tuple.
except:
traceback.print_exc()
finally:
self.signals.finished.emit() # Done
class Thread(object):
"""
Basic Rules for a Thread Object:
Cannot store the next timestamp on the object (it's a database object, I don't believe it's good practice
to be creating sessions over and over to simply read/write the access time.
ID and Active are allowed as booleans.
"""
i = -1
def __init__(self):
self.id = Thread.nextID()
self.active = True
self.refreshes = 0
def refresh(self) -> tuple:
"""
'Refreshes' a thread. Waits a specific period, then decides whether Thread object should be deactivated or
returned from additional refreshes. Chance of deactivation lowers with each refresh.
:return: The refresh result, a tuple with a boolean and the thread's ID (for identifying it later)
"""
# Represents my SQL Alchemy Model's refresh() function
self.refreshes += 1
time.sleep(random.randint(2, 5))
if random.random() <= max(0.1, 1.0 - ((self.refreshes + 5) / 10)):
self.active = False
return self.active, self.id
#staticmethod
def getRefreshTime() -> float:
"""
Represents the amount of time before a thread should be refreshed.
Should NOT be used to determine whether the thread is still active or not.
:return: The time period that should be waited.
"""
return random.uniform(10, 300)
#staticmethod
def nextID() -> int:
"""
Returns integer thread IDs in sequence to remove possibility of duplicate IDs.
:return: Integer Thread ID
"""
Thread.i += 1
return Thread.i
def __repr__(self):
return f'Thread(id={self.id} active={self.active})'
class MainWindow(QMainWindow):
"""
GUI containing a Label, Button and ListWidget showing all the active sleeping/working threads.
Manages a threadpool, a number of background singleshot timers, etc.
"""
def __init__(self, *args, **kwargs):
super(MainWindow, self).__init__(*args, **kwargs)
# Widgets Setup
layout = QVBoxLayout()
self.list = QListWidget()
self.l = QLabel("Total Active: 0")
self.button = QPushButton("Refresh List")
self.button.pressed.connect(self.refreshList)
self.button.setDisabled(True)
layout.addWidget(self.l)
layout.addWidget(self.button)
layout.addWidget(self.list)
w = QWidget()
w.setLayout(layout)
self.setCentralWidget(w)
self.show()
# Periodically add threads to the pool.
self.poolTimer = QTimer()
self.poolTimer.setInterval(5_000)
self.poolTimer.timeout.connect(self.addThreads)
# Threading Setup
self.threadpool = QThreadPool()
print("Multithreading with maximum %d threads" % self.threadpool.maxThreadCount())
self.active, self.threads = {}, {}
# Add a number of threads to start with.
for _ in range(random.randint(5, 16)):
self.setupThread(Thread())
self.poolTimer.start()
def refreshList(self):
"""
Refreshes the ListWidget in the GUI with all the active/sleeping/working threads.
"""
self.list.clear()
bold = QFont()
bold.setBold(True)
active = 0
for thread in self.threads.values():
item = QListWidgetItem(
f'Thread {thread.id}/{thread.refreshes}')
# Bold a thread if it's working
if self.active[thread.id]:
active += 1
item.setFont(bold)
self.list.addItem(item)
self.l.setText(f'Total Active: {active}/{len(self.threads)}')
def refreshResult(self, result) -> None:
"""
When a thread is finished, the result determines it's next course of action, which is either
to return to the pool again, or delete itself.
:param result: A tuple containing the result (bool) and the connected Thread ID.
"""
self.active[result[1]] = False
if result[0]:
print(f'Restarting Thread {result[1]}')
self.setupThread(self.threads[result[1]]) # Add by ID, which would normally be a database GET
else:
print(f'Thread {result[1]} shutting down.')
del self.active[result[1]]
del self.threads[result[1]]
self.refreshList()
def updateActivity(self, thread_id) -> None:
"""
Connected to the starting signal, helps signal when a thread is actually being refreshed.
:param thread_id: The Thread ID
"""
print(f'Thread {thread_id} is now active/working.')
self.active[thread_id] = True
def refresh(self, thread):
"""
Adds a new worker to the threadpool to be refreshed.
Can't be considered a real start to the thread.refresh function, as the pool has a max of 12 workers at any time.
The 'starting' signal can tell us when a specific thread is actually being refreshed, and is represented
as a Bold element in the list.
:param thread: A thread instance.
"""
print(f'Adding Thread {thread.id} to the pool.')
worker = Worker(thread.refresh, thread_id=thread.id)
worker.signals.result.connect(self.refreshResult)
worker.signals.starting.connect(self.updateActivity)
self.threadpool.start(worker)
# self.active[thread.id] = True
self.refreshList()
def setupThread(self, thread) -> None:
"""
Adds a new timer designated to start a specific thread.
:param thread: A thread instance.
"""
self.active[thread.id] = False
self.threads[thread.id] = thread
t = QTimer()
period = thread.getRefreshTime()
t.singleShot(period * 1000, partial(self.refresh, thread=thread))
print(f'Thread {thread.id} will start in {period} seconds.')
self.refreshList()
def addThreads(self):
"""
Adds a number of threads to the pool. Called automatically every couple seconds.
"""
add = max(0, 30 + random.randint(-5, 5) - len(self.threads))
if add > 0:
print(f'Adding {add} thread{"s" if add > 1 else ""}.')
for _ in range(add):
self.setupThread(Thread())
app = QApplication([])
window = MainWindow()
app.exec_()
When a Thread is requested, a Timer is created and singleShot is fired on an extra function that will add it to the threadpool. This threadpool can handle up to 12 refreshing continious 'refreshing' threads, and signals allow the GUI to update the moment a change is found.
Thousands of 'Thread' objects can be waiting and it seems singleShot is capable of adding them to the pool exactly when they need to be.
Signals help differentiate when a thread is sleeping, working and active (but inactive Thread objects are immediately removed).
The only caveats I can think of with this program is:
1) Can a QThread implementation beat it?
2) What happens to the QTimer once it's singleshot function has executed and fired? Will they be properly GC'd, or can thousands build up in the background consuming resources?
I start multiple processes in order to create a list of new objects. htop shows me in between 1 and 4 processes (I always create 3 new objects).
def foo(self):
with multiprocessing.Pool(processes=3, maxtasksperchild=10) as pool:
result = pool.map_async(self.new_obj, self.information)
self.new_objs = result.get()
pool.terminate()
gc.collect()
I call foo() multiple times, each time it is called, the whole process is running slower, the program does not even finish in the end, as it slows down to much. The program starts to eat up all my RAM, while the sequential approach does not have any significant RAM usage.
When I kill the program, most of the time this was the function the program was last executing.
->File "threading.py", line 293, in wait
waiter.acquire()
Edit
To give some information about my circumstances. I create a tree made of nodes. foo() is called by a parent node in order to create its child nodes. The result returned by the processes are these child nodes. Those are saved in a list at the parent node. I want to parallelize the creation of those child nodes instead of creating them in a sequential way.
I think your issue has mainly to do with the fact that your parallelised function is a method of the object. It's hard to be certain without more information, but consider this little toy program:
import multiprocessing as mp
import numpy as np
import gc
class Object(object):
def __init__(self, _):
self.data = np.empty((100, 100, 100), dtype=np.float64)
class Container(object):
def __new__(cls):
self = object.__new__(cls)
print("Born")
return self
def __init__(self):
self.objects = []
def foo(self):
with mp.Pool(processes=3, maxtasksperchild=10) as pool:
result = pool.map_async(self.new_obj, range(50))
self.objects.extend(result.get())
pool.terminate()
gc.collect()
def new_obj(self, i):
return Object(i)
def __del__(self):
print("Dead")
if __name__ == '__main__':
c = Container()
for j in range(5):
c.foo()
Now Container is called only once, so you'd expect to see a "Born", followed by a "Dead" being printed out; but since the code being executed by the processes is a method of the container, this means the whole container has to be executed elsewhere ! Running this, you will see a stream of intermingled "Born" and "Dead" as your container is being rebuilt on every execution of map:
Born
Born
Born
Born
Born
Dead
Born
Dead
Dead
Born
Dead
Born
...
<MANY MORE LINES HERE>
...
Born
Dead
To convince yourself that the entire container is being copied and sent around every time, try to set some non-serialisable value:
def foo(self):
with mp.Pool(processes=3, maxtasksperchild=10) as pool:
result = pool.map_async(self.new_obj, range(50))
self.fn = lambda x: x**2
self.objects.extend(result.get())
pool.terminate()
gc.collect()
Which will immediately raise an AttributeError as it cannot serialise the container.
Let's sum up: when sending 1000 requests to the pool, Container will be serialised, sent to the processes and deserialised there a 1000 times. Sure, they will eventually be dropped (assuming there's not too much weird cross-referencing going on), but that will definitely put a lot of pressure on the RAM, as the object is serialised, called, updated, reserialised... for every element in your mapped inputs.
How can you solve that ? Well, ideally, do not share state:
def new_obj(_):
return Object(_)
class Container(object):
def __new__(cls):
self = object.__new__(cls)
print("Born")
return self
def __init__(self):
self.objects = []
def foo(self):
with mp.Pool(processes=3, maxtasksperchild=10) as pool:
result = pool.map_async(new_obj, range(50))
self.objects.extend(result.get())
pool.terminate()
gc.collect()
def __del__(self):
print("Dead")
This completes in a fraction of the time, and only produces the tiniest blimp on the RAM (as a single Container is ever built). If you need some of the internal state to be passed there, extract it and send just that:
def new_obj(tup):
very_important_state, parameters = tup
return Object(very_important_state=very_important_state,
parameters=parameters)
class Container(object):
def __new__(cls):
self = object.__new__(cls)
print("Born")
return self
def __init__(self):
self.objects = []
def foo(self):
important_state = len(self.objects)
with mp.Pool(processes=3, maxtasksperchild=10) as pool:
result = pool.map_async(new_obj,
((important_state, i) for i in range(50)))
self.objects.extend(result.get())
pool.terminate()
gc.collect()
def __del__(self):
print("Dead")
This has the same behaviour as before. If you absolutely cannot avoid sharing some mutable state between the processes, checkout out the multiprocessing tools for doing that without having to copy everything everywhere everytime.
Me and a friend are having a programming challenge to who can make a good VOS (Virtual Operating System) and currently mine is running custom programs from Threads within the program, I am using Tkinter currently so the separate Threads have their own self.master.mainloop(). I have all the Threads stored in a list but I was wondering whether I could call a function in the Thread which would call a subroutine in the program telling it to do self.master.destroy(). Is there any way to do this?
I would like something along the lines of
class ToBeThread():
def __init__(self):
self.master = Tk()
self.master.mainloop()
def on_stop(self, reason):
self.master.destroy()
Then in my main class
from threading import Thread
thread = Thread(ToBeThread())
thread.setDaemon(True)
thread.on_stop += ToBeThread.on_stop # Similar to how it is done in c#
thread.start()
...
...
thread.stop() # This calls the functions related to the "on_stop"
I have found a way to do this, so for any wondering I did:
from threading import Thread
class MyThread(Thread):def __init__(self, method, delay=-1):
Thread.__init__(self)
self.method = method
self._running = False
self.delay = delay
self.setDaemon(True)
def run(self):
self._running = True
while self._running == True:
self.method()
if self.delay != -1:
time.sleep(self.delay)
def stop(self):
self._running = False
This allows me to write pass a function in through the initialiser, and it will run it ever x seconds or as many times as possible until I do thread.stop()