So I want to learn using moveToThread and see the effect of calling onTimeout() of class GenericWorker from a different thread (main thread in this case). The weird thing is that the finish_sig in GenericWorker never gets emitted (should happen at the last line of onTimeout() ). Since it connects to terminate_thread() in Sender class, it should at least print out a terminate_thread in the console, but nothing happens at all.
My original purpose for using it is to emit a signal to quit the thread after onTimeout() is done. But now I can only do t.quit() from main to quit the thread.
Thank you all for spending time taking care of my question!
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import threading
from time import sleep
import sys
class GenericWorker(QObject):
finish_sig = pyqtSignal() # this one never gets emitted!
#pyqtSlot(str, str)
def onTimeout(self, cmd1, cmd2):
print 'onTimeout get called from thread ID: '
print QThread.currentThreadId()
print 'received cmd 1: ' + cmd1
print 'received cmd 2: ' + cmd2
self.finish_sig.emit() # supposed to emit here!
class Sender(QObject):
send_sig = pyqtSignal(str, str)
terminate_sig = pyqtSignal()
def emit_sig(self, cmd):
print 'emit_sig thread ID: '
print QThread.currentThreadId()
sleep(1)
self.send_sig.emit(cmd, '2nd_cmd')
def terminate_thread(self):
print 'terminate_thread'
self.terminate_sig.emit()
if __name__ == "__main__":
app = QApplication(sys.argv)
print 'Main thread ID: '
print QThread.currentThreadId()
t = QThread()
my_worker = GenericWorker()
my_worker.moveToThread(t)
t.start()
my_sender = Sender()
my_sender.send_sig.connect(my_worker.onTimeout)
my_sender.terminate_sig.connect(t.quit)
my_worker.finish_sig.connect(my_sender.terminate_thread)
# my_worker.finish_sig.connect(t.quit)
my_sender.emit_sig('hello')
sleep(1)
# my_sender.terminate_thread()
# t.quit() # this one works
# t.wait()
exit(1)
sys.exit(app.exec_())
The output:
Main thread ID:
46965006517856
emit_sig thread ID:
46965006517856
onTimeout get called from thread ID:
1111861568
received cmd 1: hello
received cmd 2: 2nd_cmd
QThread: Destroyed while thread is still running
UPDATE:
After referring to #tmoreau and #ekhumoro's answers, there are two key problems with this code:
The exit(1) is not a proper way to exit, I need to remove this line.
I don't have a way to exit the QApplication, what I need to do is to add t.finish.connect(app.quit) to exit the application. (By the way, the last line sys.exit(app.exec_()) seems not taking care of the exiting of the QApplication)
In sum, there are basically three things that I need to exit: QThread, QApplication and sys, what I missed is to exit QApplication. Let me know if my understanding is right or not...
Your issue is that you exit the program before it's complete.
my_sender.emit_sig('hello')
sleep(1)
exit(1)
sys.exit(app.exec_())
exit() ends your program, even if the thread has not finished running, hence the error:
QThread: Destroyed while thread is still running
If you remove sleep(1), you'll see the program stops even earlier:
Main thread ID:
46965006517856
emit_sig thread ID:
46965006517856
QThread: Destroyed while thread is still running
Here's more or less what's happening in parallel:
# main thread #worker thread
my_sender.emit_sig('hello') #slot onTimeout is called
sleep(1) #print "onTimeout get called..."
exit(1) #emit finish_sig
sys.exit(app.exec_())
# slot terminate_thread is called #thread ends (t.quit)
If you remove exit(1), your program will work, because you create an event loop with app.exec_(). The event loop means your program is always waiting to catch signals, and will not stop even if there's nothing left to do. So the thread has plenty of time to end :)
In Qt, you usually stop the event loop by closing your main window. Therefore, a cleaner way to implement your thread is:
class window(QWidget):
def __init__(self,parent=None):
super(window,self).__init__(parent)
t=QThread(self)
self.my_worker = GenericWorker()
self.my_worker.moveToThread(t)
t.start()
self.my_sender = Sender()
self.my_sender.send_sig.connect(self.my_worker.onTimeout)
self.my_sender.terminate_sig.connect(t.quit)
self.my_worker.finish_sig.connect(self.my_sender.terminate_thread)
self.my_sender.emit_sig('hello')
if __name__ == "__main__":
app = QApplication(sys.argv)
win=window()
win.show()
sys.exit(app.exec_())
You need self to keep a reference to the thread and classes. Otherwise they are destroyed when __init__ ends.
Related
I'm facing problem with the thread concept i.e I have a function which will create 10 threads to do task. If any key board interruption occurs, those created threads are still executing and i would like to stop those threads and revert back the changes.
The following code sinppet is the sample approach:
def store_to_db(self,keys_size,master_key,action_flag,key_status):
for iteration in range(10):
t = threading.Thread(target=self.store_worker, args=())
t.start()
threads.append(t)
for t in threads:
t.join()
def store_worker():
print "DOING"
The idea to make this work is:
you need a "thread pool" where threads are checking against if their do_run attribute is falsy.
you need a "sentinel thread" outside that pool which checks the thread status in the pool and adjusts the do_run attribute of the "thread pool" thread on demand.
Example code:
import threading
import random
import time
import msvcrt as ms
def main_logic():
# take 10 worker threads
threads = []
for i in range(10):
t = threading.Thread(target=lengthy_process_with_brake, args=(i,))
# start and append
t.start()
threads.append(t)
# start the thread which allows you to stop all threads defined above
s = threading.Thread(target=sentinel, args=(threads,))
s.start()
# join worker threads
for t in threads:
t.join()
def sentinel(threads):
# this one runs until threads defined in "threads" are running or keyboard is pressed
while True:
# number of threads are running
running = [x for x in threads if x.isAlive()]
# if kb is pressed
if ms.kbhit():
# tell threads to stop
for t in running:
t.do_run = False
# if all threads stopped, exit the loop
if not running:
break
# you don't want a high cpu load for nothing
time.sleep(0.05)
def lengthy_process_with_brake(worker_id):
# grab current thread
t = threading.currentThread()
# start msg
print(f"{worker_id} STARTED")
# exit condition
zzz = random.random() * 20
stop_time = time.time() + zzz
# imagine an iteration here like "for item in items:"
while time.time() < stop_time:
# the brake
if not getattr(t, "do_run", True):
print(f"{worker_id} IS ESCAPING")
return
# the task
time.sleep(0.03)
# exit msg
print(f"{worker_id} DONE")
# exit msg
print(f"{worker_id} DONE")
main_logic()
This solution does not 'kill' threads, just tell them to stop iterating or whatever they do.
EDIT:
I just noticed that "Keyboard exception" was in the title and not "any key". Keyboard Exception handling is a bit different, here is a good solution for that. The point is almost the same: you tell the thread to return if a condition is met.
Could you tell me if this is a correct approach to build several independent async loops inside own threads?
def init():
print("Initializing Async...")
global loop_heavy
loop_heavy = asyncio.new_event_loop()
start_loop(loop_heavy)
def start_loop(loop):
thread = threading.Thread(target=loop.run_forever)
thread.start()
def submit_heavy(task):
future = asyncio.run_coroutine_threadsafe(task, loop_heavy)
try:
future.result()
except Exception as e:
print(e)
def stop():
loop_heavy.call_soon_threadsafe(loop_heavy.stop)
async def heavy():
print("3. heavy start %s" % threading.current_thread().name)
await asyncio.sleep(3) # or await asyncio.sleep(3, loop=loop_heavy)
print("4. heavy done")
Then I am testing it with:
if __name__ == "__main__":
init()
print("1. submit heavy: %s" % threading.current_thread().name)
submit_heavy(heavy())
print("2. submit is done")
stop()
I am expecting to see 1->3->2->4 but in fact it is 1->3->4->2:
Initializing Async...
1. submit heavy: MainThread
3. heavy start Thread-1
4. heavy done
2. submit is done
I think that I miss something in understanding async and threads.
Threads are different. Why am I waiting inside MainThread until the job inside Thread-1 is finished?
Why am I waiting inside MainThread until the job inside Thread-1 is finished?
Good question, why are you?
One possible answer is, because you actually want to block the current thread until the job is finished. This is one of the reasons to put the event loop in another thread and use run_coroutine_threadsafe.
The other possible answer is that you don't have to if you don't want. You can simply return from submit_heavy() the concurrent.futures.Future object returned by run_coroutine_threadsafe, and leave it to the caller to wait for the result (or check if one is ready) at their own leisure.
Finally, if your goal is just to run a regular function "in the background" (without blocking the current thread), perhaps you don't need asyncio at all. Take a look at the concurrent.futures module, whose ThreadPoolExecutor allows you to easily submit a function to a thread pool and leave it to execute unassisted.
I will add one of the possible solutions that I found from the asyncio documentation.
I'm not sure that it is the correct way, but it works as expected (MainThread is not blocked by the execution of the child thread)
Running Blocking Code
Blocking (CPU-bound) code should not be called directly. For example, if a function performs a CPU-intensive calculation for 1 second, all concurrent asyncio Tasks and IO operations would be delayed by 1 second.
An executor can be used to run a task in a different thread or even in a different process to avoid blocking block the OS thread with the event loop. See the loop.run_in_executor() method for more details.
Applying to my code:
import asyncio
import threading
import concurrent.futures
import multiprocessing
import time
def init():
print("Initializing Async...")
global loop, thread_executor_pool
thread_executor_pool = concurrent.futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count())
loop = asyncio.get_event_loop()
thread = threading.Thread(target=loop.run_forever)
thread.start()
def submit_task(task, *args):
loop.run_in_executor(thread_executor_pool, task, *args)
def stop():
loop.call_soon_threadsafe(loop.stop)
thread_executor_pool.shutdown()
def blocked_task(msg1, msg2):
print("3. task start msg: %s, %s, thread: %s" % (msg1, msg2, threading.current_thread().name))
time.sleep(3)
print("4. task is done -->")
if __name__ == "__main__":
init()
print("1. --> submit task: %s" % threading.current_thread().name)
submit_task(blocked_task, "a", "b")
print("2. --> submit is done")
stop()
Output:
Initializing Async...
1. --> submit task: MainThread
3. task start msg: a, b, thread: ThreadPoolExecutor-0_0
2. --> submit is done
4. task is done -->
Correct me if there are still any mistakes or it can be done in the other way.
I Am trying to send a signal from a child thread to the main thread in a multi-threaded program (cannot use multi-processes). Unfortunately even after exhausting all the reading materials available online (which I could find), I Am unable to get a clear idea of how to do so. I Am a beginner to signals AND to python so please bear with me and explain as you would to a novice.
I cannot use the join method in the process, since I want both the threads to be running simultaneously.
Here is the code that I found related to the topic here - http://pymotw.com/2/signal/#signals-and-threads
and it doesn't really work for me.
import signal
import threading
import os
import time
def signal_handler(num, stack):
print 'Received signal %d in %s' % (num, threading.currentThread())
signal.signal(signal.SIGUSR1, signal_handler)
def wait_for_signal():
print 'Waiting for signal in', threading.currentThread()
signal.pause()
print 'Done waiting'
# Start a thread that will not receive the signal
receiver = threading.Thread(target=wait_for_signal, name='receiver')
receiver.start()
time.sleep(0.1)
def send_signal():
print 'Sending signal in', threading.currentThread()
os.kill(os.getpid(), signal.SIGUSR1)
sender = threading.Thread(target=send_signal, name='sender')
sender.start()
sender.join()
# Wait for the thread to see the signal (not going to happen!)
print 'Waiting for', receiver
signal.alarm(2)
receiver.join()
Please explain with a multi-threaded example if possible.
Thanks in advance!
Signals and threads really, really don't play nice together.
Consider use an Event or other synchronization mechanism. The following example creates an 'event' object, then passes it to two threads. One waits for two seconds, then signals the other to print out a message then exit.
source
import threading, time
def flagger_thread(event):
"""
wait for two seconds, then make 'event' fire
"""
time.sleep(2)
event.set()
def waiter_thread(event):
print("Waiting for event")
if event.wait(5):
print("event set.")
else:
print("Timed out.")
stop_event = threading.Event()
threading.Thread(target=flagger_thread, args=[stop_event]).start()
threading.Thread(target=waiter_thread, args=[stop_event]).start()
# wait for all threads to exit
for t in threading.enumerate():
if t != threading.current_thread():
t.join()
output
Waiting for event
event set.
I would like to execute a certain signal handler in a multi-threading python program.
I noted that my signal handler is called for main thread and the other thread (thread_1) that I have created.
Is there any way to execute the signal handler only for thread_1?
Or is there any way to know the thread ID of the thread that calls the signal handler? Is it the main thread or thread_1?
import signal
import sys
from multiprocessing import Process
from time import sleep
x = 3;
def f(name):
print 'hello', name
while(1):
sleep (1)
print "AAAAAAAAAAAAAAAAAAAAAAAAA"
def signal_handler(signal, frame):
global x
print 'You pressed Ctrl+C!'
print x
sys.exit(0)
if __name__ == '__main__':
signal.signal(signal.SIGINT, signal_handler)
print 'Creating thread'
p = Process(target=f, args=('bob',))
p.start()
print 'Press Ctrl+C'
x += 6
signal.pause()
print 'After press CTRL+C'
Got an output:
Creating thread
Press Ctrl+C
hello bob
AAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAA
^CYou pressed Ctrl+C!
You pressed Ctrl+C!
9
3
As you can signal handler is called twice (one for main thread and another for thread_1), and as I asked above, I would like to execute handler only for thread_1 or even find a way to get thread_id of the calling thread?
The signal is handled in two separate processes rather than two distinct threads of a single process. (Hint: you import multiprocessing rather than import threading.)
The child process inherits the signal handler for SIGINT and gets its own copy of x. Now, the shell runs both your parent and child processes in a foreground process group, and sends keyboard-generated signals (like Ctrl-C => SIGINT) to the whole process group, which is why both parent and child receive the signal. Both parent and child, then, print their own value of x.
If you do switch to a threaded implementation, then only the main thread will receive signals, and your question will be moot.
I'm using a thread to read Strings from a stream (/dev/tty1) while processing other things in the main loop. I would like the Thread to terminate together with the main program when pressing CTRL-C.
from threading import Thread
class myReader(Thread):
def run(self):
with open('/dev/tty1', encoding='ascii') as myStream:
for myString in myStream:
print(myString)
def quit(self):
pass # stop reading, close stream, terminate the thread
myReader = Reader()
myReader.start()
while(True):
try:
pass # do lots of stuff
KeyboardInterrupt:
myReader.quit()
raise
The usual solution - a boolean variable inside the run() loop - doesn't work here. What's the recommended way to deal with this?
I can just set the Daemon flag, but then I won't be able to use a quit() method which might prove valuable later (to do some clean-up). Any ideas?
AFAIK, there is no built-in mechanism for that in Python 3 (just as in Python 2). Have you tried the proven Python 2 approach with PyThreadState_SetAsyncExc, documented here and here, or the alternative tracing approach here?
Here's a slightly modified version of the PyThreadState_SetAsyncExc approach from above:
import threading
import inspect
import ctypes
def _async_raise(tid, exctype):
"""raises the exception, performs cleanup if needed"""
if not inspect.isclass(exctype):
exctype = type(exctype)
res = ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(tid), ctypes.py_object(exctype))
if res == 0:
raise ValueError("invalid thread id")
elif res != 1:
# """if it returns a number greater than one, you're in trouble,
# and you should call it again with exc=NULL to revert the effect"""
ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None)
raise SystemError("PyThreadState_SetAsyncExc failed")
def stop_thread(thread):
_async_raise(thread.ident, SystemExit)
Make your thread a daemon thread. When all non-daemon threads have exited, the program exits. So when Ctrl-C is passed to your program and the main thread exits, there's no need to explicitly kill the reader.
myReader = Reader()
myReader.daemon = True
myReader.start()