I would like to log a variable (local scope) every second in python 3.5.
The call should be asyncronous (means the main thread should keep running).
I have tried the following code :
def functionX(self):
#do stuff
class BackgroundTimer(Thread):
def run(self):
while True:
print("sum_data ", sum)
sleep(60)
timer = BackgroundTimer()
timer.start()
#keep doing stuff
Thank you.
Related
the last 4 hours I have been trying to understand threading with django. Nothing seems to work. I want to let the website run in the foreground and let the backend communicate with some other devices on a thread. I want the thread to start at the startup of the website but the program is stuck when I call the thread until the thread comes to an end.
Do you know a way to fix it? Please I need help.
The urls.py file
def add(x, y):
i=0
while i < 100000000:
x += y
i += 1
def postpone(function):
t = threading.Thread(target=function, args=(1,))
t.setDaemon(True)
t.start()
return 0
print("Before thread")
postpone(add(4,4))
print("After thread")
The server will not start until the while loop is finished.
Thanks for reading, I hope someone knows an answer.
add function is called before the thread started, you need to pass add as reference though.
# decomposition
# first, add gets called
r = add(4,4)
# then the result is passed to func `postpone`
postpone(r)
# postpone accept a function and args, which eventually get passed to the function
def postpone(function, *args):
t = threading.Thread(target=function, args=args)
t.setDaemon(True)
t.start()
return 0
print("Before thread")
# pass func as a reference, also send args to the postpone func also
postpone(add, 4,4)
print("After thread")
Could you tell me if this is a correct approach to build several independent async loops inside own threads?
def init():
print("Initializing Async...")
global loop_heavy
loop_heavy = asyncio.new_event_loop()
start_loop(loop_heavy)
def start_loop(loop):
thread = threading.Thread(target=loop.run_forever)
thread.start()
def submit_heavy(task):
future = asyncio.run_coroutine_threadsafe(task, loop_heavy)
try:
future.result()
except Exception as e:
print(e)
def stop():
loop_heavy.call_soon_threadsafe(loop_heavy.stop)
async def heavy():
print("3. heavy start %s" % threading.current_thread().name)
await asyncio.sleep(3) # or await asyncio.sleep(3, loop=loop_heavy)
print("4. heavy done")
Then I am testing it with:
if __name__ == "__main__":
init()
print("1. submit heavy: %s" % threading.current_thread().name)
submit_heavy(heavy())
print("2. submit is done")
stop()
I am expecting to see 1->3->2->4 but in fact it is 1->3->4->2:
Initializing Async...
1. submit heavy: MainThread
3. heavy start Thread-1
4. heavy done
2. submit is done
I think that I miss something in understanding async and threads.
Threads are different. Why am I waiting inside MainThread until the job inside Thread-1 is finished?
Why am I waiting inside MainThread until the job inside Thread-1 is finished?
Good question, why are you?
One possible answer is, because you actually want to block the current thread until the job is finished. This is one of the reasons to put the event loop in another thread and use run_coroutine_threadsafe.
The other possible answer is that you don't have to if you don't want. You can simply return from submit_heavy() the concurrent.futures.Future object returned by run_coroutine_threadsafe, and leave it to the caller to wait for the result (or check if one is ready) at their own leisure.
Finally, if your goal is just to run a regular function "in the background" (without blocking the current thread), perhaps you don't need asyncio at all. Take a look at the concurrent.futures module, whose ThreadPoolExecutor allows you to easily submit a function to a thread pool and leave it to execute unassisted.
I will add one of the possible solutions that I found from the asyncio documentation.
I'm not sure that it is the correct way, but it works as expected (MainThread is not blocked by the execution of the child thread)
Running Blocking Code
Blocking (CPU-bound) code should not be called directly. For example, if a function performs a CPU-intensive calculation for 1 second, all concurrent asyncio Tasks and IO operations would be delayed by 1 second.
An executor can be used to run a task in a different thread or even in a different process to avoid blocking block the OS thread with the event loop. See the loop.run_in_executor() method for more details.
Applying to my code:
import asyncio
import threading
import concurrent.futures
import multiprocessing
import time
def init():
print("Initializing Async...")
global loop, thread_executor_pool
thread_executor_pool = concurrent.futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count())
loop = asyncio.get_event_loop()
thread = threading.Thread(target=loop.run_forever)
thread.start()
def submit_task(task, *args):
loop.run_in_executor(thread_executor_pool, task, *args)
def stop():
loop.call_soon_threadsafe(loop.stop)
thread_executor_pool.shutdown()
def blocked_task(msg1, msg2):
print("3. task start msg: %s, %s, thread: %s" % (msg1, msg2, threading.current_thread().name))
time.sleep(3)
print("4. task is done -->")
if __name__ == "__main__":
init()
print("1. --> submit task: %s" % threading.current_thread().name)
submit_task(blocked_task, "a", "b")
print("2. --> submit is done")
stop()
Output:
Initializing Async...
1. --> submit task: MainThread
3. task start msg: a, b, thread: ThreadPoolExecutor-0_0
2. --> submit is done
4. task is done -->
Correct me if there are still any mistakes or it can be done in the other way.
I have a simple twisted TCP server running absolutely fine, it basically deals with database requests and displays the right things its just an echo client with a bunch of functions, the database that is being read also updates I have this refresh function to open the database and refresh it however if I add this to the message functions it'll take too long to respond as the refresh function takes around 6/7 seconds to complete, my initial idea was to have this function in a while loop and running constantly refreshing every 5/10 mins but after reading about the global interpreter lock its made me think that that isn't possible, any suggestions on how to run this function in the background of my code would be greatly appreciated
I've tried having it in a thread but it doesn't seem to run at all when I start the thread, I put it under the if name == 'main': function and no luck!
Here is my refresh function
def refreshit()
Application = win32com.client.Dispatch("Excel.Application")
Workbook = Application.Workbooks.open(database)
Workbook.RefreshAll()
Workbook.Save()
Application.Quit()
xlsx = pd.ExcelFile(database)
global datess
global refss
df = pd.read_excel(xlsx, sheet_name='Sheet1')
datess = df.groupby('documentDate')
refss = df.groupby('reference')
class Echo(Protocol):
global Picked_DFS
Picked_DFS = None
label = None
global errors
global picked
errors = []
picked = []
def dataReceived(self, data):
"""
As soon as any data is received, write it back.
"""
response = self.handle_message(data)
print('responding with this')
print(response)
self.transport.write(response)
def main():
f = Factory()
f.protocol = Echo
reactor.listenTCP(8000, f)
reactor.run()
if __name__ == '__main__':
main()
I had tried this to no avail
if __name__ == '__main__':
main()
thread = Thread(target = refreshit())
thread.start()
thread.join()
You have an important error on this line:
thread = Thread(target = refreshit())
Though you have not included the definition of refreshit (perhaps a function to consider renaming), I assume refreshit is a function that performs your refresh.
In this case, what you are doing here is calling refreshit and waiting for it to return a value. Then, the value it returns is used as the target of the Thread you create here. This is probably not what you meant. Instead:
thread = Thread(target = refreshit)
That is, refreshit itself is what you want the target of the thread to be.
You also need to be sure to sequence your operations so that everything gets to run concurrently:
if __name__ == '__main__':
# Start your worker/background thread.
thread = Thread(target = refreshit)
thread.start()
# Run Twisted
main()
# Cleanup/wait on your worker/background thread.
thread.join()
You may also just want to use Twisted's thread support instead of using the threading module directly (but this is not mandatory).
if __name__ == '__main__':
# Start your worker/background thread.
thread = Thread(target = refreshit)
thread.start()
# Run Twisted
main()
# Cleanup/wait on your worker/background thread.
thread.join()
I have a question on multiprocessing and tkinter. I am having some problems getting my process to function parallel with the tkinter GUI. I have created a simple example to practice and have been reading up to understand the basics of multiprocessing. However when applying them to tkinter, only one process runs at the time. (Using Multiprocessing module for updating Tkinter GUI) Additionally, when I added the queue to communicate between processes, (How to use multiprocessing queue in Python?), the process won't even start.
Goal:
I would like to have one process that counts down and puts the values in the queue and one to update tkinter after 1 second and show me the values.
All advice is kindly appreciated
Kind regards,
S
EDIT: I want the data to be available when the after method is being called. So the problem is not with the after function, but with the method being called by the after function. It will take 0.5 second to complete the calculation each time. Consequently the GUI is unresponsive for half a second, each second.
EDIT2: Corrections were made to the code based on the feedback but this code is not running yet.
class Countdown():
"""Countdown prior to changing the settings of the flows"""
def __init__(self,q):
self.master = Tk()
self.label = Label(self.master, text="", width=10)
self.label.pack()
self.counting(q)
# Countdown()
def counting(self, q):
try:
self.i = q.get()
except:
self.label.after(1000, self.counting, q)
if int(self.i) <= 0:
print("Go")
self.master.destroy()
else:
self.label.configure(text="%d" % self.i)
print(i)
self.label.after(1000, self.counting, q)
def printX(q):
for i in range(10):
print("test")
q.put(9-i)
time.sleep(1)
return
if __name__ == '__main__':
q = multiprocessing.Queue()
n = multiprocessing.Process(name='Process2', target=printX, args = (q,))
n.start()
GUI = Countdown(q)
GUI.master.mainloop()
Multiprocessing does not function inside of the interactive Ipython notebook.
Multiprocessing working in Python but not in iPython As an alternative you can use spyder.
No code will run after you call mainloop until the window has been destroyed. You need to start your other process before you call mainloop.
You are calling wrong the after function. The second argument must be the name of the function to call, not a call to the function.
If you call it like
self.label.after(1000, self.counting(q))
It will call counting(q) and wait for a return value to assign as a function to call.
To assign a function with arguments the syntax is
self.label.after(1000, self.counting, q)
Also, start your second process before you create the window and call counting.
n = multiprocessing.Process(name='Process2', target=printX, args = (q,))
n.start()
GUI = Countdown(q)
GUI.master.mainloop()
Also you only need to call mainloop once. Either position you have works, but you just need one
Edit: Also you need to put (9-i) in the queue to make it count down.
q.put(9-i)
Inside the printX function
So I want to learn using moveToThread and see the effect of calling onTimeout() of class GenericWorker from a different thread (main thread in this case). The weird thing is that the finish_sig in GenericWorker never gets emitted (should happen at the last line of onTimeout() ). Since it connects to terminate_thread() in Sender class, it should at least print out a terminate_thread in the console, but nothing happens at all.
My original purpose for using it is to emit a signal to quit the thread after onTimeout() is done. But now I can only do t.quit() from main to quit the thread.
Thank you all for spending time taking care of my question!
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import threading
from time import sleep
import sys
class GenericWorker(QObject):
finish_sig = pyqtSignal() # this one never gets emitted!
#pyqtSlot(str, str)
def onTimeout(self, cmd1, cmd2):
print 'onTimeout get called from thread ID: '
print QThread.currentThreadId()
print 'received cmd 1: ' + cmd1
print 'received cmd 2: ' + cmd2
self.finish_sig.emit() # supposed to emit here!
class Sender(QObject):
send_sig = pyqtSignal(str, str)
terminate_sig = pyqtSignal()
def emit_sig(self, cmd):
print 'emit_sig thread ID: '
print QThread.currentThreadId()
sleep(1)
self.send_sig.emit(cmd, '2nd_cmd')
def terminate_thread(self):
print 'terminate_thread'
self.terminate_sig.emit()
if __name__ == "__main__":
app = QApplication(sys.argv)
print 'Main thread ID: '
print QThread.currentThreadId()
t = QThread()
my_worker = GenericWorker()
my_worker.moveToThread(t)
t.start()
my_sender = Sender()
my_sender.send_sig.connect(my_worker.onTimeout)
my_sender.terminate_sig.connect(t.quit)
my_worker.finish_sig.connect(my_sender.terminate_thread)
# my_worker.finish_sig.connect(t.quit)
my_sender.emit_sig('hello')
sleep(1)
# my_sender.terminate_thread()
# t.quit() # this one works
# t.wait()
exit(1)
sys.exit(app.exec_())
The output:
Main thread ID:
46965006517856
emit_sig thread ID:
46965006517856
onTimeout get called from thread ID:
1111861568
received cmd 1: hello
received cmd 2: 2nd_cmd
QThread: Destroyed while thread is still running
UPDATE:
After referring to #tmoreau and #ekhumoro's answers, there are two key problems with this code:
The exit(1) is not a proper way to exit, I need to remove this line.
I don't have a way to exit the QApplication, what I need to do is to add t.finish.connect(app.quit) to exit the application. (By the way, the last line sys.exit(app.exec_()) seems not taking care of the exiting of the QApplication)
In sum, there are basically three things that I need to exit: QThread, QApplication and sys, what I missed is to exit QApplication. Let me know if my understanding is right or not...
Your issue is that you exit the program before it's complete.
my_sender.emit_sig('hello')
sleep(1)
exit(1)
sys.exit(app.exec_())
exit() ends your program, even if the thread has not finished running, hence the error:
QThread: Destroyed while thread is still running
If you remove sleep(1), you'll see the program stops even earlier:
Main thread ID:
46965006517856
emit_sig thread ID:
46965006517856
QThread: Destroyed while thread is still running
Here's more or less what's happening in parallel:
# main thread #worker thread
my_sender.emit_sig('hello') #slot onTimeout is called
sleep(1) #print "onTimeout get called..."
exit(1) #emit finish_sig
sys.exit(app.exec_())
# slot terminate_thread is called #thread ends (t.quit)
If you remove exit(1), your program will work, because you create an event loop with app.exec_(). The event loop means your program is always waiting to catch signals, and will not stop even if there's nothing left to do. So the thread has plenty of time to end :)
In Qt, you usually stop the event loop by closing your main window. Therefore, a cleaner way to implement your thread is:
class window(QWidget):
def __init__(self,parent=None):
super(window,self).__init__(parent)
t=QThread(self)
self.my_worker = GenericWorker()
self.my_worker.moveToThread(t)
t.start()
self.my_sender = Sender()
self.my_sender.send_sig.connect(self.my_worker.onTimeout)
self.my_sender.terminate_sig.connect(t.quit)
self.my_worker.finish_sig.connect(self.my_sender.terminate_thread)
self.my_sender.emit_sig('hello')
if __name__ == "__main__":
app = QApplication(sys.argv)
win=window()
win.show()
sys.exit(app.exec_())
You need self to keep a reference to the thread and classes. Otherwise they are destroyed when __init__ ends.