I have two function, one called connect() and is used to login to my account and another called identify_2bgval() which grabs some data from my account.Since its a very long proccess i tried to use multithreading.
My question is: Can i use these two functions simultaneously?
I have thought to create a new function and then use Threads.Is it gonna work?Also, can i use double targets with the Thread() function?
run_together():
connect()
identify_2bgval()
Thread(target= run_together).start()
If the identify_2bgval does not depend on connect you can run it in a different thread
def run_together():
t = Thread(target=identify_2bgval)
t.start() #2nd thread
connect() # main thread waits here ...
t.join() # wait for the 2nd thread ...
#call it ...
run_together() #blocks until finished ...
Related
I copied the following code for my project and it's worked quite well for me but I don't really understand how the following code runs my blocking_function:
#client.event
async def on_message(message):
loop = asyncio.get_event_loop()
block_response = await loop.run_in_executor(ThreadPoolExecutor(), blocking_function)
where on_message is called every time I receive a message. If I receive multiple messages, they are processed asynchronously.
blocking_function is a synchronous function that I don't want to be run when another blocking_function is running.Then within blocking_function, should I use threading.Lock() or asyncio.lock()?
As pointed out by dirn in the comment, in blocking_function you cannot use an asyncio.Lock because it's just not async. (The opposite also applies: you cannot lock a threading.Lock from an async function because attempting to do so would block the event loop.) If you need to guard data accessed by other instances of blocking_function, you should use a threading.Lock.
but I don't really understand how the following code runs my blocking_function
It hands off blocking_function to the thread pool you created to run it. The thread pool queues and runs the function (which happens "in the background" from your perspective), and the run_in_executor arranges the event loop to be notified when the function is done, handing off its return value as the result of the await expression.
Note that you should use None as the first argument of run_in_executor. If you use ThreadPoolExecutor(), you create a whole new thread pool for each message, and you never dispose of it. A thread pool is normally meant to be created once, and reuse a fixed number ("pool") of threads for subsequent work. None tells asyncio to use the thread pool it creates for this purpose.
It seems you can easily achieve your desired objective by ensuring a single thread is used.
A simple solution would be to ensure that all calls to blocking_function is run on a single thread. This can be easily achieved by creating a ThreadPoolExecutor object with 1 worker outside of the async function. Then every subsequent calls to the blocking function will be run on that single thread
thread_pool = ThreadPoolExecutor(max_workers=1)
#client.event
async def on_message(message):
loop = asyncio.get_event_loop()
block_response = await loop.run_in_executor(thread_pool, blocking_function)
Don't forget to shutdown the thread afterwards.
I have a function that will compute something then put it on a PDF. However, this process takes time. Because of this, we implement stop button for the user to stop the process.
I tried using thread.Event(), but the function I am working with is a non-looping function. This means that, I can't regularly check if event is set. (Hard coding the application by simply putting multiple checker will do but nope -- that idea is not accepted).
def generate_data(self, event):
self.thread = threading.Thread(target=self.check_event)
self.thread.start()
...
def check_event(self):
while True:
if self.event.is_set():
self.controller.enable_run_btn_and_tab()
return
time.sleep(1)
My idea is to create another thread that will regularly check if event is set so I can return and exit the function. However, the above code for def check_event(self) will only exit the child thread, not the thread for generate_date(self).
Is there some code or modification in my code to stop the main thread using child thread?
I am making a bot for telegram, this bot will use a database (SQLite3).
I am familiar with threads and locks and I know that is safe to launch multiple thread that make query to the database.
My problem rises when I want to update/insert data.
With the use Condition and Event from the threading module, I can prevent new thread to access the database while a thread is updating/inserting data.
What I haven't figured out is how to wait that all the thread that are accessing the database are done, before updating/inserting data.
If I could get the count of semaphore I would just wait for it to drop to 0, but since is not possible, what approach should I use?
UPDATE: I can't use join() since I am using telegram bot and create thread dynamically with each request to my bot, therefore when a thread is created I don't know if I'll have to wait for it to end or not.
CLARIFICATION: join() can only be used if, at the start of a thread you know wether you'll have to wait for it to end or not. Since I create a thread for each request of my clients and I am unaware of what they'll ask or and when the request will be done, I can't know whether to use join() or not.
UPDATE2: Here the code regarding the locks. I haven't finished the code regarding the database since I am more concerned with the locks and it doesn't seems relevant to the question.
lock = threading.Lock()
evLock = threading.Event()
def addBehaviours(dispatcher):
evLock.set()
# (2) Fetch the list of events
events_handler = CommandHandler('events', events)
dispatcher.add_handler(events_handler)
# (3) Add a new event
addEvent_handler = CommandHandler('addEvent', addEvent)
dispatcher.add_handler(addEvent_handler)
# (2) Fetch the list of events
#run_async
def events(bot, update):
evLock.wait()
# fetchEvents()
# (3) Add a new event
#run_async
def addEvent(bot, update):
with lock:
evLock.clear()
# addEvent()
evLock.set()
You can use threading.Thread.join(). This will wait for a thread to end and only continue on when the thread is done.
Usage below:
import threading as thr
thread1 = thr.Thread() # some thread to be waited for
thread1 = thr.Thread() # something that runs after thread1 finishes
thread1.start() # start up this thread
thread1.join() # wait until this thread finishes
thread2.start()
...
I'm creating 5 threads for handling various tasks such as reading from sensors (Raspberry Pi), TCP connections and recently recording audio (pyAudio).
I am instantiating all threads in main() identically e.g.:
if __name__ == '__main__':
main()
def main():
global network_thread
network_thread = threading.Thread(name="NET-CONN", target=network_thread_run, args=())
network_thread.start()
I keep a global reference so I can kill the threads at shutdown with join().
Now, I have added thread #5:
global audio_thread
audio_thread = threading.Thread(name="AUDIO", target=audio_thread_run(), args=())
audio_thread.start()
...but my logging indicates it's running on the MainThread. I also double-checked inside the audio_thread_run() function and it is indeed running on MainThread:
if threading.current_thread() is threading.main_thread():
logger.warning("Audio thread is the same as MainThread!")
Why is this thread running on the MainThread? Have I hit a limit on the Pi?
Let's have a look at the two places where you create threads, modified slightly so they'll fit on one line, and with white-space inserted so they line up:
net_thread = threading.Thread(name="NET", target=net_run , args=())
aud_thread = threading.Thread(name="AUD", target=aud_run(), args=())
# Hmmm, what's this suspicious-looking thing here? ---->^^
Enough fun :-) The problem is that you are actually calling the audio_thread_run() function directly from your main thread and presumably, if it ever returned, you would then try to use the result as a callable to start a thread.
If you actually got rid of the thread start stuff altogether, it would boil down to the much simpler:
audio_thread_run()
which will very much run that function from the context of the main thread.
What you need to do is to remove the parentheses so it matches what you've down with the network threads:
audio_thread = threading.Thread(name="AUDIO", target=audio_thread_run, args=())
I have a code that calls to threads over a loop, something like this:
def SubmitData(data):
# creating the relevant command to execute
command = CreateCommand(data)
subprocess.call(command)
def Main():
while(True):
# generating some data
data = GetData()
MyThread = threading.Thread(target=SubmitData,args=(data,))
MyThread.start()
obviously, I don't use join() on the threads.
My question is how to join() those threads without making the main thread wait for them?
Do I even need to join() them? what will happend if I won't join() them?
some important points:
the while loop is suppose to for a very long time (couple of days)
the command itself is not very long (few seconds)
I'm using threading for Performance so if someone have a better idea instead, I would like to try it out.
Popen() doesn't block. Unless CreateCommand() blocks, you could call SubmitData() in the main thread:
from subprocess import Popen
processes = []
while True:
processes = [p for p in processes if p.poll() is None] # leave only running
processes.append(Popen(CreateCommand(GetData()))) # start a new one
Do I even need to join() them? what will happend if I won't join() them?
No. You don't need to join them. All non-daemonic threads are joined automatically when the main thread exits.