I tried the following:
loop = asyncio.get_running_loop
task1 = asyncio.create_task(any_coroutine())
if task1 in asyncio.all_tasks(loop):
do something...
but it never meets the if condition. Can someone help?
According to document, as create_task calls get_running_loop you don't have to get event loop - assuming you're already running event loop so get_running_loop() didn't raise following error which your code does:
RuntimeError: no running event loop
Instead, try new async and await keywords, loop arguments will be deprecated on python 3.10 so it's better move if you're learning asyncio.
Following example will meet condition in if block correctly.
import asyncio
async def my_coro():
await asyncio.sleep(5)
async def main():
task = asyncio.create_task(my_coro())
if task in asyncio.all_tasks():
print('Task found!')
else:
print('Missed!')
if __name__ == '__main__':
asyncio.run(main())
Related
I've a situation like below,
event_loop = asyncio.new_event_loop()
async def second_async():
# some async job
print("I'm here")
return 123
def sync():
return asyncio.run_coroutine_threadsafe(second_async(), loop=event_loop).result()
async def first_async():
sync()
event_loop.run_until_complete(first_async())
I call the sync function from a different thread(where the event_loop is not running), it works fine. The problem is if I run the event_loop.run_complete... line, the .result() call on the Task returned by run_coroutine_threadsafe blocks the execution of the loop, which makes sense. To avoid this, I tried changing this as follows,
event_loop = asyncio.new_event_loop()
async def second_async():
# some async job
print("I'm here")
return 123
def sync():
# if event_loop is running on current thread
res = loop.create_task(second_async()).result()
# else
res = asyncio.run_coroutine_threadsafe(second_async(), loop=event_loop).result()
# Additional processing on res
# Need to evaluate the result of task right here in sync.
return res
async def first_async():
sync()
event_loop.run_until_complete(first_async())
This works fine, but the .result() call on the Task object returned by create_task always raises an InvalidStateError. The set_result is never called on the Task object.
Basically, I want the flow to be as such
(async code) -> sync code (a non blocking call ->) async code
I know this is a bad way of doing things, but I'm integrating stuff, so I don't really have an option.
Here is a little single-threaded program that illustrates the problem.
If you un-comment the line asyncio.run(first_async1()), you see the same error as you're seeing, and for the same reason. You're trying to access the result of a task without awaiting it first.
import asyncio
event_loop = asyncio.new_event_loop()
async def second_async():
# some async job
print("I'm here")
return 123
def sync1():
return asyncio.create_task(second_async()).result()
async def first_async1():
print(sync1())
def sync2():
return asyncio.create_task(second_async())
async def first_async2():
print(await sync2())
# This prints I'm here,
# the raises invalid state error:
# asyncio.run(first_async1())
# This works, prints "I'm here" and "123"
asyncio.run(first_async2())
With that line commented out again, the second version of the program (first_async2) runs just fine. The only difference is that the ordinary function, sync2, returns an awaitable instead of a result. The await is done in the async function that called it.
I don't see why this is a bad practice. To me, it seems like there are situations where it's absolutely necessary.
Another approach is to create a second daemon thread and set up an event loop there. Coroutines can be executed in this second thread with asyncio.run_coroutine_threadsafe, which returns a concurrent.futures.Future. Its result method will block until the Future's value is set by the other thread.
#! python3.8
import asyncio
import threading
def a_second_thread(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
loop2 = asyncio.new_event_loop()
threading.Thread(target=a_second_thread, args=(loop2,), daemon=True).start()
async def second_async():
# some async job
print("I'm here")
for _ in range(4):
await asyncio.sleep(0.25)
print("I'm done")
return 123
def sync1():
# Run the coroutine in the second thread -> get a concurrent.futures.Future
fut = asyncio.run_coroutine_threadsafe(second_async(), loop2)
return fut.result()
async def first_async1():
print(sync1())
def sync2():
return asyncio.create_task(second_async())
async def first_async2():
print(await sync2())
# This works, prints "I'm here", "I'm done", and "123"
asyncio.run(first_async1())
# This works, prints "I'm here", "I'm done", and "123"
asyncio.run(first_async2())
Of course this will still block the event loop in the main thread until fut.result() returns. There is no avoiding that. But the program runs.
So apologies because i've seen this question asked a bunch, but having looked through all of the questions none seem to fix my problem. My code looks like this
TDSession = TDClient()
TDSession.grab_refresh_token()
q = queue.Queue(10)
asyncio.run(listener.startStreaming(TDSession, q))
while True:
message = q.get()
print('oh shoot!')
print(message)
orderEntry.placeOrder(TDSession=TDSession)
I have tried doing asyncio.create_task(listener.startStreaming(TDSession,q)), the problem is I get
RuntimeError: no running event loop
sys:1: RuntimeWarning: coroutine 'startStreaming' was never awaited
which confused me because this seemed to work here Can an asyncio event loop run in the background without suspending the Python interpreter? which is what i'm trying to do.
with the listener.startStreaming func looking like this
async def startStreaming(TDSession, q):
streamingClient = TDSession.create_streaming_session()
streamingClient.account_activity()
await streamingClient.build_pipeline()
while True:
message = await streamingClient.start_pipeline()
message = parseMessage(message)
if message != None:
print('putting message into q')
print( dict(message) )
q.put(message)
Is there a way to make this work where I can run the listener in the background?
EDIT: I've tried this as well, but it only runs the consumer function, instead of running both at the same time
TDSession.grab_refresh_token()
q = queue.Queue(10)
loop = asyncio.get_event_loop()
loop.create_task(listener.startStreaming(TDSession, q))
loop.create_task(consumer(TDSession, q))
loop.run_forever()
As you found out, the asyncio.run function runs the given coroutine until it is complete. In other words, it waits for the coroutine returned by listener.startStreaming to finish before proceeding to the next line.
Using asyncio.create_task, on the other hand, requires the caller to be already running inside an asyncio loop already. From the docs:
The task is executed in the loop returned by get_running_loop(), RuntimeError is raised if there is no running loop in current thread.
What you need is to combine the two, by creating a function that's async, and then call create_task inside that async function.
For example:
async def main():
TDSession = TDClient()
TDSession.grab_refresh_token()
q = asyncio.Queue(10)
streaming_task = asyncio.create_task(listener.startStreaming(TDSession, q))
while True:
message = await q.get()
print('oh shoot!')
print(message)
orderEntry.placeOrder(TDSession=TDSession)
await streaming_task # If you want to wait for `startStreaming` to complete after the while loop
if __name__ == '__main__':
asyncio.run(main())
Edit: From your comment I realized you want to use the producer-consumer pattern, so I also updated the example above to use asyncio.Queue instead of a queue.Queue, in order for the thread to be able to jump between the producer (startStreaming) and the consumer (the while loop)
Recently, I have moved my REST server code in express.js to using FastAPI. So far, I've been successful in the transition until recently. I've noticed based on the firebase python admin sdk documention, unlike node.js, the python sdk is blocking. The documentation says here:
In Python and Go Admin SDKs, all write methods are blocking. That is, the write methods do not return until the writes are committed to the database.
I think this feature is having a certain effect on my code. It also could be how I've structured my code as well. Some code from one of my files is below:
from app.services.new_service import nService
from firebase_admin import db
import json
import redis
class TryNewService:
async def tryNew_func(self, request):
# I've already initialized everything in another file for firebase
ref = db.reference()
r = redis.Redis()
holdingData = await nService().dialogflow_session(request)
fulfillmentText = json.dumps(holdingData[-1])
body = await request.json()
if ("user_prelimInfo_address" in holdingData):
holdingData.append("session")
holdingData.append(body["session"])
print(holdingData)
return(holdingData)
else:
if (("Default Welcome Intent" in holdingData)):
pass
else:
UserVal = r.hget(name='{}'.format(body["session"]), key="userId").decode("utf-8")
ref.child("users/{}".format(UserVal)).child("c_data").set({holdingData[0]:holdingData[1]})
print(holdingData)
return(fulfillmentText)
Is there any workaround for the blocking effect of usingref.set() line in my code? Kinda like adding a callback in node.js? I'm new to the asyncio world of python 3.
Update as of 06/13/2020: So I added following code and am now getting a RuntimeError: Task attached to a different loop. In my second else statement I do the following:
loop = asyncio.new_event_loop()
UserVal = r.hget(name='{}'.format(body["session"]), key="userId").decode("utf-8")
with concurrent.futures.ThreadPoolExecutor(max_workers=20) as pool:
result = await loop.run_in_executor(pool, ref.child("users/{}".format(UserVal)).child("c_data").set({holdingData[0]:holdingData[1]}))
print("custom thread pool:{}".format(result))
With this new RuntimeError, I would appreciate some help in figuring out.
If you want to run synchronous code inside an async coroutine, then the steps are:
loop = get_event_loop()
Note: Get and not new. Get provides current event_loop, and new_even_loop returns a new one
await loop.run_in_executor(None, sync_method)
First parameter = None -> use default executor instance
Second parameter (sync_method) is the synchronous code to be called.
Remember that resources used by sync_method need to be properly synchronized:
a) either using asyncio.Lock
b) or using asyncio.run_coroutine_threadsafe function(see an example below)
Forget for this case about ThreadPoolExecutor (that provides a way to I/O parallelism, versus concurrency provided by asyncio).
You can try following code:
loop = asyncio.get_event_loop()
UserVal = r.hget(name='{}'.format(body["session"]), key="userId").decode("utf-8")
result = await loop.run_in_executor(None, sync_method, ref, UserVal, holdingData)
print("custom thread pool:{}".format(result))
With a new function:
def sync_method(ref, UserVal, holdingData):
result = ref.child("users/{}".format(UserVal)).child("c_data").set({holdingData[0]:holdingData[1]}))
return result
Please let me know your feedback
Note: previous code it's untested. I have only tested next minimum example (using pytest & pytest-asyncio):
import asyncio
import time
import pytest
#pytest.mark.asyncio
async def test_1():
loop = asyncio.get_event_loop()
delay = 3.0
result = await loop.run_in_executor(None, sync_method, delay)
print(f"Result = {result}")
def sync_method(delay):
time.sleep(delay)
print(f"dddd {delay}")
return "OK"
Answer #jeff-ridgeway comment:
Let's try to change previous answer to clarify how to use run_coroutine_threadsafe, to execute from a sync worker thread a coroutine that gather these shared resources:
Add loop as additional parameter in run_in_executor
Move all shared resources from sync_method to a new async_method, that is executed with run_coroutine_threadsafe
loop = asyncio.get_event_loop()
UserVal = r.hget(name='{}'.format(body["session"]), key="userId").decode("utf-8")
result = await loop.run_in_executor(None, sync_method, ref, UserVal, holdingData, loop)
print("custom thread pool:{}".format(result))
def sync_method(ref, UserVal, holdingData, loop):
coro = async_method(ref, UserVal, holdingData)
future = asyncio.run_coroutine_threadsafe(coro, loop)
future.result()
async def async_method(ref, UserVal, holdingData)
result = ref.child("users/{}".format(UserVal)).child("c_data").set({holdingData[0]:holdingData[1]}))
return result
Note: previous code is untested. And now my tested minimum example updated:
#pytest.mark.asyncio
async def test_1():
loop = asyncio.get_event_loop()
delay = 3.0
result = await loop.run_in_executor(None, sync_method, delay, loop)
print(f"Result = {result}")
def sync_method(delay, loop):
coro = async_method(delay)
future = asyncio.run_coroutine_threadsafe(coro, loop)
return future.result()
async def async_method(delay):
time.sleep(delay)
print(f"dddd {delay}")
return "OK"
I hope this can be helpful
Run blocking database calls on the event loop using a ThreadPoolExecutor. See https://medium.com/#hiranya911/firebase-python-admin-sdk-with-asyncio-d65f39463916
This first example does not work, I try to emit an async task but don't care about the response in that situation, output is empty:
from typing import Iterable
import asyncio
async def example_task():
print('example_task')
def emit() -> Iterable:
event_loop = asyncio.get_event_loop()
yield event_loop.create_task(example_task())
async def main():
emit()
await asyncio.sleep(0.5) # wait some time so the task can run
asyncio.run(main())
When I add next(emit()) to actually "read" the yielded task the output works and also in the next example it works when I put all the task into a list first:
from typing import Iterable
import asyncio
async def example_task():
print('example_task')
def emit() -> Iterable:
event_loop = asyncio.get_event_loop()
return iter([event_loop.create_task(example_task())])
async def main():
emit()
await asyncio.sleep(0.5) # wait some time so the task can run
asyncio.run(main())
This is just a simple example, the final version should be able to emit an "event" and run 1..n async tasks that can return a value but don't need to. The caller of emit should be able to decide if he awaits the result at some point or just ignore it like in the examples.
Is there any way I can do this with a generator / yield or is the only possible way to store all the tasks in a list and return an iterator after that?
The issue is that you are returning a generator with the first example where the second example has the task object which needs to be executed.
The modified version of your first example would be something like
async def main():
next(emit())
await asyncio.sleep(0.5) # wait some time so the task can run
or
async def main():
for task in emit():
await task
await asyncio.sleep(0.5) # wait some time so the task can run
Hope this explains the difference between using a generator and a iterator while creating your tasks.
I've include asyncio to asynchronous my code in my library twbotlib (https://github.com/truedl/twbotlib).
I tried the asynced commands some versions ago and all go well, but I don't check about if is really asynced. Then I've tried to create a giveaway command and use await asyncio.sleep(5). I realized that is blocking all my other code...
After many tries to play with the asyncio code I don't reach the result is running without blocking...
(My class Bot in main.py have an attribute that called self.loop and is actually asyncio.get_event_loop)
I don't know if I do all correctly because I'm just after calling the Run function, I call all later operations with await.
I've tried to replace the just await with
await self.loop.create_task(foo).
I tried to do
await self.loop.ensure_future(foo) but nothing...
Too I've tried to split the code to two functions (mainloop and check_data).
First of all in the code is the Run function here I start the loop (just creating task and run_forever):
def run(self, startup_function=None) -> None:
""" Run the bot and start the main while. """
self.loop.create_task(self.mainloop(startup_function))
self.loop.run_forever()
Secondly here the mainloop function (all the await functions are blocking...):
async def mainloop(self, startup_function) -> None:
""" The main loop that reads and process the incoming data. """
if startup_function:
await startup_function()
self.is_running = True
while self.is_running:
data = self.sock.recv(self.buffer).decode('utf-8').split('\n')
await self.check_data(data)
And the last one is the check_data (is splitted mainloop [I've replace the long if's with "condition" for readability], here too the await's is blocking):
async def check_data(self, data: str) -> None:
for line in data:
if confition:
message = self.get_message_object_from_str(line)
if condition:
if condition:
await self.commands[message.command](message, message.args)
else:
await self.commands[message.command](message)
elif hasattr(self.event, 'on_message'):
await self.event.on_message(message)
if self.logs:
print(line)
There is no error message.
The code is blocking and I'm trying to change it to not block the code.
The loop for line in data: is blocking you code.