It's clear for my how I use asyncio in python3.
import asyncio
import time
async def do_something(number):
print(f"do something no.{number} at timestamp : {time.perf_counter():6.2f}")
await asyncio.sleep(1)
print(f"do something no.{number} ended at timestamp: {time.perf_counter():6.2f}")
async def main():
await asyncio.gather(
do_something(1),
do_something(2)
)
asyncio.run(main() )
However, I have no idea how I could create an own "await"-able object like asyncio.sleep. In this "await"-able I could encapsulate urllib.request, isn't it?
Can someone post an example?
Many thanks.
Please, take a look at this answer it uses old yield for-based syntax, but the idea stays the same: using asyncio.Future and loop.call_later() you can cast callback-based code into coroutine-based code:
import asyncio
async def my_sleep(delay):
fut = asyncio.Future()
loop = asyncio.get_event_loop()
loop.call_later(
delay,
lambda *_: fut.set_result(True)
)
return await fut
async def main():
await asyncio.gather(
my_sleep(1),
my_sleep(2),
my_sleep(3)
)
print('ok')
asyncio.run(main())
I believe, urllib.request is blocking and doesn't provide callbacks so it can't be cast into coroutine-based form directly. A common way to handle the situation is to run it in async thread (see links in this answer).
But if you want just to make async http reqeust, forget all above and use aiohttp: it's created for the purpose.
You can await all coroutines, so any function with the prefix async is awaitable
For example:
import asyncio
async def some_function():
await asyncio.sleep(1)
print("Hello")
async def main():
await some_function()
asyncio.run(main())
You can find some more information about coroutines at the python docs:
https://docs.python.org/3/library/asyncio-task.html
Because urllib is blocking (does not allow other code to be running before it finishes) by default, it is not that easy to just "wrap" it in an awaitable function.
It is probably possible to offload it to something like another thread and then have an awaiable wait for that thread to finish, but it is probably easier to use an async web request library like aiohttp.
Related
Let's say there's some API that's running in production already and you created another API which you kinda want to A/B test using the incoming requests that's hitting the production-api. Now I was wondering, is it possible to do something like this, (I am aware of people doing traffic splits by keeping two different API versions for A/B testing etc)
As soon as you get the incoming request for your production-api, you make an async request to your new API and then carry on with the rest of the code for the production-api and then, just before returning the final response to the caller back, you check whether you have the results computed for that async task that you had created before. If it's available, then you return that instead of the current API.
I am wondering, what's the best way to do something like this? Do we try to write a decorator for this or something else? i am a bit worried about lot of edge cases that can happen if we use async here. Anyone has any pointers on making the code or the whole approach better?
Thanks for your time!
Some pseudo-code for the approach above,
import asyncio
def call_old_api():
pass
async def call_new_api():
pass
async def main():
task = asyncio.Task(call_new_api())
oldResp = call_old_api()
resp = await task
if task.done():
return resp
else:
task.cancel() # maybe
return oldResp
asyncio.run(main())
You can't just execute call_old_api() inside asyncio's coroutine. There's detailed explanation why here. Please, ensure you understand it, because depending on how your server works you may not be able to do what you want (to run async API on a sync server preserving the point of writing an async code, for example).
In case you understand what you're doing, and you have an async server, you can call the old sync API in thread and use a task to run the new API:
task = asyncio.Task(call_new_api())
oldResp = await in_thread(call_old_api())
if task.done():
return task.result() # here you should keep in mind that task.result() may raise exception if the new api request failed, but that's probably ok for you
else:
task.cancel() # yes, but you should take care of the cancelling, see - https://stackoverflow.com/a/43810272/1113207
return oldResp
I think you can go even further and instead of always waiting for the old API to be completed, you can run both APIs concurrently and return the first that's done (in case new api works faster than the old one). With all checks and suggestions above, it should look something like this:
import asyncio
import random
import time
from contextlib import suppress
def call_old_api():
time.sleep(random.randint(0, 2))
return "OLD"
async def call_new_api():
await asyncio.sleep(random.randint(0, 2))
return "NEW"
async def in_thread(func):
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, func)
async def ensure_cancelled(task):
task.cancel()
with suppress(asyncio.CancelledError):
await task
async def main():
old_api_task = asyncio.Task(in_thread(call_old_api))
new_api_task = asyncio.Task(call_new_api())
done, pending = await asyncio.wait(
[old_api_task, new_api_task], return_when=asyncio.FIRST_COMPLETED
)
if pending:
for task in pending:
await ensure_cancelled(task)
finished_task = done.pop()
res = finished_task.result()
print(res)
asyncio.run(main())
I would like to attach some data to a currently running loop. This can be very useful in sharing secret keys, or reusing sessions. For example, something like
import asyncio
import aiohttp
async def func1():
loop=asyncio.get_running_loop()
loop["client_session"] = aiohttp.ClientSession()
async def func2():
loop=asyncio.get_running_loop()
return await loop["client_session"].get("https://google.com")
async def main():
await func1()
return await func2()
if __name__ == "__main__":
asyncio.run(main())
My question is, would the above code work, is it reliable, or is there a better way?
I know that data can be attached to an aiohttp.Application, and so I'm wondering if the same can be done for general asyncio loops.
I'd like to do a non-blocking http request in Python 3.7. What I'm trying to do is described well in this SO post, but it doesn't yet have an accepted answer.
Here's my code so far:
import asyncio
from aiohttp import ClientSession
[.....]
async def call_endpoint_async(endpoint, data):
async with ClientSession() as session, session.post(url=endpoint, data=data) as result:
response = await result.read()
print(response)
return response
class CreateTestScores(APIView):
permission_classes = (IsAuthenticated,)
def post(self, request):
[.....]
asyncio.run(call_endpoint_async(url, data))
print('cp #1') # <== `async.io` BLOCKS -- PRINT STATEMENT DOESN'T RUN UNTIL `asyncio.run` RETURNS
What is the correct way to do an Ajax-style non-blocking http request in Python?
Asyncio makes it easy to make a non-blocking request if your program runs in asyncio. For example:
async def doit():
task = asyncio.create_task(call_endpoint_async(url, data))
print('cp #1')
await asyncio.sleep(1)
print('is it done?', task.done())
await task
print('now it is done')
But this requires that the "caller" be async as well. In your case you want the whole asyncio event loop to run in the background, so that. This can be achieved by running it in a separate thread, e.g.:
pool = concurrent.futures.ThreadPoolExecutor()
# ...
def post(self, request):
fut = pool.submit(asyncio.run, call_endpoint_async(url, data))
print('cp #1')
However, in that case you're not getting anything by using asyncio. Since you're using threads anyway, you could as well call a sync function such as requests.get() to begin with.
In my code I have a class with properties, that occasionally need to run asynchronous code. Sometimes I need to access the property from asynchronous function, sometimes from synchronous - that's why I don't want my properties to be asynchronous. Besides, I have an impression that asynchronous properties in general is a code smell. Correct me if I'm wrong.
I have a problem with executing the asynchronous method from the synchronous property and blocking the further execution until the asynchronous method will finish.
Here is a sample code:
import asyncio
async def main():
print('entering main')
synchronous_property()
print('exiting main')
def synchronous_property():
print('entering synchronous_property')
loop = asyncio.get_event_loop()
try:
# this will raise an exception, so I catch it and ignore
loop.run_until_complete(asynchronous())
except RuntimeError:
pass
print('exiting synchronous_property')
async def asynchronous():
print('entering asynchronous')
print('exiting asynchronous')
asyncio.run(main())
Its output:
entering main
entering synchronous_property
exiting synchronous_property
exiting main
entering asynchronous
exiting asynchronous
First, the RuntimeError capturing seems wrong, but if I won't do that, I'll get RuntimeError: This event loop is already running exception.
Second, the asynchronous() function is executed last, after the synchronous one finish. I want to do some processing on the data set by asynchronous method so I need to wait for it to finish.
If I'll add await asyncio.sleep(0) after calling synchronous_property(), it will call asynchronous() before main() finish, but it doesn't help me. I need to run asynchronous() before synchronous_property() finish.
What am I missing? I'm running python 3.7.
Asyncio is really insistent on not allowing nested loops, by design. However, you can always run another event loop in a different thread. Here is a variant that uses a thread pool to avoid having to create a new thread each time around:
import asyncio, concurrent.futures
async def main():
print('entering main')
synchronous_property()
print('exiting main')
pool = concurrent.futures.ThreadPoolExecutor()
def synchronous_property():
print('entering synchronous_property')
result = pool.submit(asyncio.run, asynchronous()).result()
print('exiting synchronous_property', result)
async def asynchronous():
print('entering asynchronous')
await asyncio.sleep(1)
print('exiting asynchronous')
return 42
asyncio.run(main())
This code creates a new event loop on each sync->async boundary, so don't expect high performance if you're doing that a lot. It could be improved by creating only one event loop per thread using asyncio.new_event_loop, and caching it in a thread-local variable.
The easiest way is using an existing "wheel",
like
asgiref.async_to_sync
from asgiref.sync import async_to_sync
then:
async_to_sync(main)()
in general:
async_to_sync(<your_async_func>)(<.. arguments for async function ..>)
This is a caller class which turns an awaitable that only works on the thread with
the event loop into a synchronous callable that works in a subthread.
If the call stack contains an async loop, the code runs there.
Otherwise, the code runs in a new loop in a new thread.
Either way, this thread then pauses and waits to run any thread_sensitive
code called from further down the call stack using SyncToAsync, before
finally exiting once the async task returns.
There appears to a problem with the question as stated. Restating the question:
How to communicate between a thread (containing no async processes and hence considered sync) and an async proces (running in some event loop). One approach is to use two sync Queues. The sync process puts its request/parameters into the QtoAsync, and waits on the QtoSync. The async process reads the QtoAsync WITHOUT wait, and if it finds a request/parameters, executes the request, and places the result in QtoSync.
import queue
QtoAsync = queue.Queue()
QtoSync = queue.Queue()
...
async def asyncProc():
while True:
try:
data=QtoAsync.get_nowait()
result = await <the async that you wish to execute>
QtoAsync.put(result) #This can block if queue is full. you can use put_nowait and handle the exception.
except queue.Empty:
await asyncio.sleep(0.001) #put a nominal delay forcing this to wait in event loop
....
#start the sync process in a different thread here..
asyncio.run(main()) #main invokes the async tasks including the asyncProc
The sync thread puts it request to async using:
req = <the async that you wish to execute>
QtoAsync.put(req)
result = QtoSync.get()
This should work.
Problem with the question as stated:
1. When the async processes are started with asyncio.run (or similar) execution blocks until the async processes are completed. A separate sync thread has to be started explicity before calling asyncio.run
2. In general asyncio processes depend on other asyncio processes in that loop. So calling a async process from another thread is not permitted directly. The interaction should be with the event loop, and using two queues is one approach.
I want to make the async call to execute from sync and block it's execution
Just make the sync func async and await the asynchronous function. Async functions are just like normal functions and you can put whatever code you want in them. If you still have a problem modify your question using actual code you are trying to run.
import asyncio
async def main():
print('entering main')
await synchronous_property()
print('exiting main')
async def synchronous_property():
print('entering synchronous_property')
await asynchronous()
# Do whatever sync stuff you want who cares
print('exiting synchronous_property')
async def asynchronous():
print('entering asynchronous')
print('exiting asynchronous')
asyncio.run(main())
I'm using asyncio with requests to try to make a core module asynchronous program. I've ran into a difficulty when trying to do something like this
import asyncio
import requests
async def main():
await r = requests.get(URL)
What I thought this would do is wait for the get request to finish, then take the return value and put it in r, but this error happens
File "prog.py", line 20
await r = requests.get(URL)
^
SyntaxError: can't assign to await expression
r = await requests.get(URL) doesn't seem to work either, giving
prog.py:31: RuntimeWarning: coroutine 'coroutine' was never awaited
coroutine(args)
Does anyone know how to do this?
How to use await?
await can be used only to await coroutine - special object returned by calling function defined with async def:
import asyncio
async def test():
return True
async def main():
# test() returns coroutine:
coro = test()
print(coro) # <coroutine object test at ...>
# we can await for coroutine to get result:
res = await coro
print(res) # True
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Read also this answer about using asyncio.
Why await requests.get(URL) doesn't work?
Because requests.get is not a coroutine (it's not defined with async def), it's regular function by nature.
If you want to make request asynchronously you should either use special async module like aiohttp for this or wrap requests into coroutine using threads. See code snippets here for both examples.