After perusing many docs on AsyncIO and articles I still could not find an answer to this : Run a function asynchronously (without using a thread) and also ensure the function calling this async function continues its execution.
Pseudo - code :
async def functionAsync(p):
#...
#perform intensive calculations
#...
print ("Async loop done")
def functionNormal():
p = ""
functionAsync(p)
return ("Main loop ended")
print ("Start Code")
print functionNormal()
Expected Output :
Start code
Main loop ended
Async loop done
Searched examples where loop.run_until_complete is used, but that will not return the print value of functionNormal() as it is blocking in nature.
asyncio can't run arbitrary code "in background" without using threads. As user4815162342 noted is asyncio you run event loop that blocks main thread and manages execution of coroutines.
If you want to use asyncio and take advantage of using it, you should rewrite all your functions that uses coroutines to be coroutines either up to main function - entry point of your program. This main coroutine is usually passed to run_until_complete. This little post reveals this topic in more detail.
Since you're interested in Flask, take a look Quart: it's a web framework that tries to implement Flask API (as much as it's possible) in terms of asyncio. Reason this project exists is because pure Flask isn't compatible with asyncio. Quart is written to be compatible.
If you want to stay with pure Flask, but have asynchronous stuff, take a look at gevent. Through monkey-patching it can make your code asynchronous. Although this solution has its own problems (which why asyncio was created).
Maybe it's a bit late, but I'm running into a similar situation and I solved it in Flask by using run_in_executor:
def work(p):
# intensive work being done in the background
def endpoint():
p = ""
loop = asyncio.get_event_loop()
loop.run_in_executor(None, work, p)
I'm not sure however how safe this is since the loop is not being closed.
Here is an implementation of a helper function which you can use like this:
result = synchronize_async_helper(some_async_function(parmater1,parameter2))
import asyncio
def synchronize_async_helper(to_await):
async_response = []
async def run_and_capture_result():
r = await to_await
async_response.append(r)
loop = asyncio.get_event_loop()
coroutine = run_and_capture_result()
loop.run_until_complete(coroutine)
return async_response[0]
Assuming the synchronous function is inside an asynchronous function, you can solve it using exceptions.
Pseudo code:
class CustomError(Exception):
pass
async def main():
def test_syn():
time.sleep(2)
# Start Async
raise CustomError
try:
test_syn()
except CustomError:
await asyncio.sleep(2)
Related
This is probably the second time I've asked for help here so my apologies for little or extra detail / wording in my question *
The following code, though very basic is something I've written as an imitation (in its simplest form) of another piece of code written by an ex-employee at my firm. I am currently working on a project she was working on, and I do not understand how the following is executing without it being awaited or gathered.
In the original code, the 'wait_and_print' function is of course is an async function that that does a single RESTful web API call using aiohttp.ClientSession (using an async context manager) which returns nothing yet appends/extends a list with the response it gets.
So far, it has been 2 weeks since I've been using (and trying to understand) asyncio or asynchronous programming thus I am not very savvy with it. I've used Python, however for 3 years - on and off. I understand what task creation does and how asyncio.gather can run multiple API calls concurrently. BUT, this is something I do not get;
import asyncio
import time
L = []
async def wait_and_print(wait_time):
print(f"starting function {wait_time}")
for x in range(1, wait_time + 1):
print("Sleeping for {} time.".format(x))
await asyncio.sleep(1)
print(f"ending function {wait_time}")
L.append(wait_time)
async def main_loop():
tasks = [ asyncio.create_task(wait_and_print(x)) for x in [3,1,2]]
while len(tasks) != 0:
tasks = [t for t in tasks if not t.done()]
await asyncio.sleep(0) # HOW IS THIS MAKING IT WORK WITHOUT ACTUALLY AWAITING tasks?
print("Main loop ended!")
def final(func):
a = time.time()
asyncio.run(func())
b = time.time()
print(b-a, "seconds taken to run all!")
print(L)
final(main_loop)
I'm trying to wrap an async function up so that I can use it without importing asyncio in certain files. The ultimate goal is to use asynchronous functions but being able to call them normally and get back the result.
How can I access the result from the callback function printing(task) and use it as the return of my make_task(x) function?
MWE:
#!/usr/bin/env python3.7
import asyncio
loop = asyncio.get_event_loop()
def make_task(x): # Can be used without asyncio
task = loop.create_task(my_async(x))
task.add_done_callback(printing)
# return to get the
def printing(task):
print('sleep done: %s' % task.done())
print('results: %s' % task.result())
return task.result() # How can i access this return?
async def my_async(x): # Handeling the actual async running
print('Starting my async')
res = await my_sleep(x)
return res # The value I want to ultimately use in the real callback
async def my_sleep(x):
print('starting sleep for %d' % x)
await asyncio.sleep(x)
return x**2
async def my_coro(*coro):
return await asyncio.gather(*coro)
val1 = make_task(4)
val2 = make_task(5)
loop.run_until_complete(my_coro(asyncio.sleep(6)))
print(val1)
print(val2)
If I understand correctly you want to use asynchronous functions but don't want to write async/await in top-level code.
If that's the case, I'm afraid it's not possible to achieve with asyncio. asyncio wants you to write async/await everywhere asynchronous stuff happens and this is intentional: forcing to explicitly mark places of possible context switch is a asyncio's way to fight concurrency-related problems (which is very hard to fight otherwise). Read this answer for more info.
If you still want to have asynchronous stuff and use it "as usual code" take a look at alternative solutions like gevent.
Instead of using a callback, you can make printing a coroutine and await the original coroutine, such as my_async. make_task can then create a task out of printing(my_async(...)), which will make the return value of printing available as the task result. In other words, to return a value out of printing, just - return it.
For example, if you define make_task and printing like this and leave the rest of the program unchanged:
def make_task(x):
task = loop.create_task(printing(my_async(x)))
return task
async def printing(coro):
coro_result = await coro
print('sleep done')
print('results: %s' % coro_result)
return coro_result
The resulting output is:
Starting my async
starting sleep for 4
Starting my async
starting sleep for 5
sleep done
results: 16
sleep done
results: 25
<Task finished coro=<printing() done, defined at result1.py:11> result=16>
<Task finished coro=<printing() done, defined at result1.py:11> result=25>
When using await/async, does it have to go "all the way up", meaning, does every function in the call chain have to use it?
E.g.:
def a():
# can't call b() here
async def b():
return await c
async def c():
return ...
I recently wondered this in the context of a flask app running under gevent, where one of the endpoints was a long running call that should be "checked upon", while not blocking other calls
def handler0():
# short running
return ...
def handler(): # blocks handler0
return await some_long_thing()
async def some_long_thinig():
# ..do somethiing
return ...
does every function in the call chain have to use it?
When you use asyncio module every function that await for something should be defined as async (should be a coroutine itself).
Most top level coroutine usually is main entry point of your script and executed by event loop using asyncio.run() or similar function.
This is how asyncio designed: this way you always know if context can be or can't be switched in particular place.
I'm practicing asyncio after writing multithreaded code many years.
Noticed something which i find it strange. Both in asyncio and in concurrent there is a Future object.
from asyncio import Future
from concurrent.futures import Future
Guess each onee has it's own role..
My question is if i can transfer concurrent.future.Future to asyncio.Future (or the opposite)?
My question is if i can transfer concurrent.future.Future to asyncio.Future (or the opposite)?
If by "transfer" you mean convert one to the other, yes, it's possible, although bridging the impedance mismatch can take some work.
To convert a concurrent.futures.Future into an asyncio.Future, you can call asyncio.wrap_future. The returned asyncio future is awaitable in the asyncio event loop and will complete when the underlying threading future completes. This is effectively how run_in_executor is implemented.
There is no public functionality to directly convert an asyncio future to concurrent.futures future, but there is the asyncio.run_coroutine_threadsafe function, which takes a coroutine, submits it to an event loop, and returns a concurrent future which completes when the asyncio future does. This can be used to effectively convert any asyncio-awaitable future to concurrent future, like this:
def to_concurrent(fut, loop):
async def wait():
await fut
return asyncio.run_coroutine_threadsafe(wait(), loop)
The returned future will behave like you'd expect from a concurrent future, e.g. its result() method will block, etc. One thing you might want to be careful of is that callbacks added to the concurrent future with add_done_callback run in the thread that marked the future completed, which in this case is the event loop thread. This means that if you add some done callbacks, you need to be careful not to invoke blocking calls in their implementation lest you block the event loop.
Note that calling run_coroutine_threadsafe requires the event loop to actually run in some other thread. (For example, you can start a background thread and have it execute loop.run_forever.)
For the "concurrent future to asyncio future" part, here is an utility I use.
from typing import List, Any
from concurrent.futures.thread import ThreadPoolExecutor
import asyncio
class AsyncThreadPool(ThreadPoolExecutor):
_futures: List[asyncio.Future]
_loop: asyncio.AbstractEventLoop
def __init__(self, max_workers=None):
super().__init__(max_workers)
self._futures = []
def queue(self, fn):
self._loop = asyncio.get_event_loop()
fut = self._loop.create_future()
self._futures.append(fut)
self.submit(self._entry, fn, fut)
def queueAsync(self, coroutine):
def newLoop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
return loop.run_until_complete(coroutine)
self.queue(newLoop)
def _entry(self, fn, fut: asyncio.Future):
try:
result = fn()
self._loop.call_soon_threadsafe(fut.set_result, result)
except Exception as e:
self._loop.call_soon_threadsafe(fut.set_exception, e)
async def gather(self) -> List[Any]:
return await asyncio.gather(*self._futures)
You can use it like that:
with AsyncThreadPool() as pool:
# Queue some sync function (will be executed on another thread)
pool.queue(someHeavySyncFunction)
# Queue a coroutine that will be executed on a new event loop running on another thread
pool.queue(otherAsyncFunction())
# Gather results (non blocking for your current loop)
res: List[Any] = await pool.gather()
There is a function called wrap_future in asyncio.
Wrap a concurrent.futures.Future object in a asyncio.Future object.
See https://docs.python.org/3/library/asyncio-future.html#asyncio.wrap_future
I am trying to run a asyncio sub-process in a pyramid view but the view hangs and the async task appears to never complete. I can run this example outside of a pyramid view and it works.
With that said I have tested originally using loop = asyncio.get_event_loop() but this tells me RuntimeError: There is no current event loop in thread 'Dummy-2'
There are certainly things I don't fully understand here. Like maybe the view thread is different to the main thread so get_event_loop doesn't work.
So does anybody know why my async task might not yield its result in this scenario? This is a naive example.
#asyncio.coroutine
def async_task(dir):
# This task can be of varying length for each handled directory
print("Async task start")
create = asyncio.create_subprocess_exec(
'ls',
'-l',
dir,
stdout=asyncio.subprocess.PIPE)
proc = yield from create
# Wait for the subprocess exit
data = yield from proc.stdout.read()
exitcode = yield from proc.wait()
return (exitcode, data)
#view_config(
route_name='test_async',
request_method='GET',
renderer='json'
)
def test_async(request):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
dirs = ['/tmp/1/', '/tmp/2/', '/tmp/3/']
tasks = []
for dir in dirs:
tasks.append(asyncio.ensure_future(async_task(dir), loop=loop))
loop.run_until_complete(asyncio.gather(*tasks))
loop.close()
return
You are invoking loop.run_until_complete in your view so clearly it is going to block until complete!
If you want to use asyncio with a WSGI app then you need to do so in another thread. For example you could spin up a thread that contains the eventloop and executes your async code. WSGI code is all synchronous and so any async code must be done this way, with it's own issues, or you can just live with it blocking the request thread like you're doing now.