We're using tornadoweb which already runs an event loop single-threaded, and we want to use asyncio.run to run other coroutines, but it show error 'Event loop already running', found this library nest-asyncio that allows event loop to be nested, sorry not expert on threading or event loop, does nest-asyncio runs coroutines on separated thread?or no connection on threading?
If the event loop is already running, you can't call asyncio.run.
You have a few options:
1. Tornado's IOLoop.add_callback:
This will run the coroutine in the background (that means you won't be able to get it's result).
from tornado import ioloop
loop = ioloop.IOLoop.current() # get the current running event loop
loop.add_callback(coro_func) # coro_func is the coroutine you want to run
2. Use asyncio.create_task:
This will also allow you to get the coroutine's result.
Related
Currently working with a server simulator that is initialized via a call to asyncio.run().
After calling await on a number of coroutines, it is designed to run as an infinite loop that serves continuously.
One of the coroutines that inits the server creates a ModbusTcpServer object that I need to access from the top level (the file that initially makes the call to asyncio.run()). How can I access this data if the coroutines are designed to run infinitely and never return? I'd ultimately like to get the data so I can change values in a GUI such as PyQT5. Here is sample of my code:
def start_server():
cmd_args = get_commandline()
run_args = setup_simulator(cmd_args)
asyncio.run(run_server_simulator(run_args), debug=True)
######
async def run_server_simulator(args):
await StartAsyncTcpServer(
context=args.context,
address=("", args.port),
framer=args.framer,
allow_reuse_address=True,
)
The StartAsyncTcpServer() function creates the object that I want to modify and then awaits a coroutine that creates a task that runs forever.
I've tried manually creating an event loop using
asyncio.new_event_loop()
yet this still doesn't provide me access to the data from within the coroutines, only the loop itself. Not only that, but doing this gives me an error that my first coroutine was never awaited.
I've also attempted initializing the first of my coroutines via asyncio.create_task(), and calling await on the task object but awaiting them never provides return values because the coroutines never finish.
I'm trying to understand the pattern for indefinitely running asyncio Tasks
and the difference that a custom loop signal handler makes.
I create workers using loop.create_task() so that they run concurrently.
In my regular workers' code I am polling for data and act accordingly when data is there.
I'm trying to handle the shutdown process gracefully on a signal.
When a signal is delivered - I again create_task() with the shutdown function, so that currently running tasks continue, and shutdown gets executed in next iteration of the event loop.
Now - when a single worker's while loop doesn't actually do any IO or work then it prevents the signal handler from being executed. It never ends and does not give back execution so that other tasks could be run.
When I don't attach a custom signal handler to a loop and run this program, then a signal is delivered and the program stops. I assume it's a main thread that stops the loop itself.
This is obviously different from trying to schedule a (new) shutdown task on a running loop, because that running loop is stuck in a single coroutine which is blocked in a while loop and doesn't give back any control or time for other tasks.
Is there any standard pattern for such cases?
Do I need to asyncio.sleep() if there's no work to do, do I replace the while loop with something else (e.g. rescheduling the work function itself)?
If the range(5) is replaced with range(1, 5) then all workers do await asyncio.sleep,
but if one of them does not, then everything gets blocked. How to handle this case, is there any standard approach?
The code below illustrates the problem.
async def shutdown(loop, sig=None):
print("SIGNAL", sig)
tasks = [t for t in asyncio.all_tasks()
if t is not asyncio.current_task()]
[t.cancel() for t in tasks]
results = await asyncio.gather(*tasks, return_exceptions=True)
# handle_task_results(results)
loop.stop()
async def worker(intval):
print("start", intval)
while True:
if intval:
print("#", intval)
await asyncio.sleep(intval)
loop = asyncio.get_event_loop()
for sig in {signal.SIGINT, signal.SIGTERM}:
loop.add_signal_handler(
sig,
lambda s=sig: asyncio.create_task(shutdown(loop, sig=s)))
workers = [loop.create_task(worker(i)) for i in range(5)] # this range
loop.run_forever()
I have tried the following code in Python 3.6 for asyncio:
Example 1:
import asyncio
import time
async def hello():
print('hello')
await asyncio.sleep(1)
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output is as expected:
hello
hello
hello again
hello again
Then I want to change the asyncio.sleep into another def:
async def sleep():
time.sleep(1)
async def hello():
print('hello')
await sleep()
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output:
hello
hello again
hello
hello again
It seems it is not running in an asynchronous mode, but a normal sync mode.
The question is: Why is it not running in an asynchronous mode and how can I change the old sync module into an 'async' one?
Asyncio uses an event loop, which selects what task (an independent call chain of coroutines) in the queue to activate next. The event loop can make intelligent decisions as to what task is ready to do actual work. This is why the event loop also is responsible for creating connections and watching file descriptors and other I/O primitives; it gives the event loop insight into when there are I/O operations in progress or when results are available to process.
Whenever you use await, there is an opportunity to return control to the loop which can then pass control to another task. Which task then is picked for execution depends on the exact implementation; the asyncio reference implementation offers multiple choices, but there are other implementations, such as the very, very efficient uvloop implementation.
Your sample is still asynchronous. It just so happens that by replacing the await.sleep() with a synchronous time.sleep() call, inside a new coroutine function, you introduced 2 coroutines into the task callchain that don't yield, and thus influenced in what order they are executed. That they are executed in what appears to be synchronous order is a coincidence. If you switched event loops, or introduced more coroutines (especially some that use I/O), the order can easily be different again.
Moreover, your new coroutines use time.sleep(); this makes your coroutines uncooperative. The event loop is not notified that your code is waiting (time.sleep() will not yield!), so no other coroutine can be executed while time.sleep() is running. time.sleep() simply doesn't return or lets any other code run until the requested amount of time has passed. Contrast this with the asyncio.sleep() implementation, which simply yields to the event loop with a call_later() hook; the event loop now knows that that task won't need any attention until a later time.
Also see asyncio: why isn't it non-blocking by default for a more in-depth discussion of how tasks and the event loop interact. And if you must run blocking, synchronous code that can't be made to cooperate, then use an executor pool to have the blocking code executed in a separate tread or child process to free up the event loop for other, better behaved tasks.
As I understand , a Node JS server continues to listen on a port for any incoming requests which means the thread is continuously busy ? When does it break from this continuous never ending loop and check if there are any events to be processed from the call back queue?
2) When Node JS starts executing code of callback functions, the server is essentially stopped? It is not listening for further requests? I mean since only single thread is going to do both the task only one can be done at a time?
Is this understanding correct or there's more to it?
Yes, you are right. Everything in nodejs runs on the main thread except the I/o calls and fs file calls which go into to OS kernel and thread pool respectively for execution. All the synchronous code runs on the main thread and while this code is running, nodejs cannot process any further requests. That is why it is not advisable to use a long for loop because it may block the main thread for much time. You need to make child threads to handle that.
Node thread keeps an event loop and whenever any task get completed, it fires the corresponding event which signals the event listener function to get executed. The event loop simply iterate over the event queue which is basically a list of events and callbacks of completed operations. there is generally a main loop that listens for events, and then triggers a callback function when one of those events is detected.
(source: abdelraoof.com)
Similar event loop questions are here:
Node.js Event loop
Understanding the Event Loop
Source:
http://abdelraoof.com/blog/2015/10/28/understanding-nodejs-event-loop/
http://www.tutorialspoint.com/nodejs/nodejs_event_loop.htm
http://chimera.labs.oreilly.com/books/1234000001808/ch03.html#chap3_id35941348
Hi need some help on my lua script. I have a script here that will run a server like application (infinite loop). Problem here is it doesn't execute the second coroutine.
Could you tell me whats wrong Thank you.
function startServer()
print( "...Running server" )
--run a server like application infinite loop
os.execute( "server.exe" )
end
function continue()
print("continue")
end
co = coroutine.create( startServer() )
co1 = coroutine.create( continue() )
Lua have cooperative multithreading. Threads are not swtiched automatically, but must yield to others. When one thread is running, every other thread is waiting for it to finish or yield. Your first thread in this example seems to run server.exe, which, I assume, never finishes until interrupted. Thus second thread never gets its turn to run.
You also run threads wrong. In your example you're not running any threads at all. You execute function and then would try to create coroutine with its output, which naturally would fail. But since you never get back from server.exe you didn't notice this problem yet. Remove those brackets after startServer and continue to fix it.
As already noted, there are several issues with the script that prevent you from getting what you want:
os.execute("...") is blocked until the command is completed and in your case it doesn't complete (as it runs an infinite loop). Solution: you need to detach that process from yours by using something like io.popen() instead of os.execute()
co = coroutine.create( startServer() ) doesn't create a coroutine in your case. coroutine.create call accepts a function reference and you pass it the result of startServer call, which is nil. Solution: use co = coroutine.create( startServer ) (note that parenthesis are dropped, so it's not a function call anymore).
You are not yielding from your coroutines; if you want several coroutines to work together, they need to be cooperating by giving control to each other when appropriate. That's what yield command is for and that's why it's called non-preemptive multithreading. Solution: you need to use a combination of resume and yield calls after you create your coroutine.
startServer doesn't need to be a coroutine as you are not giving control back to it; its only purpose is to start the server.
In your case, the solution may not even need coroutines as all you need to do is: (1) start the server and let it detach from your process (for example, using popen) and (2) work with your process using whatever communication protocol it requires (pipes, sockets, etc.).
There are more complex and complete solutions (like LuaLanes) and also several good descriptions on creating simple coroutine dispatchers.
Your coroutine is not yielding