I have been trying to run tornado with aiomysql in multi-process mode with the following code
#asyncio.coroutine
def get_mysql_connection(loop):
return (yield from aiomysql.create_pool(host=host,port=3306,user=user, password=pass, db=db, loop=loop))
if __name__ == "__main__":
tornado.platform.asyncio.AsyncIOMainLoop().install()
ioloop = asyncio.get_event_loop()
mysql = ioloop.run_until_complete(get_mysql_connection(ioloop))
options.parse_config_file("app.conf")
app = make_app(mysql)
print('listening on %s:%s...' %(options.host, options.port))
server = tornado.httpserver.HTTPServer(app)
server.listen(options.port)
server.start(0) #this is my problem
ioloop.run_forever()
But I keep getting the below error
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()
Everything is working fine except for the ioloop.start(0) line, is it possible to make the two libraries aiomysql and tornado work nicely in multi-process mode? If no what are my other options
Tornado version 4.4.2
python version 3.6.0
aiomysql version 0.0.9
Like the message says, you can only fork worker processes if you do it before the IOLoop is initialized. This gets a little delicate with AsyncIOMainLoop since you want to install that as early as possible. The solution is to start your program with this sequence:
tornado.options.parse_config_file(...)
socks = tornado.netutil.bind_sockets(options.port, options.host)
tornado.process.fork_processes(0)
tornado.asyncio.AsyncIOMainLoop().install()
# Initialize the rest of your app, create the HTTPServer,
# and instead of listen() or start(), do
server.add_sockets(socks)
what are my other options
In my opinion it's better to use an external process manager like supervisord instead of forking multiple processes within your app. That avoids most of these initialization-order traps.
Related
I am trying to use my telegram bot with Django. I want the code to keep running in the background. I am Using the apps.py to do this but there's one problem when the bot starts as it's an infinite loop, the Django server is never started.
Apps.py:
from django.apps import AppConfig
import os
class BotConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'bot'
def ready(self):
from . import jobs
if os.environ.get('RUN_MAIN', None) != 'true':
jobs.StartBot()
Jobs.py:
def StartBot():
updater = Updater("API KEY")
dp = updater.dispatcher
dp.add_handler(ChatMemberHandler(GetStatus, ChatMemberHandler.CHAT_MEMBER))
updater.start_polling(allowed_updates=Update.ALL_TYPES)
updater.idle()
What's the best way to run my bot in the background? while making sure that the Django server runs normally. I tried Django background tasks but it's not compatible with Django 4.0.
The purpose of Updater.idle is to keep the main thread alive because start_polling only starts some background threads that don't block the main thread. If you want to run other stuff in the main thread, skip updater.idle() and instead call Updater.stop manually when the program should shut down.
Disclaimer: I'm currently the maintainer of python-telegram-bot
In the python asyncio websockets library, the example calls run_forever(). Why is this required?
Shouldn't run_until_complete() block and run the websockets loop?
#!/usr/bin/env python
# WS server example
import asyncio
import websockets
async def hello(websocket, path):
name = await websocket.recv()
print(f"< {name}")
greeting = f"Hello {name}!"
await websocket.send(greeting)
print(f"> {greeting}")
start_server = websockets.serve(hello, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
# if you comment out the above line, this doesn't work, i.e., the server
# doesn't actually block waiting for data...
If I comment out run_forever(), the program ends immediately.
start_server is an awaitable returned by the library. Why isn't run_until_complete sufficient to cause it to block/await on hello()?
websockets.serve simply starts the server and exits immediately. (It still needs to be awaited because configuring the server can require network communication.) Because of that, you need to actually run the event loop.
Since the server is designed to run indefinitely, you cannot run the event loop in the usual way, by passing a coroutine to run_until_complete. As the server has already started, there is no coroutine to run, you just need to let the event loop run and do its job. This is where run_forever comes in useful - it tells the event loop to run (executing the tasks previously scheduled, such as those belonging to the server) indefinitely, or until told to stop by a call to loop.stop.
In Python 3.7 and later one should use asyncio.run to run asyncio code, which will create a new event loop, so the above trick won't work. A good way to accomplish the above in modern asyncio code would be to use the serve_forever method (untested):
async def my_server():
ws_server = await websockets.serve(hello, "localhost", 8765)
await ws_server.server.serve_forever()
asyncio.run(my_server())
I have numerous Bokeh Server files in a directory say.. /dir/bokeh/, assume the bokeh servers are called bokeh1.py, bokeh2.py, bokeh3.py
The file structure is like so:
|--dir
|---flask.py
|---bokeh
|--bokeh1.py
|--bokeh2.py
I am deploying them all on flask like so:
files=[]
for file in os.listdir("/dir/bokeh/"):
if file.endswith('.py'):
file="bokeh/"+file
files.append(file)
argvs = {}
urls = []
for i in files:
argvs[i] = None
urls.append(i.split('\\')[-1].split('.')[0])
host = 'myhost.com'
apps = build_single_handler_applications(files, argvs)
bokeh_tornado = BokehTornado(apps, extra_websocket_origins=["myhost.com"])
bokeh_http = HTTPServer(bokeh_tornado)
sockets, port = bind_sockets("myhost.com", 0)
bokeh_http.add_sockets(sockets)
On update to Tornado 6.0.2, and deploying Flask, I get the Runtimerror There is no current event loop in thread Thread-1. On deeper research Tornado uses asyncio by default and imposes some restrictions. So I add asyncio.set_event_loop(asyncio.new_event_loop()) to the following.
def bk_worker():
asyncio.set_event_loop(asyncio.new_event_loop())####
server = BaseServer(IOLoop.current(), bokeh_tornado, bokeh_http)
server.start()
server.io_loop.start()
gc.collect()
from threading import Thread
Thread(target=bk_worker).start()
However, upon opening the bokeh server url through flask, the bokeh server selected (any of them) do not load and simply return a blank page. How can I circumvent this?
setting asyncio.set_event_loop_policy(AnyThreadEventLoopPolicy)) yields the same result.
edit:The previous code works with python 2/3, Tornado 4.5.3
I think this is a known Bokeh issue. The best way for now is to downgrade to Tornado 4.5.3.
pip install tornado==4.5.3
I'm trying to request API in multi threading by cron job. I look like to stop or to laze the thread requesting.
It's no problem by pserve. I want to use uWSGI but I have some problem.
Python 3.5.2
My code is like this:
import threading
import requests
def worker(settings):
lookup_url = settings['lookup_url']
api_sid = settings['api_sid']
auth_token = settings['auth_token']
args = settings['args']
resp = requests.post(lookup_url,
auth=(api_sid, auth_token),
data={'data': args},
timeout=5.0,
)
def main(request):
registry = request.registry
settings = registry.settings
for _ in range(3):
threading.Thread(target=worker, args=(settings,)).start()
Please let me know any solutions.
UWSGI does not enable Python threads by default:
By default the Python plugin does not initialize the GIL. This means
your app-generated threads will not run. If you need threads, remember
to enable them with enable-threads. Running uWSGI in multithreading
mode (with the threads options) will automatically enable threading
support. This “strange” default behaviour is for performance reasons,
no shame in that.
http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
This is yet another flask dev server reloader question. There are a million questions asking why it loads everything twice, and this is not one of them. I understand that it loads everything twice, my question involves dealing with this reality and I haven't found an answer that I think addresses what I'm trying to do.
My question is, how can I cleanup all app objects at exit?
My current approach is shown below. In this example I run my cleanup code using an atexit function.
from flask import Flask
app = Flask(__name__)
print("start_app_id: ", '{}'.format(id(app)))
import atexit
#atexit.register
def shutdown():
print("AtExit_app_id: ", '{}'.format(id(app)))
#do some cleanup on the app object here
if __name__ == "__main__":
import os
if os.environ.get('WERKZEUG_RUN_MAIN') == "true":
print("reloaded_main_app_id: ", '{}'.format(id(app)))
else:
print("first_main_app_id: ", '{}'.format(id(app)))
app.run(host='0.0.0.0', debug=True)
The output of this code is as follows:
start_app_id: 140521561348864
first_main_app_id: 140521561348864
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
start_app_id: 140105598483312
reloaded_main_app_id: 140105598483312
* Debugger is active!
* Debugger pin code: xxx-xxx-xxx
^CAtExit_app_id: 140521561348864
Note that when first loaded, an app object with ID '864 is created. During the automatic reloading, a new app object with ID '312 is created. Then when I hit Ctrl-C (last line), the atexit routine is called and the original '864 app object is the one that is accessible using the app variable -- not the newer '312 app object.
I want to be able to do cleanup on all app objects floating around when the server closes or is Ctrl-C'd (in this case both '864 and '312). Any recs on how to do this?
Or alternately, if I could just run the cleanup on the newer '312 object created after reloading I could also make that work -- however my current approach only lets me cleanup the original app object.
Thanks.
UPDATE1: I found a link that suggested using try/finally instead of the atexit hook to accomplish what I set out to do above. Switching to this results in exactly the same behavior as atexit and therefore doesn't help with my issue:
from flask import Flask
app = Flask(__name__)
print("start_app_id: ", '{}'.format(id(app)))
if __name__ == "__main__":
import os
if os.environ.get('WERKZEUG_RUN_MAIN') == "true":
print("reloaded_main_app_id: ", '{}'.format(id(app)))
else:
print("first_main_app_id: ", '{}'.format(id(app)))
try:
app.run(host='0.0.0.0', debug=True)
finally:
print("Finally_app_id: ", '{}'.format(id(app)))
#do app cleanup code here
After some digging through the werkzeug source I found the answer. The answer is that it isn't possible to do what I wanted -- and this is by design.
When using the flask dev server (werkzeug) it isn't possible to cleanup all existing app objects upon termination (e.g. ctrl-C) because the werkzeug server catches the keyboardinterrupt exception and "passes" on it. You can see this in the last lines of werkzeug's _reloader.py in the run_with_reloader function:
def run_with_reloader(main_func, extra_files=None, interval=1,
reloader_type='auto'):
"""Run the given function in an independent python interpreter."""
import signal
reloader = reloader_loops[reloader_type](extra_files, interval)
signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
try:
if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
t = threading.Thread(target=main_func, args=())
t.setDaemon(True)
t.start()
reloader.run()
else:
sys.exit(reloader.restart_with_reloader())
except KeyboardInterrupt:
pass
If you replace the above "except KeyboardInterrupt:" with "finally:", and then run the second code snippet in the original question, you observe that both of the created app objects are cleaned up as desired. Interestingly, the first code snippet (that uses #atexit) still doesn't work as desired after making these changes.
So in conclusion, you can cleanup all existing app objects when using the flask dev server, but you need to modify the werkzeug source to do so.