I'm trying to request API in multi threading by cron job. I look like to stop or to laze the thread requesting.
It's no problem by pserve. I want to use uWSGI but I have some problem.
Python 3.5.2
My code is like this:
import threading
import requests
def worker(settings):
lookup_url = settings['lookup_url']
api_sid = settings['api_sid']
auth_token = settings['auth_token']
args = settings['args']
resp = requests.post(lookup_url,
auth=(api_sid, auth_token),
data={'data': args},
timeout=5.0,
)
def main(request):
registry = request.registry
settings = registry.settings
for _ in range(3):
threading.Thread(target=worker, args=(settings,)).start()
Please let me know any solutions.
UWSGI does not enable Python threads by default:
By default the Python plugin does not initialize the GIL. This means
your app-generated threads will not run. If you need threads, remember
to enable them with enable-threads. Running uWSGI in multithreading
mode (with the threads options) will automatically enable threading
support. This “strange” default behaviour is for performance reasons,
no shame in that.
http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
Related
I am trying to use my telegram bot with Django. I want the code to keep running in the background. I am Using the apps.py to do this but there's one problem when the bot starts as it's an infinite loop, the Django server is never started.
Apps.py:
from django.apps import AppConfig
import os
class BotConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'bot'
def ready(self):
from . import jobs
if os.environ.get('RUN_MAIN', None) != 'true':
jobs.StartBot()
Jobs.py:
def StartBot():
updater = Updater("API KEY")
dp = updater.dispatcher
dp.add_handler(ChatMemberHandler(GetStatus, ChatMemberHandler.CHAT_MEMBER))
updater.start_polling(allowed_updates=Update.ALL_TYPES)
updater.idle()
What's the best way to run my bot in the background? while making sure that the Django server runs normally. I tried Django background tasks but it's not compatible with Django 4.0.
The purpose of Updater.idle is to keep the main thread alive because start_polling only starts some background threads that don't block the main thread. If you want to run other stuff in the main thread, skip updater.idle() and instead call Updater.stop manually when the program should shut down.
Disclaimer: I'm currently the maintainer of python-telegram-bot
I've got a simple FastAPI webapp going and I'd like to be able to check the database connection on startup (and retry connection if it fails)
I've got the following code, but it doesn't feel right
# main.py
import uvicorn
from backend.app import app
if __name__ == "__main__":
uvicorn.run(app, port=8001)
# app.py
# ... omitted for brevity
from backend.database import notes, tags
# ... omitted for brevity
# database.py
from motor.motor_asyncio import AsyncIOMotorClient
from asyncio import get_event_loop
client = AsyncIOMotorClient("localhost", 27027)
loop = get_event_loop()
data = loop.run_until_complete(client.server_info())
db = client.notes_db
notes = db.notes
tags = db.tags
Without get_event_loop() and the subsequent loop.run_until_complete() call it won't test the database connection until you actually try to access / write to it.
My goal is to be able to halt the startup process until it can successfully connect to a database, is there any clean way to do this with Python and motor.io (https://motor.readthedocs.io/, sorry there's no tag for it) ?
the startup event in FastAPI is the deal here I guess. I addition this repository is a nice example and this thread could even provide you with more information. You could execute your tests within the startup event. This means the application won't start until the startup event has been successfully executed.
I am using CherryPy to speak to an authentication server. The script runs fine if all the inputted information is fine. But if they make an mistake typing their ID the internal HTTP error screen fires ok, but the server keeps running and nothing else in the script will run until the CherryPy engine is closed so I have to manually kill the script. Is there some code I can put in the index along the lines of
if timer >10 and connections == 0:
close cherrypy (< I have a method for this already)
Im mostly a data mangler, so not used to web servers. Googling shows lost of hits for closing CherryPy when there are too many connections but not when there have been no connections for a specified (short) time. I realise the point of a web server is usually to hang around waiting for connections, so this may be an odd case. All the same, any help welcome.
Interesting use case, you can use the CherryPy plugins infrastrcuture to do something like that, take a look at this ActivityMonitor plugin implementation, it shutdowns the server if is not handling anything and haven't seen any request in a specified amount of time (in this case 10 seconds).
Maybe you have to adjust the logic on how to shut it down or do anything else in the _verify method.
If you want to read a bit more about the publish/subscribe architecture take a look at the CherryPy Docs.
import time
import threading
import cherrypy
from cherrypy.process.plugins import Monitor
class ActivityMonitor(Monitor):
def __init__(self, bus, wait_time, monitor_time=None):
"""
bus: cherrypy.engine
wait_time: Seconds since last request that we consider to be active.
monitor_time: Seconds that we'll wait before verifying the activity.
If is not defined, wait half the `wait_time`.
"""
if monitor_time is None:
# if monitor time is not defined, then verify half
# the wait time since the last request
monitor_time = wait_time / 2
super().__init__(
bus, self._verify, monitor_time, self.__class__.__name__
)
# use a lock to make sure the thread that triggers the before_request
# and after_request does not collide with the monitor method (_verify)
self._active_request_lock = threading.Lock()
self._active_requests = 0
self._wait_time = wait_time
self._last_request_ts = time.time()
def _verify(self):
# verify that we don't have any active requests and
# shutdown the server in case we haven't seen any activity
# since self._last_request_ts + self._wait_time
with self._active_request_lock:
if (not self._active_requests and
self._last_request_ts + self._wait_time < time.time()):
self.bus.exit() # shutdown the engine
def before_request(self):
with self._active_request_lock:
self._active_requests += 1
def after_request(self):
with self._active_request_lock:
self._active_requests -= 1
# update the last time a request was served
self._last_request_ts = time.time()
class Root:
#cherrypy.expose
def index(self):
return "Hello user: current time {:.0f}".format(time.time())
def main():
# here is how to use the plugin:
ActivityMonitor(cherrypy.engine, wait_time=10, monitor_time=5).subscribe()
cherrypy.quickstart(Root())
if __name__ == '__main__':
main()
I have numerous Bokeh Server files in a directory say.. /dir/bokeh/, assume the bokeh servers are called bokeh1.py, bokeh2.py, bokeh3.py
The file structure is like so:
|--dir
|---flask.py
|---bokeh
|--bokeh1.py
|--bokeh2.py
I am deploying them all on flask like so:
files=[]
for file in os.listdir("/dir/bokeh/"):
if file.endswith('.py'):
file="bokeh/"+file
files.append(file)
argvs = {}
urls = []
for i in files:
argvs[i] = None
urls.append(i.split('\\')[-1].split('.')[0])
host = 'myhost.com'
apps = build_single_handler_applications(files, argvs)
bokeh_tornado = BokehTornado(apps, extra_websocket_origins=["myhost.com"])
bokeh_http = HTTPServer(bokeh_tornado)
sockets, port = bind_sockets("myhost.com", 0)
bokeh_http.add_sockets(sockets)
On update to Tornado 6.0.2, and deploying Flask, I get the Runtimerror There is no current event loop in thread Thread-1. On deeper research Tornado uses asyncio by default and imposes some restrictions. So I add asyncio.set_event_loop(asyncio.new_event_loop()) to the following.
def bk_worker():
asyncio.set_event_loop(asyncio.new_event_loop())####
server = BaseServer(IOLoop.current(), bokeh_tornado, bokeh_http)
server.start()
server.io_loop.start()
gc.collect()
from threading import Thread
Thread(target=bk_worker).start()
However, upon opening the bokeh server url through flask, the bokeh server selected (any of them) do not load and simply return a blank page. How can I circumvent this?
setting asyncio.set_event_loop_policy(AnyThreadEventLoopPolicy)) yields the same result.
edit:The previous code works with python 2/3, Tornado 4.5.3
I think this is a known Bokeh issue. The best way for now is to downgrade to Tornado 4.5.3.
pip install tornado==4.5.3
I have been trying to run tornado with aiomysql in multi-process mode with the following code
#asyncio.coroutine
def get_mysql_connection(loop):
return (yield from aiomysql.create_pool(host=host,port=3306,user=user, password=pass, db=db, loop=loop))
if __name__ == "__main__":
tornado.platform.asyncio.AsyncIOMainLoop().install()
ioloop = asyncio.get_event_loop()
mysql = ioloop.run_until_complete(get_mysql_connection(ioloop))
options.parse_config_file("app.conf")
app = make_app(mysql)
print('listening on %s:%s...' %(options.host, options.port))
server = tornado.httpserver.HTTPServer(app)
server.listen(options.port)
server.start(0) #this is my problem
ioloop.run_forever()
But I keep getting the below error
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()
Everything is working fine except for the ioloop.start(0) line, is it possible to make the two libraries aiomysql and tornado work nicely in multi-process mode? If no what are my other options
Tornado version 4.4.2
python version 3.6.0
aiomysql version 0.0.9
Like the message says, you can only fork worker processes if you do it before the IOLoop is initialized. This gets a little delicate with AsyncIOMainLoop since you want to install that as early as possible. The solution is to start your program with this sequence:
tornado.options.parse_config_file(...)
socks = tornado.netutil.bind_sockets(options.port, options.host)
tornado.process.fork_processes(0)
tornado.asyncio.AsyncIOMainLoop().install()
# Initialize the rest of your app, create the HTTPServer,
# and instead of listen() or start(), do
server.add_sockets(socks)
what are my other options
In my opinion it's better to use an external process manager like supervisord instead of forking multiple processes within your app. That avoids most of these initialization-order traps.