I am trying to use my telegram bot with Django. I want the code to keep running in the background. I am Using the apps.py to do this but there's one problem when the bot starts as it's an infinite loop, the Django server is never started.
Apps.py:
from django.apps import AppConfig
import os
class BotConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'bot'
def ready(self):
from . import jobs
if os.environ.get('RUN_MAIN', None) != 'true':
jobs.StartBot()
Jobs.py:
def StartBot():
updater = Updater("API KEY")
dp = updater.dispatcher
dp.add_handler(ChatMemberHandler(GetStatus, ChatMemberHandler.CHAT_MEMBER))
updater.start_polling(allowed_updates=Update.ALL_TYPES)
updater.idle()
What's the best way to run my bot in the background? while making sure that the Django server runs normally. I tried Django background tasks but it's not compatible with Django 4.0.
The purpose of Updater.idle is to keep the main thread alive because start_polling only starts some background threads that don't block the main thread. If you want to run other stuff in the main thread, skip updater.idle() and instead call Updater.stop manually when the program should shut down.
Disclaimer: I'm currently the maintainer of python-telegram-bot
Related
The telegram bot I'm making can execute a function that takes a few minutes to process and I'd like to be able to continue to use the bot while it's processing the function.
I'm using aiogram, asyncio and I tried using Python threading to make this possible.
The code I currently have is:
import asyncio
from queue import Queue
from threading import Thread
import time
import logging
from aiogram import Bot, types
from aiogram.types.message import ContentType
from aiogram.contrib.middlewares.logging import LoggingMiddleware
from aiogram.contrib.fsm_storage.memory import MemoryStorage
from aiogram.dispatcher import Dispatcher, FSMContext
from aiogram.utils.executor import start_webhook
from aiogram.types import InputFile
...
loop = asyncio.get_event_loop()
bot = Bot(token=BOT_TOKEN, loop=loop)
dp = Dispatcher(bot, storage=MemoryStorage())
dp.middleware.setup(LoggingMiddleware())
task_queue = Queue()
...
async def send_result(id):
logging.warning("entered send_result function")
image_res = InputFile(path_or_bytesio="images/result/res.jpg")
await bot.send_photo(id, image_res, FINISHED_MESSAGE)
def queue_processing():
while True:
if not task_queue.empty():
task = task_queue.get()
if task["type"] == "nst":
nst.run(task["style"], task["content"])
send_fut = asyncio.run_coroutine_threadsafe(send_result(task['id']), loop)
send_fut.result()
task_queue.task_done()
time.sleep(2)
if __name__ == "__main__":
executor_images = Thread(target=queue_processing, args=())
executor_images.start()
start_webhook(
dispatcher=dp,
webhook_path=WEBHOOK_PATH,
skip_updates=False,
on_startup=on_startup,
host=WEBAPP_HOST,
port=WEBAPP_PORT,
)
So I'm trying to setup a separate thread that's running a loop that is processing a queue of slow tasks thus allowing to continue chatting with the bot in the meantime and which would send the result message (image) to the chat after it's finished with a task.
However, this doesn't work. My friend came up with this solution while doing a similar task about a year ago, and it does work in his bot, but it doesn't seem to work in mine.
Judging by logs, it never even enters the send_result function, because the warning never comes through. The second thread does work properly though and the result image is saved and is located in its assigned path by the time nst.run finishes working.
I tried A LOT of different things and I'm very puzzled why this solution doesn't work for me because it does work with another bot. For example, I tried using asyncio.create_task instead of asyncio.run_coroutine_threadsafe, but to no avail.
To my understanding, you don't need to pass a loop to aiogram's Bot or Dispatcher anymore, but in that case I don't know how to send a task to the main thread from the second one.
Versions I'm using: aiogram 2.18, asyncio 3.4.3, Python 3.9.10.
Solved, the issue was that you can't access the bot's loop directly (with bot.loop or dp.loop) even if you pass your own asyncio loop to the bot or the dispatcher.
So the solution was to access the main thread's loop by using asyncio.get_event_loop() (which returns currently running loop, if there's one) from within one of the message handlers, because the loop is running at this point, and pass it to asyncio.run_coroutine_threadsafe (I used the "task" dictionary for that) like this: asyncio.run_coroutine_threadsafe(send_result(task['id']), task['loop']).
I've got a simple FastAPI webapp going and I'd like to be able to check the database connection on startup (and retry connection if it fails)
I've got the following code, but it doesn't feel right
# main.py
import uvicorn
from backend.app import app
if __name__ == "__main__":
uvicorn.run(app, port=8001)
# app.py
# ... omitted for brevity
from backend.database import notes, tags
# ... omitted for brevity
# database.py
from motor.motor_asyncio import AsyncIOMotorClient
from asyncio import get_event_loop
client = AsyncIOMotorClient("localhost", 27027)
loop = get_event_loop()
data = loop.run_until_complete(client.server_info())
db = client.notes_db
notes = db.notes
tags = db.tags
Without get_event_loop() and the subsequent loop.run_until_complete() call it won't test the database connection until you actually try to access / write to it.
My goal is to be able to halt the startup process until it can successfully connect to a database, is there any clean way to do this with Python and motor.io (https://motor.readthedocs.io/, sorry there's no tag for it) ?
the startup event in FastAPI is the deal here I guess. I addition this repository is a nice example and this thread could even provide you with more information. You could execute your tests within the startup event. This means the application won't start until the startup event has been successfully executed.
I'd like to write a Telegram bot which can provide Scrapy stats on request.
My try mostly works, the only issue is that forcefully closing the spider (obviously) doesn't stop the bot.
So I have two questions :
is my general approach the correct one?
is it possible to close the bot even on forceful shutdown of the spider?
Here is the relevant class:
class TelegramBot(object):
telegram_token = telegram_credentials.token
#classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def __init__(self, crawler):
self.crawler = crawler
cs = crawler.signals
cs.connect(self._spider_closed, signal=signals.spider_closed)
"""Start the bot."""
# Create the Updater and pass it your bot's token.
# Make sure to set use_context=True to use the new context based callbacks
# Post version 12 this will no longer be necessary
self.updater = Updater(self.telegram_token, use_context=True)
# Get the dispatcher to register handlers
dp = self.updater.dispatcher
# on different commands - answer in Telegram
dp.add_handler(CommandHandler("stats", self.stats))
# Start the Bot
self.updater.start_polling()
def _spider_closed(self, spider, reason):
# Stop the Bot
self.updater.stop()
def stats(self, update, context):
# Send a message with the stats
msg = (
"Spider "
+ self.crawler.spider.name
+ " stats: "
+ str(self.crawler.stats.get_stats())
)
update.message.reply_text(msg)
Here you can find my full code inside the Scrapy tutorial quotes spider https://github.com/jtommi/scrapy_telegram-bot_example/blob/master/tutorial/tutorial/telegram-bot.py
My code is a combination of
latencies extension from the "Learning Scrapy" book https://github.com/scalingexcellence/scrapybook/blob/master/ch09/properties/properties/latencies.py
echobot example from the python-telegram-bot library https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/echobot.py
official scrapy documentation on stats collection https://docs.scrapy.org/en/latest/topics/stats.html
Calling the updater.stop() will definitely stop the bot. From the docs of python-telegram-bot,
"""Stops the polling/webhook thread, the dispatcher and the job queue."""
Check if updater.stop() is called after spider closes. Bot might not stop immediately, but eventually will stop.
I have been trying to run tornado with aiomysql in multi-process mode with the following code
#asyncio.coroutine
def get_mysql_connection(loop):
return (yield from aiomysql.create_pool(host=host,port=3306,user=user, password=pass, db=db, loop=loop))
if __name__ == "__main__":
tornado.platform.asyncio.AsyncIOMainLoop().install()
ioloop = asyncio.get_event_loop()
mysql = ioloop.run_until_complete(get_mysql_connection(ioloop))
options.parse_config_file("app.conf")
app = make_app(mysql)
print('listening on %s:%s...' %(options.host, options.port))
server = tornado.httpserver.HTTPServer(app)
server.listen(options.port)
server.start(0) #this is my problem
ioloop.run_forever()
But I keep getting the below error
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()
Everything is working fine except for the ioloop.start(0) line, is it possible to make the two libraries aiomysql and tornado work nicely in multi-process mode? If no what are my other options
Tornado version 4.4.2
python version 3.6.0
aiomysql version 0.0.9
Like the message says, you can only fork worker processes if you do it before the IOLoop is initialized. This gets a little delicate with AsyncIOMainLoop since you want to install that as early as possible. The solution is to start your program with this sequence:
tornado.options.parse_config_file(...)
socks = tornado.netutil.bind_sockets(options.port, options.host)
tornado.process.fork_processes(0)
tornado.asyncio.AsyncIOMainLoop().install()
# Initialize the rest of your app, create the HTTPServer,
# and instead of listen() or start(), do
server.add_sockets(socks)
what are my other options
In my opinion it's better to use an external process manager like supervisord instead of forking multiple processes within your app. That avoids most of these initialization-order traps.
This is yet another flask dev server reloader question. There are a million questions asking why it loads everything twice, and this is not one of them. I understand that it loads everything twice, my question involves dealing with this reality and I haven't found an answer that I think addresses what I'm trying to do.
My question is, how can I cleanup all app objects at exit?
My current approach is shown below. In this example I run my cleanup code using an atexit function.
from flask import Flask
app = Flask(__name__)
print("start_app_id: ", '{}'.format(id(app)))
import atexit
#atexit.register
def shutdown():
print("AtExit_app_id: ", '{}'.format(id(app)))
#do some cleanup on the app object here
if __name__ == "__main__":
import os
if os.environ.get('WERKZEUG_RUN_MAIN') == "true":
print("reloaded_main_app_id: ", '{}'.format(id(app)))
else:
print("first_main_app_id: ", '{}'.format(id(app)))
app.run(host='0.0.0.0', debug=True)
The output of this code is as follows:
start_app_id: 140521561348864
first_main_app_id: 140521561348864
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
start_app_id: 140105598483312
reloaded_main_app_id: 140105598483312
* Debugger is active!
* Debugger pin code: xxx-xxx-xxx
^CAtExit_app_id: 140521561348864
Note that when first loaded, an app object with ID '864 is created. During the automatic reloading, a new app object with ID '312 is created. Then when I hit Ctrl-C (last line), the atexit routine is called and the original '864 app object is the one that is accessible using the app variable -- not the newer '312 app object.
I want to be able to do cleanup on all app objects floating around when the server closes or is Ctrl-C'd (in this case both '864 and '312). Any recs on how to do this?
Or alternately, if I could just run the cleanup on the newer '312 object created after reloading I could also make that work -- however my current approach only lets me cleanup the original app object.
Thanks.
UPDATE1: I found a link that suggested using try/finally instead of the atexit hook to accomplish what I set out to do above. Switching to this results in exactly the same behavior as atexit and therefore doesn't help with my issue:
from flask import Flask
app = Flask(__name__)
print("start_app_id: ", '{}'.format(id(app)))
if __name__ == "__main__":
import os
if os.environ.get('WERKZEUG_RUN_MAIN') == "true":
print("reloaded_main_app_id: ", '{}'.format(id(app)))
else:
print("first_main_app_id: ", '{}'.format(id(app)))
try:
app.run(host='0.0.0.0', debug=True)
finally:
print("Finally_app_id: ", '{}'.format(id(app)))
#do app cleanup code here
After some digging through the werkzeug source I found the answer. The answer is that it isn't possible to do what I wanted -- and this is by design.
When using the flask dev server (werkzeug) it isn't possible to cleanup all existing app objects upon termination (e.g. ctrl-C) because the werkzeug server catches the keyboardinterrupt exception and "passes" on it. You can see this in the last lines of werkzeug's _reloader.py in the run_with_reloader function:
def run_with_reloader(main_func, extra_files=None, interval=1,
reloader_type='auto'):
"""Run the given function in an independent python interpreter."""
import signal
reloader = reloader_loops[reloader_type](extra_files, interval)
signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
try:
if os.environ.get('WERKZEUG_RUN_MAIN') == 'true':
t = threading.Thread(target=main_func, args=())
t.setDaemon(True)
t.start()
reloader.run()
else:
sys.exit(reloader.restart_with_reloader())
except KeyboardInterrupt:
pass
If you replace the above "except KeyboardInterrupt:" with "finally:", and then run the second code snippet in the original question, you observe that both of the created app objects are cleaned up as desired. Interestingly, the first code snippet (that uses #atexit) still doesn't work as desired after making these changes.
So in conclusion, you can cleanup all existing app objects when using the flask dev server, but you need to modify the werkzeug source to do so.