I am a Discord bot developer, and recently completed an order. The client upon setting the application up on their server initially had no issues, but according to them after running for "about three hours" the program begins spitting a specific stack trace error and no longer accepting commands.
The bot is built using Discord.py and uses Peewee as an ORM, using PyMySQL as the database driver. The server the client is running it on is hosted by DigitalOcean and if any information about the hardware, etc. is needed the client is able to give me that information on request. We have already attempted uninstalling and reinstalling all dependencies, as well as trying different distributions of them, but the errors persist.
This is the exact trace that the client is receiving:
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 2666, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 179, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 2875, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 750, in _execute_command
raise err.InterfaceError("(0, '')")
peewee.InterfaceError: (0, '')
The relevant portions from my database.py file, where the database connection is opened:
import discord
from peewee import *
from config import db_host, db_port, mysql_db_name, \
mysql_db_username, mysql_db_password
db_connection = MySQLDatabase(
mysql_db_name,
user = mysql_db_username,
password = mysql_db_password,
host = db_host,
port = db_port
)
def create_db_tables():
# create_tables() does safe creation by default, and will simply not create
# table if it already exists
db_connection.create_tables([User])
The relevant portions from my bot.py file, specifically the bot startup function that runs when the bot is first opened, and the lines that create and start the bot client:
client = discord.Client()
async def bot_startup():
# Check for config changes
if client.user.name != config.bot_username:
print("Username Updated To: {}".format(config.bot_username))
await client.edit_profile(username=config.bot_username)
# Start 'playing' message
await client.change_presence(
game=discord.Game( name=config.playing_message )
)
# Prepare database
database.create_db_tables()
print("Database Connected")
print("Connected Successfully")
# ...
#client.event
async def on_ready():
await bot_startup()
# ...
client.run(config.token)
According to the client, restarting the bot temporarily solves the problem and it runs fine for a few hours before the errors start up again. The bot no longer responds to any incoming commands once the errors start, and if enough errors are thrown, crashes completely.
What is typically the cause of this error, and what steps should be taken to fix whatever is causing it?
discord.py is asynchronous whilst PyMySQL is not - therefore it is blocking the discord.py runtime. Instead of PyMySQL use AIOMySQL which is non-blocking and might just solve your error.
The timing of when the error appears makes me think of a problem I also encountered when communicating to a database.
The problem could be that since the connection is opened when the bot starts up (or even when the program begins its execution) and is used sporadically, the database might close the connection, therefore any further execution will result in an error. To combat this, I create this decorator for all the methods of my Database Class
def ensures_connected(f):
def wrapper(*args):
args[0].connection.ping(reconnect=True, attempts=3, delay=2)
return f(*args)
return wrapper
to be placed above any method or function that has to communicate with the database.
The line args[0].connection.ping(reconnect=True, attempts=3, delay=2) means that we will call on self.connection (since it is the first argument passed to the method when called) the method ping, that allows to reconnect if the connection was dropped.
self.connection in my code is an object returned by the method call MySQLConnection.connect and should be equivalent to your obejct db_connection
This decorator should be placed above the method definition like the following example :
def ensures_connected(f):
def wrapper(*args):
args[0].connection.ping(reconnect=True, attempts=3, delay=2)
return f(*args)
return wrapper
class Database:
def __init__(self):
self.connection = mysql.connector.connect(
user=self.username, password=password, host=self.host, database=self.database)
# ...
#ensures_connected
def is_member_registered(self, guild: Guild, member: Member):
# ...
return
According to this comment you can use following code for reconnect automatically:
from peewee import *
from playhouse.shortcuts import ReconnectMixin
class ReconnectMySQLDatabase(ReconnectMixin, MySQLDatabase):
pass
db = ReconnectMySQLDatabase(...)
Related
I have an API server that runs asynchronous functions using FastAPI & uvicorn on Windows. Every now and then the program throws a network error via asyncio that causes the API to become unresponsive, but can't be caught by my script. I'd like to be able to put a trycatch somewhere that will catch this type of error so I can shut down & restart the server, but the traceback for the error doesn't have any of my code in it.
The following is a simple example to illustrate the file structure; first the file app.py defines the API:
from fastapi import FastAPI
app = FastAPI()
#app.get("/check-status")
async def root():
return {"Alive": True}
the server parameters are defined in a config.json:
{
"host": "0.0.0.0",
"port": 80
}
then the server is launched programmatically from another script run_app.py, with a trycatch block that should catch any errors thrown during the execution, show it on the console, and restart:
import uvicorn
import json
from datetime import datetime
if __name__ == '__main__':
with open('path/to/config.json', 'r') as f:
config = uvicorn.Config(**json.load(f))
server = uvicorn.Server(config)
try:
server.run()
except Exception as e:
err_timestamp = datetime.now().ctime()
print(f'{err_timestamp}: Exception raised\n{e}\nRestarting...')
server.shutdown()
server.run()
However, the server still ends up crashing after the following error:
Task exception was never retrieved
future: <Task finished name='Task-33094' coro=<IocpProactor.accept.<locals>.accept_coro() done, defined at C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py:563> exception=OSError(22, 'The specified network name is no longer available', None, 64, None)>
Traceback (most recent call last):
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 566, in accept_coro
await future
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 812, in _poll
value = callback(transferred, key, ov)
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 555, in finish_accept
ov.getresult()
OSError: [WinError 64] The specified network name is no longer available
Accept failed on a socket
socket: <asyncio.TransportSocket fd=732, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 80)>
Traceback (most recent call last):
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 818, in loop
conn, addr = f.result()
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 566, in accept_coro
await future
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 812, in _poll
value = callback(transferred, key, ov)
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 555, in finish_accept
ov.getresult()
OSError: [WinError 64] The specified network name is no longer available
Clearly, the error happens purely within asyncio, and isn't actually thrown at any point in the execution of my code, so I haven't been able to replicate this error myself to test a solution. I know this type of WinError is usually down to network issues and isn't something that can actually be solved by the API, but I need to at least be able to deal with it when it occurs.
Environment; Python 3.8.10, Uvicorn version 0.17.4, FastAPI version 0.73.0, Windows 10
I'm testing my database query middleware (Django docs here) on a sample django app with a Postgres db. The app is the cookiecutter boilerplate. My goal with the middleware is simply to log the user ID for all database queries. Using Python3.9 and Django3.2.13. My middleware code is below:
# Middleware code
import logging
import django
from django.db import connection
django_version = django.get_version()
logger = logging.getLogger(__name__)
class Logger:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
with connection.execute_wrapper(QueryWrapper(request)):
return self.get_response(request)
class QueryWrapper:
def __init__(self, request):
self.request = request
def __call__(self, execute, sql, params, many, context):
# print(self.request.user)
return execute(sql, params, many, context)
If print(self.request.user.id) is commented out, everything works fine. However, I've found that uncommenting it, or any type of interaction with the user field in the self.request object, causes a Recursion Error:
RecursionError at /about/
maximum recursion depth exceeded
Request Method: GET
Request URL: http://127.0.0.1:8000/about/
Django Version: 3.2.13
Exception Type: RecursionError
Exception Value:
maximum recursion depth exceeded
Exception Location: /opt/homebrew/lib/python3.9/site-packages/django/db/models/sql/query.py, line 192, in __init__
Python Executable: /opt/homebrew/opt/python#3.9/bin/python3.9
Python Version: 3.9.13
In the error page, that is followed by many repetitions of the below error:
During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
/opt/homebrew/lib/python3.9/site-packages/django/contrib/sessions/backends/base.py, line 233, in _get_session
return self._session_cache …
During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
/opt/homebrew/lib/python3.9/site-packages/django/contrib/sessions/backends/base.py, line 233, in _get_session
return self._session_cache …
From consulting other SO posts, it seems accessing the user field should work fine. I've checked that the django_session table exists, and my middleware is also located at the very bottom of my middlewares (that include "django.contrib.sessions.middleware.SessionMiddleware" and "django.contrib.auth.middleware.AuthenticationMiddleware")
What's wrong here?
The following things happen when you write self.request.user:
The request is checked for an attribute _cached_user, if present the cached user is returned if not auth.get_user is called with the request.
Using the session key, Django checks the session to get the authentication backend used for the current user. Here if you are using database based sessions a database query is fired.
Using the above authentication backend Django makes a database query to get the current user using their ID.
As noted from the above points, unless there is a cache hit this process is going to cause a database query.
Using database instrumentation you have installed a wrapper around database queries, the problem is that this wrapper itself is trying to make more queries (trying to get the current user), causing it to call itself. One solution would be to get the current user before installing your wrapper function:
class Logger:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
_user_authenticated = request.user.is_authenticated # Making sure user is fetched (Lazy object)
with connection.execute_wrapper(QueryWrapper(request)):
return self.get_response(request)
class QueryWrapper:
def __init__(self, request):
self.request = request
def __call__(self, execute, sql, params, many, context):
print(self.request.user)
return execute(sql, params, many, context)
The thing thats weird is that, I dont even have 343 lines of code, I only have like 30, so I'm really not sure why this is happening
The error :
Ignoring exception in on_message
Traceback (most recent call last):
File "C:\Users\aiosdj\PycharmProjects\nerdeyes\venv\lib\site-packages\discord\client.py", line 343, in _run_event
await coro(*args, **kwargs)
File "C:/Users/aiosdj/PycharmProjects/nerdeyes/main.py", line 28, in on_message
await bot.send_message(message.channel, msg)
AttributeError: 'Bot' object has no attribute 'send_message'
The code :
import asyncio
from discord.ext import commands
bot = commands.Bot(command_prefix='.nerd ', description = 'nerd eyes')
#bot.event
async def on_ready():
guild_count = 0
for guild in bot.guilds:
print(f"- {guild.id} (name: {guild.name})")
guild_count = guild_count + 1
print('Nerdeyes has awoken in ' + str(guild_count) + " servers")
#bot.event
async def on_message(message):
if message.author == bot.user:
return
if message.content.startswith('.nerd'):
msg = 'eyes'.format(message)
await bot.send_message(message.channel, msg)
bot.run("TOKEN")
Bot.send_message doesn't exist, which is what your error also says. You need something that implements Messageable, like a Channel, User, or Member - and the method is called send, not send_message. A quick look at the API docs for Bot also shows that this method is nowhere to be found. It's generally not a bad idea to first check if a method exists before using it.
await message.channel.send(msg)
Also, msg = 'eyes'.format(message) - the .format here does nothing at all, this is the same as just 'eyes'.
I dont even have 343 lines of code
The error says it happens in \venv\lib\site-packages\discord\client.py line 343, not your file. This is a line in the Discord library, which your code calls, so at one point or another one of those methods will crash. The line below in the error says which line of yours it comes from (28).
ps. You just leaked your bot token, generate a new one.
I have set up a dask cluster. I can access a web dashboard, but when I'm trying to connect to the scheduler:
from dask.distributed import Client
client = Client('192.168.0.10:8786')
I get the following error:
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
File "/home/user/venv/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 285, in result
raise CancelledError
concurrent.futures._base.CancelledError
Also, when I'm trying to execute some tasks on the cluster, all tasks are computed correctly and the result is fetched but then the above error occurs at the end.
Do you have any ideas how to fix it? I think it's a client problem, but where. Thank you very much.
You are running sync function in async tornado
Try this one:
from dask.distributed import Client
from tornado.ioloop import IOLoop
async def f():
client = await Client(address='192.168.0.10:8786', asynchronous=True)
future = client.submit(DO SOMETHING HERE)
result = await future
await client.close()
return result
IOLoop().run_sync(f)
I'm getting an aiohttp client_exception.ServerDisconnectedError whenever I do more than ~200 requests to an API I'm hitting using asyncio & aiohttp. It doesn't seem to be my code because it works consistently with smaller number of requests, but fails on any larger number. Trying to understand if this error is related to aiohttp, or my code, or with the API endpoint itself? There doesn't seem to be much info online around this.
Traceback (most recent call last):
File "C:/usr/PycharmProjects/api_framework/api_framework.py", line 27, in <module>
stuff = abc.do_stuff_2()
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 72, in do_stuff
self.queue_manager(self.do_stuff(json_data))
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 115, in queue_manager
loop.run_until_complete(future)
File "C:\Python36x64\lib\asyncio\base_events.py", line 466, in run_until_complete
return future.result()
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 96, in do_stuff
result = await asyncio.gather(*tasks)
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 140, in async_post
async with session.post(self.api_attr.api_endpoint + resource, headers=self.headers, data=data) as response:
File "C:\Python36x64\lib\site-packages\aiohttp\client.py", line 843, in __aenter__
self._resp = await self._coro
File "C:\Python36x64\lib\site-packages\aiohttp\client.py", line 387, in _request
await resp.start(conn)
File "C:\Python36x64\lib\site-packages\aiohttp\client_reqrep.py", line 748, in start
message, payload = await self._protocol.read()
File "C:\Python36x64\lib\site-packages\aiohttp\streams.py", line 533, in read
await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None
here's some of the code to generate the async requests:
def some_other_method(self):
self.queue_manager(self.do_stuff(all_the_tasks))
def queue_manager(self, method):
print('starting event queue')
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(method)
loop.run_until_complete(future)
loop.close()
async def async_post(self, resource, session, data):
async with session.post(self.api_attr.api_endpoint + resource, headers=self.headers, data=data) as response:
resp = await response.read()
return resp
async def do_stuff(self, data):
print('queueing tasks')
tasks = []
async with aiohttp.ClientSession() as session:
for row in data:
task = asyncio.ensure_future(self.async_post('my_api_endpoint', session, row))
tasks.append(task)
result = await asyncio.gather(*tasks)
self.load_results(result)
Once the tasks have completed, self.load_results() method just parses the json and updates the DB.
It is most likely caused by the configuration of the HTTP server. There are at least two possible reasons for the ServerDisconnectedError:
The server could limit the number of parallel TCP connections that can be made from a single IP address. By default, aiohttp already limits the number of parallel connections to 100. You can try reducing the limit and see if it solve the issue. To do so, you can create a custom TCPConnector with a different limit value and pass it to the ClientSession:
connector = aiohttp.TCPConnector(limit=50)
async with aiohttp.ClientSession(connector=connector) as session:
# Use your session as usual here
The server could limit the duration of a TCP connection. By default, aiohttp uses HTTP keep-alive so that the same TCP connection can be used for multiple requests. This improves performances since a new TCP connection does not have to be made for each request. However, some servers limit the duration of a TCP connection, and if you use the same TCP connection for many requests, the server can close it before you are done with it. You can disable HTTP keep-alive as a workaround. To do so, you can create a custom TCPConnector with the parameter force_close set to True, and pass it to the ClientSession:
connector = aiohttp.TCPConnector(force_close=True)
async with aiohttp.ClientSession(connector=connector) as session:
# Use your session as usual here
I had the same issue and disabling HTTP keep-alive was the solution for me. Hope this helps.
This is most likely the server's API not being happy with multiple requests being done asynchronously.
You can limit the amount of concurrent calls with asyncio's semaphores.
In your case I would use it within a context manager as:
async def do_stuff(self, data):
print('queueing tasks')
tasks = []
semaphore = asyncio.Semaphore(200)
async with semaphore:
async with aiohttp.ClientSession() as session:
for row in data:
task = asyncio.ensure_future(self.async_post('my_api_endpoint', session, row))
tasks.append(task)
result = await asyncio.gather(*tasks)
self.load_results(result)
I think it's quite possible the other answers are correct, but there's also one more possibility - it seems aiohttp has at least one currently [June 2021] unfixed race condition in it's streams code:
https://github.com/aio-libs/aiohttp/issues/4581
I see the same issue in my project, and it's rare enough (and server disconnect isn't the only symptom, I sometimes get "payload not complete") it feels more like a race condition. I also saw issues like aiohttp putting a packet of data from one response into a different response.
In the end, I switched to https://www.python-httpx.org - this decreased the number of problems, and let me to eventually that some of the 'payload not complete' error was probably related to timeout for sending a large binary response on the server that was occasionally triggering. In general I found httpx to be more reliable, and it's really good that you can use the same package/APIs to support both sync and async.