Related
I have an API server that runs asynchronous functions using FastAPI & uvicorn on Windows. Every now and then the program throws a network error via asyncio that causes the API to become unresponsive, but can't be caught by my script. I'd like to be able to put a trycatch somewhere that will catch this type of error so I can shut down & restart the server, but the traceback for the error doesn't have any of my code in it.
The following is a simple example to illustrate the file structure; first the file app.py defines the API:
from fastapi import FastAPI
app = FastAPI()
#app.get("/check-status")
async def root():
return {"Alive": True}
the server parameters are defined in a config.json:
{
"host": "0.0.0.0",
"port": 80
}
then the server is launched programmatically from another script run_app.py, with a trycatch block that should catch any errors thrown during the execution, show it on the console, and restart:
import uvicorn
import json
from datetime import datetime
if __name__ == '__main__':
with open('path/to/config.json', 'r') as f:
config = uvicorn.Config(**json.load(f))
server = uvicorn.Server(config)
try:
server.run()
except Exception as e:
err_timestamp = datetime.now().ctime()
print(f'{err_timestamp}: Exception raised\n{e}\nRestarting...')
server.shutdown()
server.run()
However, the server still ends up crashing after the following error:
Task exception was never retrieved
future: <Task finished name='Task-33094' coro=<IocpProactor.accept.<locals>.accept_coro() done, defined at C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py:563> exception=OSError(22, 'The specified network name is no longer available', None, 64, None)>
Traceback (most recent call last):
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 566, in accept_coro
await future
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 812, in _poll
value = callback(transferred, key, ov)
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 555, in finish_accept
ov.getresult()
OSError: [WinError 64] The specified network name is no longer available
Accept failed on a socket
socket: <asyncio.TransportSocket fd=732, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 80)>
Traceback (most recent call last):
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 818, in loop
conn, addr = f.result()
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 566, in accept_coro
await future
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 812, in _poll
value = callback(transferred, key, ov)
File "C:\Users\Administrateur\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 555, in finish_accept
ov.getresult()
OSError: [WinError 64] The specified network name is no longer available
Clearly, the error happens purely within asyncio, and isn't actually thrown at any point in the execution of my code, so I haven't been able to replicate this error myself to test a solution. I know this type of WinError is usually down to network issues and isn't something that can actually be solved by the API, but I need to at least be able to deal with it when it occurs.
Environment; Python 3.8.10, Uvicorn version 0.17.4, FastAPI version 0.73.0, Windows 10
I am a Discord bot developer, and recently completed an order. The client upon setting the application up on their server initially had no issues, but according to them after running for "about three hours" the program begins spitting a specific stack trace error and no longer accepting commands.
The bot is built using Discord.py and uses Peewee as an ORM, using PyMySQL as the database driver. The server the client is running it on is hosted by DigitalOcean and if any information about the hardware, etc. is needed the client is able to give me that information on request. We have already attempted uninstalling and reinstalling all dependencies, as well as trying different distributions of them, but the errors persist.
This is the exact trace that the client is receiving:
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 2666, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 179, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 2875, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 750, in _execute_command
raise err.InterfaceError("(0, '')")
peewee.InterfaceError: (0, '')
The relevant portions from my database.py file, where the database connection is opened:
import discord
from peewee import *
from config import db_host, db_port, mysql_db_name, \
mysql_db_username, mysql_db_password
db_connection = MySQLDatabase(
mysql_db_name,
user = mysql_db_username,
password = mysql_db_password,
host = db_host,
port = db_port
)
def create_db_tables():
# create_tables() does safe creation by default, and will simply not create
# table if it already exists
db_connection.create_tables([User])
The relevant portions from my bot.py file, specifically the bot startup function that runs when the bot is first opened, and the lines that create and start the bot client:
client = discord.Client()
async def bot_startup():
# Check for config changes
if client.user.name != config.bot_username:
print("Username Updated To: {}".format(config.bot_username))
await client.edit_profile(username=config.bot_username)
# Start 'playing' message
await client.change_presence(
game=discord.Game( name=config.playing_message )
)
# Prepare database
database.create_db_tables()
print("Database Connected")
print("Connected Successfully")
# ...
#client.event
async def on_ready():
await bot_startup()
# ...
client.run(config.token)
According to the client, restarting the bot temporarily solves the problem and it runs fine for a few hours before the errors start up again. The bot no longer responds to any incoming commands once the errors start, and if enough errors are thrown, crashes completely.
What is typically the cause of this error, and what steps should be taken to fix whatever is causing it?
discord.py is asynchronous whilst PyMySQL is not - therefore it is blocking the discord.py runtime. Instead of PyMySQL use AIOMySQL which is non-blocking and might just solve your error.
The timing of when the error appears makes me think of a problem I also encountered when communicating to a database.
The problem could be that since the connection is opened when the bot starts up (or even when the program begins its execution) and is used sporadically, the database might close the connection, therefore any further execution will result in an error. To combat this, I create this decorator for all the methods of my Database Class
def ensures_connected(f):
def wrapper(*args):
args[0].connection.ping(reconnect=True, attempts=3, delay=2)
return f(*args)
return wrapper
to be placed above any method or function that has to communicate with the database.
The line args[0].connection.ping(reconnect=True, attempts=3, delay=2) means that we will call on self.connection (since it is the first argument passed to the method when called) the method ping, that allows to reconnect if the connection was dropped.
self.connection in my code is an object returned by the method call MySQLConnection.connect and should be equivalent to your obejct db_connection
This decorator should be placed above the method definition like the following example :
def ensures_connected(f):
def wrapper(*args):
args[0].connection.ping(reconnect=True, attempts=3, delay=2)
return f(*args)
return wrapper
class Database:
def __init__(self):
self.connection = mysql.connector.connect(
user=self.username, password=password, host=self.host, database=self.database)
# ...
#ensures_connected
def is_member_registered(self, guild: Guild, member: Member):
# ...
return
According to this comment you can use following code for reconnect automatically:
from peewee import *
from playhouse.shortcuts import ReconnectMixin
class ReconnectMySQLDatabase(ReconnectMixin, MySQLDatabase):
pass
db = ReconnectMySQLDatabase(...)
Im trying to accelerate multiple get requests to a web service using asyncio and aiohttp.
For that im fetching my data from a postgresql database using psycopg2 module .fetchmany() inside a function and constructing a dictionary of 100 records to send as lists of dictionary urls to an async function named batch() . batch by batch process.
The problem im facing in batch() function is that some requests are logging the message below although the script continues and dont fail but im not able to catch and log this exceptions to later reprocess them.
Task exception was never retrieved
future: <Task finished coro=<batch.<locals>.fetch() done, defined at C:/PythonProjects/bindings/batch_fetch.py:34> exception=ClientOSError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)>
Traceback (most recent call last):
File "C:/PythonProjects/bindings/batch_fetch.py", line 36, in fetch
async with session.get(url) as resp:
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
self._resp = await self._coro
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 497, in _request
await resp.start(conn)
File "C:\Miniconda3\lib\site-packages\aiohttp\client_reqrep.py", line 844, in start
message, payload = await self._protocol.read() # type: ignore # noqa
File "C:\Miniconda3\lib\site-packages\aiohttp\streams.py", line 588, in read
await self._waiter
aiohttp.client_exceptions.ClientOSError: [WinError 10054] An existing connection was forcibly closed by the remote host
Task exception was never retrieved
future: <Task finished coro=<batch.<locals>.fetch() done, defined at C:/PythonProjects/bindings/batch_fetch.py:34> exception=ClientConnectorError(10060, "Connect call failed ('xx.xxx.xx.xxx', 80)")>
Traceback (most recent call last):
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 924, in _wrap_create_connection
await self._loop.create_connection(*args, **kwargs))
File "C:\Miniconda3\lib\asyncio\base_events.py", line 778, in create_connection
raise exceptions[0]
File "C:\Miniconda3\lib\asyncio\base_events.py", line 765, in create_connection
yield from self.sock_connect(sock, address)
File "C:\Miniconda3\lib\asyncio\selector_events.py", line 450, in sock_connect
return (yield from fut)
File "C:\Miniconda3\lib\asyncio\selector_events.py", line 480, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
TimeoutError: [Errno 10060] Connect call failed ('xx.xxx.xx.xxx', 80)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/PythonProjects/bindings/batch_fetch.py", line 36, in fetch
async with session.get(url) as resp:
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
self._resp = await self._coro
File "C:\Miniconda3\lib\site-packages\aiohttp\client.py", line 476, in _request
timeout=real_timeout
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 522, in connect
proto = await self._create_connection(req, traces, timeout)
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 854, in _create_connection
req, traces, timeout)
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 992, in _create_direct_connection
raise last_exc
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 974, in _create_direct_connection
req=req, client_error=client_error)
File "C:\Miniconda3\lib\site-packages\aiohttp\connector.py", line 931, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host cms-uat.cme.in.here.com:80 ssl:None [Connect call failed ('xx.xxx.xx.xxx', 80)]
Im just entering into asyncio world as you can depict from my code, so all the advises on the full code approach for this scenario are very welcomme.
Thank you
full code below.
import psycopg2.extras
import asyncio
import json
from aiohttp import ClientSession
from aiohttp import TCPConnector
base_url = 'http://url-example/{}'
def query_db():
urls = []
# connection to postgres table , fetch data.
conn = psycopg2.connect("dbname='pac' user='user' host='db'")
cursor = conn.cursor('psycopg2 request', cursor_factory=psycopg2.extras.NamedTupleCursor)
sql = "select gid, paid from table"
cursor.execute(sql)
while True:
rec = cursor.fetchmany(100)
for item in rec:
record = {"gid": item.gid, "url": base_url.format(item.paid)}
urls.append(record.get('url'))
if not rec:
break
# send batch for async batch request
batch(urls)
# empty list of urls for new async batch request
urls = []
def batch(urls):
async def fetch(url):
async with ClientSession() as session:
async with session.get(url) as resp:
if resp.status == 200:
response = await resp.json()
# parse the url to fetch the point address id.
paid = str(resp.request_info.url).split('/')[4].split('?')[0]
# build the dictionary with pa id and full response.
resp_dict = {'paid': paid, 'response': response}
with open('sucessful.json', 'a') as json_file:
json.dump(resp_dict, json_file)
json_file.write("\n")
elif resp.status is None:
print(resp.status)
elif resp.status != 200:
print(resp.status)
response = await resp.json()
# parse the url to fetch the paid.
paid = str(resp.request_info.url).split('/')[4].split('?')[0]
# build the dictionary with paid and full response.
resp_dict = {'paid': paid, 'response': response}
with open('failed.json', 'a') as json_file:
json.dump(resp_dict, json_file)
json_file.write("\n")
loop = asyncio.get_event_loop()
tasks = []
for url in urls:
task = asyncio.ensure_future(fetch(url))
tasks.append(task)
try:
loop.run_until_complete(asyncio.wait(tasks))
except Exception:
print("exception consumed")
if __name__ == "__main__":
query_db()
Task exception was never retrieved
You see this warning when you've created some task, it finished with exception, but you never explicitly retrieved (awaited) for its result. Here's related doc section.
I bet in your case problem is with the line
loop.run_until_complete(asyncio.wait(tasks))
asyncio.wait() by default just waits when all tasks are done. It doesn't distinguish tasks finished normally or with exception, it just blocks until everything finished. In this case it's you job to retrieve exceptions from finished tasks and following part won't help you with this since asyncio.wait() will never raise an error:
try:
loop.run_until_complete(asyncio.wait(tasks))
except Exception:
print('...') # You will probably NEVER see this message
If you want to catch error as soon as it happened in one of tasks I can advice you to use asyncio.gather(). By default it will raise first happened exception. Note however that it is you job to cancel pending tasks if you want their graceful shutdown.
I'm getting an aiohttp client_exception.ServerDisconnectedError whenever I do more than ~200 requests to an API I'm hitting using asyncio & aiohttp. It doesn't seem to be my code because it works consistently with smaller number of requests, but fails on any larger number. Trying to understand if this error is related to aiohttp, or my code, or with the API endpoint itself? There doesn't seem to be much info online around this.
Traceback (most recent call last):
File "C:/usr/PycharmProjects/api_framework/api_framework.py", line 27, in <module>
stuff = abc.do_stuff_2()
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 72, in do_stuff
self.queue_manager(self.do_stuff(json_data))
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 115, in queue_manager
loop.run_until_complete(future)
File "C:\Python36x64\lib\asyncio\base_events.py", line 466, in run_until_complete
return future.result()
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 96, in do_stuff
result = await asyncio.gather(*tasks)
File "C:\usr\PycharmProjects\api_framework\api\abc\abc.py", line 140, in async_post
async with session.post(self.api_attr.api_endpoint + resource, headers=self.headers, data=data) as response:
File "C:\Python36x64\lib\site-packages\aiohttp\client.py", line 843, in __aenter__
self._resp = await self._coro
File "C:\Python36x64\lib\site-packages\aiohttp\client.py", line 387, in _request
await resp.start(conn)
File "C:\Python36x64\lib\site-packages\aiohttp\client_reqrep.py", line 748, in start
message, payload = await self._protocol.read()
File "C:\Python36x64\lib\site-packages\aiohttp\streams.py", line 533, in read
await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None
here's some of the code to generate the async requests:
def some_other_method(self):
self.queue_manager(self.do_stuff(all_the_tasks))
def queue_manager(self, method):
print('starting event queue')
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(method)
loop.run_until_complete(future)
loop.close()
async def async_post(self, resource, session, data):
async with session.post(self.api_attr.api_endpoint + resource, headers=self.headers, data=data) as response:
resp = await response.read()
return resp
async def do_stuff(self, data):
print('queueing tasks')
tasks = []
async with aiohttp.ClientSession() as session:
for row in data:
task = asyncio.ensure_future(self.async_post('my_api_endpoint', session, row))
tasks.append(task)
result = await asyncio.gather(*tasks)
self.load_results(result)
Once the tasks have completed, self.load_results() method just parses the json and updates the DB.
It is most likely caused by the configuration of the HTTP server. There are at least two possible reasons for the ServerDisconnectedError:
The server could limit the number of parallel TCP connections that can be made from a single IP address. By default, aiohttp already limits the number of parallel connections to 100. You can try reducing the limit and see if it solve the issue. To do so, you can create a custom TCPConnector with a different limit value and pass it to the ClientSession:
connector = aiohttp.TCPConnector(limit=50)
async with aiohttp.ClientSession(connector=connector) as session:
# Use your session as usual here
The server could limit the duration of a TCP connection. By default, aiohttp uses HTTP keep-alive so that the same TCP connection can be used for multiple requests. This improves performances since a new TCP connection does not have to be made for each request. However, some servers limit the duration of a TCP connection, and if you use the same TCP connection for many requests, the server can close it before you are done with it. You can disable HTTP keep-alive as a workaround. To do so, you can create a custom TCPConnector with the parameter force_close set to True, and pass it to the ClientSession:
connector = aiohttp.TCPConnector(force_close=True)
async with aiohttp.ClientSession(connector=connector) as session:
# Use your session as usual here
I had the same issue and disabling HTTP keep-alive was the solution for me. Hope this helps.
This is most likely the server's API not being happy with multiple requests being done asynchronously.
You can limit the amount of concurrent calls with asyncio's semaphores.
In your case I would use it within a context manager as:
async def do_stuff(self, data):
print('queueing tasks')
tasks = []
semaphore = asyncio.Semaphore(200)
async with semaphore:
async with aiohttp.ClientSession() as session:
for row in data:
task = asyncio.ensure_future(self.async_post('my_api_endpoint', session, row))
tasks.append(task)
result = await asyncio.gather(*tasks)
self.load_results(result)
I think it's quite possible the other answers are correct, but there's also one more possibility - it seems aiohttp has at least one currently [June 2021] unfixed race condition in it's streams code:
https://github.com/aio-libs/aiohttp/issues/4581
I see the same issue in my project, and it's rare enough (and server disconnect isn't the only symptom, I sometimes get "payload not complete") it feels more like a race condition. I also saw issues like aiohttp putting a packet of data from one response into a different response.
In the end, I switched to https://www.python-httpx.org - this decreased the number of problems, and let me to eventually that some of the 'payload not complete' error was probably related to timeout for sending a large binary response on the server that was occasionally triggering. In general I found httpx to be more reliable, and it's really good that you can use the same package/APIs to support both sync and async.
I have a python server setup with aiohttp that is accepting files POST'd to a specific endpoint. I only want to accept a json body, or gzip'd json files. My code is as follows:
class Uploader(View):
async def post(self):
if not self.request.can_read_body:
return json_response({'message': 'Cannot read body'}, status=400)
elif self.request.content_type != 'application/json' and self.request.content_type != 'multipart/form-data':
return json_response({'message': 'Incorrect data type sent to the server'}, status=400)
try:
json_body = await self.request.json()
# Other bits of code using the json body
except RequestPayloadError as e:
# Internal logging here
return json_response({'message': 'Unable to read payload'}, status=400)
# Other code for handling ValidationError, JSONDecodeError, Exception
return json_response({'message': 'File successfully uploaded'}, status=201)
When I test this by uploading something that isn't json or gzip'd json, the RequestPayloadError exception is correctly being hit, the internal logging is being done as expected, and the client is being returned the expected response. However, I'm also seeing the following unhandled exception:
Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 428, in start
await payload.readany()
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 325, in readany
raise self._exception
File "/web_api/app/views/resources/Uploader.py", line 49, in post
json_body = await self.request.json()
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_request.py", line 512, in json
body = await self.text()
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_request.py", line 506, in text
bytes_body = await self.read()
File "/usr/local/lib/python3.6/site-packages/aiohttp/web_request.py", line 494, in read
chunk = await self._payload.readany()
File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 325, in readany
raise self._exception
aiohttp.web_protocol.RequestPayloadError: 400, message='Can not decode content-encoding: gzip'
How am I supposed to handle this currently unhandled exception given that it doesn't seem to be originating in my code, and I'm already handling the one that I'm expecting? Can I suppress aiohttp exceptions somehow?
EDIT: I'm using version 3.1.1 of aiohttp
Can not decode content-encoding: gzip points on the problem source.
Your peer sends data with Content-Encoding: gzip HTTP header but actually the data is not gzip compressed (other compressor is used or no compressor at all).
As result aiohttp fails on decompressing such data with RequestPayloadError exception.