Sanic web framework performance - python-3.x

Question

I have a question to ask about sanic / asyncpg performance.
During a testing a weird things kept happening (Maybe it is by design).
First let me explain testing procedure. It is simple.
I use locust to push the server as much as possible by setting max user count.
The testing script is:
from locust import HttpLocust, TaskSet, task, between
class UserActions(TaskSet):
#task(1)
def test_point_1(self):
self.client.get(
'/json_1',
headers={'Content-Type': 'application/json'}
)
#task(2)
def test_point_2(self):
self.client.get(
'/json_2',
headers={'Content-Type': 'application/json'}
)
class ApplicationUser(HttpLocust):
task_set = UserActions
wait_time = between(0, 0)
It is used to test the following code. Notice asyncpg is calling potgresql sleep function to simulate a load:
import asyncio
import uvloop
from asyncpg import create_pool
from sanic import Sanic, response
from sanic.log import logger
import aiotask_context as context
app = Sanic(__name__)
DATABASE = {
'type': 'postgresql',
'host': '127.0.0.1',
'user': 'test_user',
'port': '5432',
'password': 'test_password',
'database': 'test_database'
}
conn_uri = '{0}://{1}:{2}#{3}:{4}/{5}'.format(
'postgres',
DATABASE['user'], DATABASE['password'], DATABASE['host'],
DATABASE['port'], DATABASE['database'])
#app.route("/json_1")
async def handler_json_1(request):
async with request.app.pg.acquire() as connection:
await connection.fetchrow('SELECT pg_sleep(0.85);')
return response.json({"foo": "bar"})
#app.route("/json_2")
async def handler_json_2(request):
async with request.app.pg.acquire() as connection:
await connection.fetchrow('SELECT pg_sleep(0.2);')
return response.json({"foo": "bar"})
#app.listener("before_server_start")
async def listener_before_server_start(*args, **kwargs):
try:
pg_pool = await create_pool(
conn_uri, min_size=2, max_size=10,
server_settings={'application_name': 'test_backend'})
app.pg = pg_pool
except Exception as bss_error:
logger.error('before_server_start_test erred with :{}'.format(bss_error))
app.pg = None
#app.listener("after_server_start")
async def listener_after_server_start(*args, **kwargs):
# print("after_server_start")
pass
#app.listener("before_server_stop")
async def listener_before_server_stop(*args, **kwargs):
# print("before_server_stop")
pass
#app.listener("after_server_stop")
async def listener_after_server_stop(*args, **kwargs):
# print("after_server_stop")
pass
if __name__ == '__main__':
asyncio.set_event_loop(uvloop.new_event_loop())
server = app.create_server(host="0.0.0.0", port=8282, return_asyncio_server=True)
loop = asyncio.get_event_loop()
loop.set_task_factory(context.task_factory)
task = asyncio.ensure_future(server)
try:
loop.run_forever()
except Exception as lerr:
logger.error('Loop run error: {}'.format(lerr))
loop.stop()
The issue is, after a random amount of time server becomes unresponsive
(Does not return 503 or any other code) for a cca. 60 seconds.
Also process hangs (I can see it with ps aux and CTRL+C cannot kill it.)
That might be problematic because for one it is hard to detect and it is difficult to determine a rate at which we can send request to the server.
Could that be an issue with the configuration (sanic/asyncpg)?
Could setting nginx / sanic request timeout be the only option to circumvent this problem ?

Answer 1

Your aiopg pool is limited to 10 connections. So 10 requests at a time max, each takes 0.2 sec, your max possible load would be 1 sec / 0.2 sec * 10 pool size = 50 RPS. After that all incoming requests would just wait for a connection and queue of requests to serve would grow much faster than your ability to serve and your server would become irresponsive.