aiohttp client not handling cookies correctly - python-3.x

I am trying to use cookies with aiohttp.ClientSession but it doesn't seem to be handling the cookies across multiple requests. That or I am not using it correctly.
I have a simple server which saves a cookie on the client. This works fine when accessed from the browser.
Server code: (modified from here)
import asyncio
import time
from aiohttp import web
from aiohttp_session import get_session, setup
from aiohttp_session.cookie_storage import EncryptedCookieStorage
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
async def handler(request):
session = await get_session(request)
last_visit = session.get('last_visit', 'Never')
if last_visit == 'Never':
message = "Welcome, I don't think you've visited here before."
else:
message = 'Welcome back, last visited: {} secs ago'.format(time.time() -
last_visit)
session['last_visit'] = time.time()
return web.Response(body=message.encode('utf-8'))
async def init(loop):
app = web.Application()
setup(app,
EncryptedCookieStorage(b'Thirty two length bytes key.'))
app.router.add_route('GET', '/', handler)
srv = await loop.create_server(
app.make_handler(), '0.0.0.0', 8080)
return srv
loop = asyncio.get_event_loop()
loop.run_until_complete(init(loop))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
When accessed from a browser I get:
Welcome back, last visited: 1176.336282491684 secs ago
Now, I am trying to mimic this behavior within a python client, which is currently not working. I am aware that I am not persisting the cookie to file but I am trying multiple requests within the same session so this should work right?
The cookie within the client session doesn't seem to be persisting across multiple requests.
The client code:
import aiohttp
import asyncio
jar = aiohttp.CookieJar(unsafe=True)
async def blah():
async with aiohttp.ClientSession(cookie_jar=jar) as session:
for i in range(2):
async with session.get('http://localhost:8080') as resp:
print(resp.status)
print(await resp.text())
print(session.cookies)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(blah())
When I run the client code I get:
> python client.py
200
Welcome, I don't think you've visited here before.
Set-Cookie: AIOHTTP_SESSION="gAAAAABXpF_SYEhcMpT-1Q_g6V-SwDuWh-XZipIMre5GgYvJ513ao4BaVgN4kcQM4b91umGCgWuoCEe5RCpZ5ryA30rchUAaojH3B35OL9LjH-kLJ3Md0PhfaylWl3_ct5K2aSwdBdMU_mACaHeTV0FA7yiT0DrMI_n9ct3D-jRTYCsKc5xLI2I="; Domain=localhost; HttpOnly; Path=/
200
Welcome, I don't think you've visited here before.
Set-Cookie: AIOHTTP_SESSION="gAAAAABXpF_SiCl07HDerId98tjI6hTrWOcEmCRVELV3F_sif3XkzgjS_hfwlkMK4HpoWbRrNoxJZpERPKkxRJi9AOpUeleWTkfkjXUcNk13OX5GCOZDSLbSbTkqdoiiAYfAsQ3CNHZWGWd2xzlha_E54ig3Jq1sQsAXV6rgcrqxh0xMGYWfseM="; Domain=localhost; HttpOnly; Path=/

Related

Why is there so much overhead when using https instead of http with aiohttp

I am transforming a stript from sync requests to the async aiohttp, because I would like to make really fast requests to multiple servers. One thing I am struggling with now is that the response time of HTTPS is much higher with
aiohttp than with requests.
Below is a code example that is as short as possible where I make a request to HTTP or HTTPS and for the requests library this doesn't make that big of a difference, but with aiohttp it does. The two used methods to make the request are:
requests.request('GET', url) and
aiohttp.request('GET', url)
So what I don't understand is: Why is aiohttp so much worse in handling https requests?
import time
import aiohttp
import asyncio
import requests
URL = 'https://google.com' # 'https://google.com' # takes asyncio ~x2 as long
time1 = time.time()
responseAsync = []
async def fetch(session):
async with session.request(method='GET', url=URL) as response:
return await response.read()
async def run():
uris = ['list of uris, not used now']
tasks = []
async with aiohttp.ClientSession() as session:
for uri in uris:
task = asyncio.ensure_future(fetch(session))
tasks.append(task)
responseAsync.append(await asyncio.gather(*tasks))
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run())
loop.run_until_complete(future)
time2 = time.time()
session = requests.Session()
responseSync = [session.request('GET', URL)] # and some extra more requests..
time3 = time.time()
totaltimeAsyncio = (time2 - time1) * 1000
totaltimeRequests = (time3 - time2) * 1000
print('Response time async request:', totaltimeAsyncio)
print('Response time sync request:', totaltimeRequests)
output HTTP:
Response time async request: 130.8588981628418
Response time sync request: 107.31077194213867
output HTTPS:
Response time async request: 571.195125579834
Response time sync request: 167.56391525268555

Passing an external queue to flask route function?

I'm trying to modify flask request callback so it can communicate with other code while executing the callback. The example explains it better:
from flask import Flask, request
from queue import Queue
flask_input_queue = Queue()
flask_output_queue = Queue()
app = Flask(__name__)
#app.route("/voice", methods=['GET', 'POST'])
def voice():
# The receiver on the other end gets notified we got a request
flask_output_queue.put(str(request))
# This blocks until the external party responds with something
response = flask_input_queue.get()
# But how do the queues end up in the function scope to begin with?
return response
app.run(debug=True)
Here the external code would have a channel using the queues into the web server. This allows me to completely abstract the concept of the web server on the other part of the code.
However for that, I need to be able to pass information to the callback method in ways other that just URLs. Frankly it doesn't have to be a queue other IPC mechanisms will also work ok but they all rely on having a way to pass data into the callback.
Is there a way to do that in flask?
The _URLCallbackClass in combination with add_url_rule is used instead of the decorator. That _URLCallbackClass gets the queue as instance attributes. Given that the actual callback function is the method of _URLCallbackClass, we smuggled the queues into the callback function.
The rest of the complexity just arises from providing a working example.
logging.basicConfig(format='[Thread: %(threadName)s-%(thread)d] %(message)s', level=logging.INFO) [0/0]
logger = logging.getLogger(__name__)
class ControllableServer(threading.Thread):
class _URLCallbackClass():
def __init__(self, input_queue, output_queue):
self.input_queue = input_queue
self.output_queue = output_queue
def url_callback(self):
self.output_queue.put("[URL callback] I just got called")
response_from_the_queue = self.input_queue.get()
return Response(response_from_the_queue, 200)
def __init__(self, input_queue, output_queue):
super().__init__(daemon=True)
self.input_queue = input_queue
self.output_queue = output_queue
self._flask = Flask(__name__)
def run(self):
callback_class = ControllableServer._URLCallbackClass(self.input_queue, self.output_queue)
self._flask.add_url_rule('/endpoint', 'url_callback', callback_class.url_callback)
logger.info(f"Starting flask")
self._flask.run()
def call_URL_in_separate_thread(url):
def call_URL(url):
logger.info(f"Calling {url}")
response = requests.get(url)
logger.info(f"Got response: {response.text}")
return response.text
url_caller_thread = threading.Thread(target=call_URL, args=(url,))
url_caller_thread.start()
if __name__ == "__main__":
flask_input_queue = Queue()
flask_output_queue = Queue()
controllable_server = ControllableServer(flask_input_queue, flask_output_queue)
controllable_server.start()
call_URL_in_separate_thread("http://127.0.0.1:5000/endpoint")
message_from_within_the_callback = flask_output_queue.get()
logger.info(f"Got message from queue: {message_from_within_the_callback}")
message_to_the_callback = "I come from the outside !###$#"
flask_input_queue.put(message_to_the_callback)
logger.info(f"Sending message to queue: {message_to_the_callback}")
Output:
[Thread: Thread-1-140465413375744] Starting flask
[Thread: Thread-2-140465404983040] Calling http://127.0.0.1:5000/endpoint
* Serving Flask app "basic_flask_passing_variable" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
[Thread: Thread-1-140465413375744] * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
[Thread: MainThread-140465450415936] Got message from queue: [URL callback] I just got called
[Thread: MainThread-140465450415936] Sending message to queue: I come from the outside !###$#
[Thread: Thread-3-140465396041472] 127.0.0.1 - - [03/Mar/2020 18:33:32] "GET /endpoint HTTP/1.1" 200 -
[Thread: Thread-2-140465404983040] Got response: I come from the outside !###$#

Sanic web framework performance

I have a question to ask about sanic / asyncpg performance.
During a testing a weird things kept happening (Maybe it is by design).
First let me explain testing procedure. It is simple.
I use locust to push the server as much as possible by setting max user count.
The testing script is:
from locust import HttpLocust, TaskSet, task, between
class UserActions(TaskSet):
#task(1)
def test_point_1(self):
self.client.get(
'/json_1',
headers={'Content-Type': 'application/json'}
)
#task(2)
def test_point_2(self):
self.client.get(
'/json_2',
headers={'Content-Type': 'application/json'}
)
class ApplicationUser(HttpLocust):
task_set = UserActions
wait_time = between(0, 0)
It is used to test the following code. Notice asyncpg is calling potgresql sleep function to simulate a load:
import asyncio
import uvloop
from asyncpg import create_pool
from sanic import Sanic, response
from sanic.log import logger
import aiotask_context as context
app = Sanic(__name__)
DATABASE = {
'type': 'postgresql',
'host': '127.0.0.1',
'user': 'test_user',
'port': '5432',
'password': 'test_password',
'database': 'test_database'
}
conn_uri = '{0}://{1}:{2}#{3}:{4}/{5}'.format(
'postgres',
DATABASE['user'], DATABASE['password'], DATABASE['host'],
DATABASE['port'], DATABASE['database'])
#app.route("/json_1")
async def handler_json_1(request):
async with request.app.pg.acquire() as connection:
await connection.fetchrow('SELECT pg_sleep(0.85);')
return response.json({"foo": "bar"})
#app.route("/json_2")
async def handler_json_2(request):
async with request.app.pg.acquire() as connection:
await connection.fetchrow('SELECT pg_sleep(0.2);')
return response.json({"foo": "bar"})
#app.listener("before_server_start")
async def listener_before_server_start(*args, **kwargs):
try:
pg_pool = await create_pool(
conn_uri, min_size=2, max_size=10,
server_settings={'application_name': 'test_backend'})
app.pg = pg_pool
except Exception as bss_error:
logger.error('before_server_start_test erred with :{}'.format(bss_error))
app.pg = None
#app.listener("after_server_start")
async def listener_after_server_start(*args, **kwargs):
# print("after_server_start")
pass
#app.listener("before_server_stop")
async def listener_before_server_stop(*args, **kwargs):
# print("before_server_stop")
pass
#app.listener("after_server_stop")
async def listener_after_server_stop(*args, **kwargs):
# print("after_server_stop")
pass
if __name__ == '__main__':
asyncio.set_event_loop(uvloop.new_event_loop())
server = app.create_server(host="0.0.0.0", port=8282, return_asyncio_server=True)
loop = asyncio.get_event_loop()
loop.set_task_factory(context.task_factory)
task = asyncio.ensure_future(server)
try:
loop.run_forever()
except Exception as lerr:
logger.error('Loop run error: {}'.format(lerr))
loop.stop()
The issue is, after a random amount of time server becomes unresponsive
(Does not return 503 or any other code) for a cca. 60 seconds.
Also process hangs (I can see it with ps aux and CTRL+C cannot kill it.)
That might be problematic because for one it is hard to detect and it is difficult to determine a rate at which we can send request to the server.
Could that be an issue with the configuration (sanic/asyncpg)?
Could setting nginx / sanic request timeout be the only option to circumvent this problem ?
Your aiopg pool is limited to 10 connections. So 10 requests at a time max, each takes 0.2 sec, your max possible load would be 1 sec / 0.2 sec * 10 pool size = 50 RPS. After that all incoming requests would just wait for a connection and queue of requests to serve would grow much faster than your ability to serve and your server would become irresponsive.

aiohttp test with several servers

Trying to do a test that communicates with several instances of a web-server (which also communicates between them). But the second one seems to override the first however I try. Any suggestions of how to solve this.
So far I have this:
import os
from aiohttp.test_utils import TestClient, TestServer, loop_context
import pytest
from http import HTTPStatus as hs
from mycode import main
#pytest.fixture
def cli(loop):
app = main(["-t", "--location", "global", "-g"])
srv = TestServer(app, port=40080)
client = TestClient(srv, loop=loop)
loop.run_until_complete(client.start_server())
return client
#pytest.fixture
def cli_edge(loop):
app = main(["-t", "--location", "edge", "-j", "http://127.0.0.1:40080"])
srv = TestServer(app, port=40081)
client = TestClient(srv, loop=loop)
loop.run_until_complete(client.start_server())
return client
async def test_edge_namespace(cli, cli_edge):
resp = await cli.put('/do/something', json={})
assert resp.status in [hs.OK, hs.CREATED, hs.NO_CONTENT]
resp = await cli_edge.get('/do/something')
assert resp.status in [hs.OK, hs.CREATED, hs.NO_CONTENT]
The above call to cli.put goes to the server intended for cli_edge. I will have several more tests that should communicate with the servers.
Using Python 3.7 and pytest with asyncio and aiohttp extensions.
The suggested code works, the error was elsewhere in the server implementation.
You can add:
def fin():
loop.run_until_complete(srv.close())
loop.run_until_complete(client.close())
request.addfinalizer(fin)
and the request param in the pytest fixtures to close connections nicely.

Get cookie using aiohttp

I am trying to get cookies from the browser using aiohttp. From the docs and googling I have only found articles about setting cookies in aiohttp.
In flask I would get the cookies as simply as
cookie = request.cookies.get('name_of_cookie')
# do something with cookie
Is there a simple way to fetch the cookie from browser using aiohttp?
Is there a simple way to fetch the cookie from the browser using aiohttp?
Not sure about whether this is simple but there is a way:
import asyncio
import aiohttp
async def main():
urls = [
'http://httpbin.org/cookies/set?test=ok',
]
for url in urls:
async with aiohttp.ClientSession(cookie_jar=aiohttp.CookieJar()) as s:
async with s.get(url) as r:
print('JSON', await r.json())
cookies = s.cookie_jar.filter_cookies('http://httpbin.org')
for key, cookie in cookies.items():
print('Key: "%s", Value: "%s"' % (cookie.key, cookie.value))
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
The program generates the following output:
JSON: {'cookies': {'test': 'ok'}}
Key: "test", Value: "ok"
Example adapted from https://aiohttp.readthedocs.io/en/stable/client_advanced.html#custom-cookies + https://docs.aiohttp.org/en/stable/client_advanced.html#cookie-jar
Now if you want to do a request using a previously set cookie:
import asyncio
import aiohttp
url = 'http://example.com'
# Filtering for the cookie, saving it into a varibale
async with aiohttp.ClientSession(cookie_jar=aiohttp.CookieJar()) as s:
cookies = s.cookie_jar.filter_cookies('http://example.com')
for key, cookie in cookies.items():
if key == 'test':
cookie_value = cookie.value
# Using the cookie value to do anything you want:
# e.g. sending a weird request containing the cookie in the header instead.
headers = {"Authorization": "Basic f'{cookie_value}'"}
async with s.get(url, headers=headers) as r:
print(await r.json())
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
For testing urls containing a host part made up by an IP address use aiohttp.ClientSession(cookie_jar=aiohttp.CookieJar(unsafe=True)), according to https://github.com/aio-libs/aiohttp/issues/1183#issuecomment-247788489
Yes, the cookies are stored in request.cookies as a dict, just like in flask, so request.cookies.get('name_of_cookie') works the same.
In the examples section of the aiohttp repository there is a file, web_cookies.py that shows how to retrieve, set, and delete a cookie. Here's the section from that script that reads the cookies and returns it to the template as a preformatted string:
from pprint import pformat
from aiohttp import web
tmpl = '''\
<html>
<body>
Login<br/>
Logout<br/>
<pre>{}</pre>
</body>
</html>'''
async def root(request):
resp = web.Response(content_type='text/html')
resp.text = tmpl.format(pformat(request.cookies))
return resp
You can get the cookie value, domain, path etc, without having to loop thru all cookies.
s.cookie_jar._cookies
gives you all the cookies in a defaultdict with the domains as keys and their respective cookies as values. aiohttp uses SimpleCookie
So, to get the value of a cookie
s.cookie_jar._cookies.get("https://httpbin.org")["cookie_name"].value
for domain, path:
s.cookie_jar._cookies.get("https://httpbin.org")["cookie_name"]["domain"]
s.cookie_jar._cookies.get("https://httpbin.org")["cookie_name"]["path"]
more info can be found here: https://docs.python.org/3/library/http.cookies.html

Resources