I'm testing my database query middleware (Django docs here) on a sample django app with a Postgres db. The app is the cookiecutter boilerplate. My goal with the middleware is simply to log the user ID for all database queries. Using Python3.9 and Django3.2.13. My middleware code is below:
# Middleware code
import logging
import django
from django.db import connection
django_version = django.get_version()
logger = logging.getLogger(__name__)
class Logger:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
with connection.execute_wrapper(QueryWrapper(request)):
return self.get_response(request)
class QueryWrapper:
def __init__(self, request):
self.request = request
def __call__(self, execute, sql, params, many, context):
# print(self.request.user)
return execute(sql, params, many, context)
If print(self.request.user.id) is commented out, everything works fine. However, I've found that uncommenting it, or any type of interaction with the user field in the self.request object, causes a Recursion Error:
RecursionError at /about/
maximum recursion depth exceeded
Request Method: GET
Request URL: http://127.0.0.1:8000/about/
Django Version: 3.2.13
Exception Type: RecursionError
Exception Value:
maximum recursion depth exceeded
Exception Location: /opt/homebrew/lib/python3.9/site-packages/django/db/models/sql/query.py, line 192, in __init__
Python Executable: /opt/homebrew/opt/python#3.9/bin/python3.9
Python Version: 3.9.13
In the error page, that is followed by many repetitions of the below error:
During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
/opt/homebrew/lib/python3.9/site-packages/django/contrib/sessions/backends/base.py, line 233, in _get_session
return self._session_cache …
During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
/opt/homebrew/lib/python3.9/site-packages/django/contrib/sessions/backends/base.py, line 233, in _get_session
return self._session_cache …
From consulting other SO posts, it seems accessing the user field should work fine. I've checked that the django_session table exists, and my middleware is also located at the very bottom of my middlewares (that include "django.contrib.sessions.middleware.SessionMiddleware" and "django.contrib.auth.middleware.AuthenticationMiddleware")
What's wrong here?
The following things happen when you write self.request.user:
The request is checked for an attribute _cached_user, if present the cached user is returned if not auth.get_user is called with the request.
Using the session key, Django checks the session to get the authentication backend used for the current user. Here if you are using database based sessions a database query is fired.
Using the above authentication backend Django makes a database query to get the current user using their ID.
As noted from the above points, unless there is a cache hit this process is going to cause a database query.
Using database instrumentation you have installed a wrapper around database queries, the problem is that this wrapper itself is trying to make more queries (trying to get the current user), causing it to call itself. One solution would be to get the current user before installing your wrapper function:
class Logger:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
_user_authenticated = request.user.is_authenticated # Making sure user is fetched (Lazy object)
with connection.execute_wrapper(QueryWrapper(request)):
return self.get_response(request)
class QueryWrapper:
def __init__(self, request):
self.request = request
def __call__(self, execute, sql, params, many, context):
print(self.request.user)
return execute(sql, params, many, context)
I have a rather contrived code here :
backend_data = {
"admins": ["Leo", "Martin", "Thomas", "Katrin"],
"members": [
"Leo",
"Martin",
"Thomas",
"Katrin",
"Subhayan",
"Clemens",
"Thoralf"
],
"juniors": ["Orianne", "Antonia", "Sarah"]
}
class Backend:
def __init__(self, data):
self.backend_data = data
def get_all_admins(self):
return self.backend_data.get("admins")
def get_all_members(self):
return self.backend_data.get("members")
def get_all_juniors(self):
return self.backend_data.get("juniors")
class BackendAdaptor:
# Does some conversion and validation
def __init__(self, backend):
self.backend = backend
def get_all_admins(self):
return (admin for admin in self.backend.get_all_admins())
def get_all_members(self):
return (member for member in self.backend.get_all_members() if member not in self.backend.get_all_admins())
def get_all_juniors(self):
return (junior for junior in self.backend.get_all_juniors())
if __name__ == "__main__":
backend = Backend(data=backend_data)
adaptor = BackendAdaptor(backend=backend)
print(f"All admins are : {list(adaptor.get_all_admins())}")
print(f"All members are : {list(adaptor.get_all_members())}")
print(f"All juniors are : {list(adaptor.get_all_juniors())}")
So the BackendAdaptor class basically would be used to do some validation and conversion of the data that we get from the Backend .
The client should only be asked to interact with the API of the BackendAdaptor which is exactly similar to that of Backend . The adaptor class sits in middle and gets data from Backend does some validation if required and the gives back the data to client.
The issue is that the validation on the data that is getting returned from the Backend is not done for every method(For ex: there is validation done on get_all_members but not on get_all_admins and also not on get_all_juniors). The method just gives back a generator on whatever data it gets from Backend.
As is the case now i still have to implement a one liner methods for them .
Is there a way in Python to avoid this ? I am thinking in lines of magic methods like __getattribute__ ? But i have no idea on how to do this for methods.
So the best case scenario is this:
I implement the methods for which i know that i have to do some validation on Backend data
For the rest of the methods it is automatically delegated to Backend and then i just return a generator from what i get back
Any help would be greatly appreciated.
You can implement __getattr__. It is only called if a non-existing attribute is accessed. This will return some generic function with the desired functionality.
class BackendAdaptor:
def __init__(self, backend):
self.backend = backend
def __getattr__(self, name):
if not hasattr(self.backend, name):
raise AttributeError(f"'{name}' not in backend.")
return lambda: (i for i in getattr(self.backend, name)())
def get_all_members(self):
return (member for member in self.backend.get_all_members() if member not in self.backend.get_all_admins())
How do I create a object that I can invoke to mimic the following api call and response. I am aware of the mock library but the use case prohibits me from using it.
response = client.users.create(email='test#gmail.com', phone=123)
outcome = response.ok
My current solution below works however I feel like there is a more pythonic and generic way to do this so I can mimic other calls without having to rewrite different inner classes
class Client:
ok = True
class users:
class create():
ok = True
def __init__(self, email, phone):
pass
Input
client = Client()
response = client.users.create(email='test#gmail.com', phone=123)
response.ok
Output
True
I've a REST API running on Python 3.7 + Tornado 5, with postgresql as database, using aiopg with SQLAlchemy core (via the aiopg.sa binding). For the unit tests, I use py.test with pytest-tornado.
All the tests go ok as soon as no query to the database is involved, where I'd get this:
Runtime error: Task cb=[IOLoop.add_future..() at venv/lib/python3.7/site-packages/tornado/ioloop.py:719]> got Future attached to a different loop
The same code works fine out of the tests, I'm capable of handling 100s of requests so far.
This is part of an #auth decorator which will check the Authorization header for a JWT token, decode it and get the user's data and attach it to the request; this is the part for the query:
partner_id = payload['partner_id']
provided_scopes = payload.get("scope", [])
for scope in scopes:
if scope not in provided_scopes:
logger.error(
'Authentication failed, scopes are not compliant - '
'required: {} - '
'provided: {}'.format(scopes, provided_scopes)
)
raise ForbiddenException(
"insufficient permissions or wrong user."
)
db = self.settings['db']
partner = await Partner.get(db, username=partner_id)
# The user is authenticated at this stage, let's add
# the user info to the request so it can be used
if not partner:
raise UnauthorizedException('Unknown user from token')
p = Partner(**partner)
setattr(self.request, "partner_id", p.uuid)
setattr(self.request, "partner", p)
The .get() async method from Partner comes from the Base class for all models in the app. This is the .get method implementation:
#classmethod
async def get(cls, db, order=None, limit=None, offset=None, **kwargs):
"""
Get one instance that will match the criteria
:param db:
:param order:
:param limit:
:param offset:
:param kwargs:
:return:
"""
if len(kwargs) == 0:
return None
if not hasattr(cls, '__tablename__'):
raise InvalidModelException()
tbl = cls.__table__
instance = None
clause = cls.get_clause(**kwargs)
query = (tbl.select().where(text(clause)))
if order:
query = query.order_by(text(order))
if limit:
query = query.limit(limit)
if offset:
query = query.offset(offset)
logger.info(f'GET query executing:\n{query}')
try:
async with db.acquire() as conn:
async with conn.execute(query) as rows:
instance = await rows.first()
except DataError as de:
[...]
return instance
The .get() method above will either return a model instance (row representation) or None.
It uses the db.acquire() context manager, as described in aiopg's doc here: https://aiopg.readthedocs.io/en/stable/sa.html.
As described in this same doc, the sa.create_engine() method returns a connection pool, so the db.acquire() just uses one connection from the pool. I'm sharing this pool to every request in Tornado, they use it to perform the queries when they need it.
So this is the fixture I've set up in my conftest.py:
#pytest.fixture
async def db():
dbe = await setup_db()
return dbe
#pytest.fixture
def app(db, event_loop):
"""
Returns a valid testing Tornado Application instance.
:return:
"""
app = make_app(db)
settings.JWT_SECRET = 'its_secret_one'
return app
I can't find an explanation of why this is happening; Tornado's doc and source makes it clear that asyncIO event loop is used as default, and by debugging it I can see the event loop is indeed the same one, but for some reason it seems to get closed or stopped abruptly.
This is one test that fails:
#pytest.mark.gen_test(timeout=2)
def test_score_returns_204_empty(app, http_server, http_client, base_url):
score_url = '/'.join([base_url, URL_PREFIX, 'score'])
token = create_token('test', scopes=['score:get'])
headers = {
'Authorization': f'Bearer {token}',
'Accept': 'application/json',
}
response = yield http_client.fetch(score_url, headers=headers, raise_error=False)
assert response.code == 204
This test fails as it returns 401 instead of 204, given the query on the auth decorator fails due to the RuntimeError, which returns then an Unauthorized response.
Any idea from the async experts here will be very appreciated, I'm quite lost on this!!!
Well, after a lot of digging, testing and, of course, learning quite a lot about asyncio, I made it work myself. Thanks for the suggestions so far.
The issue was that the event_loop from asyncio was not running; as #hoefling mentioned, pytest itself does not support coroutines, but pytest-asyncio brings such a useful feature to your tests. This is very well explained here: https://medium.com/ideas-at-igenius/testing-asyncio-python-code-with-pytest-a2f3628f82bc
So, without pytest-asyncio, your async code that needs to be tested will look like this:
def test_this_is_an_async_test():
loop = asyncio.get_event_loop()
result = loop.run_until_complete(my_async_function(param1, param2, param3)
assert result == 'expected'
We use loop.run_until_complete() as, otherwise, the loop will never be running, as this is the way asyncio works by default (and pytest makes nothing to make it work differently).
With pytest-asyncio, your test works with the well-known async / await parts:
async def test_this_is_an_async_test(event_loop):
result = await my_async_function(param1, param2, param3)
assert result == 'expected'
pytest-asyncio in this case wraps the run_until_complete() call above, summarizing it heavily, so the event loop will run and be available for your async code to use it.
Please note: the event_loop parameter in the second case is not even necessary here, pytest-asyncio gives one available for your test.
On the other hand, when you are testing your Tornado app, you usually need to get a http server up and running during your tests, listening in a well-known port, etc., so the usual way goes by writing fixtures to get a http server, base_url (usually http://localhost:, with an unused port, etc etc).
pytest-tornado comes up as a very useful one, as it offers several of these fixtures for you: http_server, http_client, unused_port, base_url, etc.
Also to mention, it gets a pytest mark's gen_test() feature, which converts any standard test to use coroutines via yield, and even to assert it will run with a given timeout, like this:
#pytest.mark.gen_test(timeout=3)
def test_fetch_my_data(http_client, base_url):
result = yield http_client.fetch('/'.join([base_url, 'result']))
assert len(result) == 1000
But, this way it does not support async / await, and actually only Tornado's ioloop will be available via the io_loop fixture (although Tornado's ioloop uses by default asyncio underneath from Tornado 5.0), so you'd need to combine both pytest.mark.gen_test and pytest.mark.asyncio, but in the right order! (which I did fail).
Once I understood better what could be the problem, this was the next approach:
#pytest.mark.gen_test(timeout=2)
#pytest.mark.asyncio
async def test_score_returns_204_empty(http_client, base_url):
score_url = '/'.join([base_url, URL_PREFIX, 'score'])
token = create_token('test', scopes=['score:get'])
headers = {
'Authorization': f'Bearer {token}',
'Accept': 'application/json',
}
response = await http_client.fetch(score_url, headers=headers, raise_error=False)
assert response.code == 204
But this is utterly wrong, if you understand how Python's decorator wrappers work. With the code above, pytest-asyncio's coroutine is then wrapped in a pytest-tornado yield gen.coroutine, which won't get the event-loop running... so my tests were still failing with the same problem. Any query to the database were returning a Future waiting for an event loop to be running.
My updated code once I made myself up of the silly mistake:
#pytest.mark.asyncio
#pytest.mark.gen_test(timeout=2)
async def test_score_returns_204_empty(http_client, base_url):
score_url = '/'.join([base_url, URL_PREFIX, 'score'])
token = create_token('test', scopes=['score:get'])
headers = {
'Authorization': f'Bearer {token}',
'Accept': 'application/json',
}
response = await http_client.fetch(score_url, headers=headers, raise_error=False)
assert response.code == 204
In this case, the gen.coroutine is wrapped inside the pytest-asyncio coroutine, and the event_loop runs the coroutines as expected!
But there were still a minor issue that took me a little while to realize, too; pytest-asyncio's event_loop fixture creates for every test a new event loop, while pytest-tornado creates too a new IOloop. And the tests were still failing, but this time with a different error.
The conftest.py file now looks like this; please note I've re-declared the event_loop fixture to use the event_loop from pytest-tornado io_loop fixture itself (please recall pytest-tornado creates a new io_loop on each test function):
#pytest.fixture(scope='function')
def event_loop(io_loop):
loop = io_loop.current().asyncio_loop
yield loop
loop.stop()
#pytest.fixture(scope='function')
async def db():
dbe = await setup_db()
yield dbe
#pytest.fixture
def app(db):
"""
Returns a valid testing Tornado Application instance.
:return:
"""
app = make_app(db)
settings.JWT_SECRET = 'its_secret_one'
yield app
Now all my tests work, I'm back a happy man and very proud of my now better understanding of the asyncio way of life. Cool!
Concerning soapUI and groovy, I'm trying to get assertion (working) and response both in XML into a variable. I get the error
groovy.lang.MissingMethodException: No signature of method: com.eviware.soapui.impl.wsdl.teststeps.WsdlTestRequestStep.getResponseAsXml() is applicable for argument types: () values: [] error at line: 6
I have tried adding import com.eviware.soapui.impl.wsdl.teststeps.WsdlTestRequestStep but still cant figure it. I did another attempt with message exchange, also to no avail - from what i understand you can't actually use messageExchange in this particular instance
import com.eviware.soapui.model.testsuite.Assertable.AssertionStatus
def TestCase = testRunner.getTestCase()
def StepList = TestCase.getTestStepList()
StepList.each
{
if(it.metaClass.hasProperty(it,'assertionStatus'))
{
if(it.assertionStatus == AssertionStatus.FAILED)
{
def ass = it.getAssertableContentAsXml()
def res = it.getResponseContentAsXml()
log.error "${it.name} " + "${it.assertionStatus}"
log.info ass + res
}
}
}
If you want to get the response from com.eviware.soapui.impl.wsdl.teststeps.WsdlTestRequestStep, a possible way is first get the testStep from this class using getTestStep() method.
This method returns a object of class com.eviware.soapui.model.testsuite.TestStep, from this object you can get the testSteps properties like request, response, endpoint... using getPropertyValue(java.lang.string) method.
So in your case to get the response use:
def res = it.getTestStep().getPropertyValue('Response')
instead of:
def res = it.getResponseContentAsXml()
As #tim_yates comments the exception description in this case it's pretty clear, so please take a look at the SOAPUI api and at the links provided in the answer for the next time :).
Hope this helps,