Repeated checking of ID path parameter in FastAPI - python-3.x

I have the following route specs:
GET /councils/{id}
PUT /councils/{id}
DELETE /councils/{id}
In all three routes, I have to check in the database whether the council with the id exists, like this:
council = crud.council.get_by_id(db=db, id=id)
if not council:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail='Council not found'
)
Which adds to a lot of boilerplate in the code. Is there any way of reducing this? I have thought of creating a dependency but then I have to write different dependency function for different models in my database. Is there any standard practice for this?
Thanks.

Using a dependency is the way to go - it allows you to extract the logic around "get a valid council from the URL"; you can generalize it further if you want to map /<model>/<id> to always retrieving something from a specific model; however - you might want to validate this against a set of possible values to avoid people trying to make you load random Python identifiers in your models class.
from fastapi import Path
def get_council_from_path(council_id: int = Path(...),
db: Session = Depends(get_db),
):
council = crud.council.get_by_id(db=db, id=id)
if not council:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail='Council not found'
)
return council
#app.get('/councils/{council_id}')
def get_council(council: Council = Depends(get_council_from_path)):
pass
You can generalize the dependency definition to make it reusable (I don't have anything available to test this right now, but the idea should work):
def get_model_from_path(model, db: Session = Depends(get_db)):
def get_model_from_id(id: int = Path(...)):
obj = model.get_by_id(db=db, id=id)
if not council:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail='{type(model)} not found'
)
return get_model_from_id
#app.get('/councils/{id}')
def get_council(council: Council = Depends(get_model_from_path(crud.council)):
pass

Here is one solution that works. FastAPI supports classes as dependencies. Therefore I can have a class like this:
class DbObjIdValidator:
def __init__(self, name: str):
if name not in dir(crud):
raise ValueError('Invalid model name specified')
self.crud = getattr(crud, name)
def __call__(self, db: Session = Depends(get_db), *, id: Any):
db_obj = self.crud.get_by_id(db=db, id=id)
if not db_obj:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail='not found'
)
return db_obj
Considering crud module imports the crud classes for all the ORM models. My example is closely following the fastAPI cookiecutter project by Tiangolo, and the crud design pattern he followed.
Now in order to use it, for example in councils_route.py, I can do the following:
router = APIRouter(prefix='/councils')
council_id_dep = DbObjIdValidator('council') # name must be same as import name
#router.get('/{id}', response_model=schemas.CouncilDB)
def get_council_by_id(
id: UUID,
db: Session = Depends(get_db),
council: Councils = Depends(council_id_dep)
):
return council

Related

Sync to Async Django ORM queryset foreign key property

Seemingly simple situation:
Django model has foreign key:
class Invite(models.Model):
inviter = models.ForeignKey(User, on_delete=models.CASCADE)
...
In async context, I do:
# get invite with sync_to_async decorator, then
print(invite.inviter)
Get async's favorite error:
You cannot call this from an async context - use a thread or sync_to_async
print(sync_to_async(invite.inviter)) # -> throws the same error
Sure, I can do:
#sync_to_async
def get_inviter(self, invite):
return invite.inviter
But, this is senile, if I have to do this for every queryset property call.
Is there a sane way to handle this?
Perhaps, there is a way to do this for all calls like that at once?
Yes, resolve the extra fields using select_related:
# Good: pick the foreign_key fields using select_related
user = await Invite.objects.select_related('user').aget(key=key).user
Your other string non-foreign such as strings and ints attributes should already
exist on the model.
Won't work, (although they feel like they should)
# Error django.core.exceptions.SynchronousOnlyOperation ... use sync_to_async
user = await Model.objects.aget(key=key).user
# Error (The field is actually missing from the `_state` fields cache.
user = await sync_to_async(Invite.objects.get)(key=key).user
Other examples for research
A standard aget, followed by a foreign key inspection yields a SynchronousOnlyOperation error.
I have a string key, and a ForeignKey user to the standard user model.
class Invite(models.Model):
user = fields.user_fk()
key = fields.str_uuid()
An example with alternatives that mostly don't work:
Invite = get_model('invites.Invite')
User = get_user_model()
def _get_invite(key):
return Invite.objects.get(key=key)
async def invite_get(self, key):
# (a) works, the related field is populated on response.
user = await Invite.objects.select_related('user').aget(key=key).user
async def intermediate_examples(self, key):
# works, but is clunky.
user_id = await Invite.objects.aget(key=key).user_id
# The `user_id` (any `_id` key) exists for a FK
user = await User.objects.aget(id=user_id)
async def failure_examples(self, key):
# (b) does not work.
user = await sync_to_async(Invite.objects.get)(key=key).user
invite = await sync_to_async(Invite.objects.get)(key=key)
# (c) these are not valid, although the error may say so.
user = await invite.user
user = await sync_to_async(invite.user)
# same as the example (b)
get_invite = sync_to_async(_get_invite, thread_sensitive=True)
invite = get_invite(key)
user = invite.user # Error
# (d) Does not populate the additional model
user = await Invite.objects.aget(key=key).user # Error
print(sync_to_async(invite.inviter)) # -> throws the same error
That's because it's equivalent to:
i = invite.inviter # -> throws the error here
af = sync_to_async(i)
print(af)
The correct usage is:
f = lambda: invite.inviter
af = sync_to_async(f)
i = await af()
print(i)
# As a one-liner
print(await sync_to_async(lambda: invite.inviter)())
Is there a sane way to handle this?
Perhaps, there is a way to do this for all calls like that at once?
(Disclaimer: Not tested in production.)
With nest_asyncio, you could do this:
def do(f):
import nest_asyncio
nest_asyncio.apply()
return asyncio.run(sync_to_async(f)())
print(do(lambda: invite.inviter))
Or take it even further:
class SynchronousOnlyAttributeHandler:
def __getattribute__(self, item):
from django.core.exceptions import SynchronousOnlyOperation
try:
return super().__getattribute__(item)
except SynchronousOnlyOperation:
from asgiref.sync import sync_to_async
import asyncio
import nest_asyncio
nest_asyncio.apply()
return asyncio.run(sync_to_async(lambda: self.__getattribute__(item))())
class Invite(models.Model, AsyncUnsafeAttributeHandler):
inviter = models.ForeignKey(User, on_delete=models.CASCADE, null=True)
...
# Do this even in async context
print(invite.inviter)
Does something like this work? Instead of invite.inviter you do await async_resolve_attributes(invite, "inviter")
#sync_to_async
def async_resolve_attributes(instance, *attributes):
current_instance = instance
for attribute in attributes:
current_instance = getattr(current_instance, attribute)
resolved_attribute = current_instance
return resolved_attribute

Pytest+FastAPI+SQLAlchemy+Postgres InterfaceError

I've met some problem with running tests using FastAPI+SQLAlchemy and PostgreSQL, which leads to lots of errors (however, it works well on SQLite). I've created a repo with MVP app and Pytest on Docker Compose testing.
The basic error is sqlalchemy.exc.InterfaceError('cannot perform operation: another operation is in progress'). This may be related to the app/DB initialization, though I checked that all the operations get performed sequentially. Also I tried to use single instance of TestClient for the all tests, but got no better results. I hope to find a solution, a correct way for testing such apps 🙏
Here are the most important parts of the code:
app.py:
app = FastAPI()
some_items = dict()
#app.on_event("startup")
async def startup():
await create_database()
# Extract some data from env, local files, or S3
some_items["pi"] = 3.1415926535
some_items["eu"] = 2.7182818284
#app.post("/{name}")
async def create_obj(name: str, request: Request):
data = await request.json()
if data.get("code") in some_items:
data["value"] = some_items[data["code"]]
async with async_session() as session:
async with session.begin():
await create_object(session, name, data)
return JSONResponse(status_code=200, content=data)
else:
return JSONResponse(status_code=404, content={})
#app.get("/{name}")
async def get_connected_register(name: str):
async with async_session() as session:
async with session.begin():
objects = await get_objects(session, name)
result = []
for obj in objects:
result.append({
"id": obj.id, "name": obj.name, **obj.data,
})
return result
tests.py:
#pytest.fixture(scope="module")
def event_loop():
loop = asyncio.get_event_loop()
yield loop
loop.close()
#pytest_asyncio.fixture(scope="module")
#pytest.mark.asyncio
async def get_db():
await delete_database()
await create_database()
#pytest.mark.parametrize("test_case", test_cases_post)
def test_post(get_db, test_case):
with TestClient(app)() as client:
response = client.post(f"/{test_case['name']}", json=test_case["data"])
assert response.status_code == test_case["res"]
#pytest.mark.parametrize("test_case", test_cases_get)
def test_get(get_db, test_case):
with TestClient(app)() as client:
response = client.get(f"/{test_case['name']}")
assert len(response.json()) == test_case["count"]
db.py:
DATABASE_URL = environ.get("DATABASE_URL", "sqlite+aiosqlite:///./test.db")
engine = create_async_engine(DATABASE_URL, future=True, echo=True)
async_session = sessionmaker(engine, expire_on_commit=False, class_=AsyncSession)
Base = declarative_base()
async def delete_database():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.drop_all)
async def create_database():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
class Model(Base):
__tablename__ = "smth"
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
data = Column(JSON, nullable=False)
idx_main = Index("name", "id")
async def create_object(db: Session, name: str, data: dict):
connection = Model(name=name, data=data)
db.add(connection)
await db.flush()
async def get_objects(db: Session, name: str):
raw_q = select(Model) \
.where(Model.name == name) \
.order_by(Model.id)
q = await db.execute(raw_q)
return q.scalars().all()
At the moment the testing code is quite coupled, so the test suite seems to work as follows:
the database is created once for all tests
the first set of tests runs and populates the database
the second set of tests runs (and will only succeed if the database is fully populated)
This has value as an end-to-end test, but I think it would work better if the whole thing were placed in a single test function.
As far as unit testing goes, it is a bit problematic. I'm not sure whether pytest-asyncio makes guarantees about test running order (there are pytest plugins that exist solely to make tests run in a deterministic order), and certainly the principle is that unit tests should be independent of each other.
The testing is coupled in another important way too - the database I/O code and the application logic are being tested simultaneously.
A practice that FastAPI encourages is to make use of dependency injection in your routes:
from fastapi import Depends, FastAPI, Request
...
def get_sessionmaker() -> Callable:
# this is a bit baroque, but x = Depends(y) assigns x = y()
# so that's why it's here
return async_session
#app.post("/{name}")
async def create_obj(name: str, request: Request, get_session = Depends(get_sessionmaker)):
data = await request.json()
if data.get("code") in some_items:
data["value"] = some_items[data["code"]]
async with get_session() as session:
async with session.begin():
await create_object(session, name, data)
return JSONResponse(status_code=200, content=data)
else:
return JSONResponse(status_code=404, content={})
When it comes to testing, FastAPI then allows you to swap out your real dependencies so that you can e.g. mock the database and test the application logic in isolation from database I/O:
from app import app, get_sessionmaker
from mocks import mock_sessionmaker
...
client = TestClient(app)
...
async def override_sessionmaker():
return mock_sessionmaker
app.dependency_overrides[get_sessionmaker] = override_sessionmaker
# now we can run some tests
This will mean that when you run your tests, whatever you put in mocks.mock_sessionmaker will give you the get_session function in your tests, rather than get_sessionmaker. We could have our mock_sessionmaker return a function called get_mock_session.
In other words, rather than with async_session() as session:, in the tests we'd have with get_mock_session() as session:.
Unfortunately this get_mock_session has to return something a little complicated (let's call it mock_session), because the application code then does an async with session.begin().
I'd be tempted to refactor the application code for simplicity, but if not then it will have to not throw errors when you call .begin, .add, and .flush on it, in this example, and those methods have to be async. But they don't have to do anything, so it's not too bad...
The FastAPI docs have an alternative example of databases + dependencies that does leave the code a little coupled, but uses SQLite strictly for the purpose of unit tests, leaving you free to do something different for an end-to-end test and in the application itself.

Error in django unittest: 'object has no attribute' object

I started testing views, but I found an error in the test, tell me where I went wrong, I'm just starting to learn, so don't judge strictly
my views:
`class MovieDetailView(Genre, DetailView):
model = Movie
slug_field = "url"
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context["star_form"] = RatingForm()
return context
`
my test looks like this:
`def test_starform_set_in_context(self):
request = RequestFactory().get('some-slug')
view = MovieDetailView()
view.setup(request)
context = view.get_context_data()
self.assertIn('star_form', context))`
url:
`path("<slug:slug>/", views.MovieDetailView.as_view(), name="movie_detail"),`
I think this is because Django's get_context_data() function uses the 'object' to pass it into the context.So, I suggest you to use get_object() method.
See this for more details: https://ccbv.co.uk/projects/Django/3.0/django.views.generic.detail/DetailView/
basically you need to return an object e.g.
def get_object(self,queryset=None):
obj = Movie.object.get(slug=self.kwargs.get('slug'))
return obj #also handle 404
or try one more thing :
slug_fields = slug
path should be
path("<str:slug>/", views.MovieDetailView.as_view(), name="movie_detail"),

Automatic method delegation in Python

I have a rather contrived code here :
backend_data = {
"admins": ["Leo", "Martin", "Thomas", "Katrin"],
"members": [
"Leo",
"Martin",
"Thomas",
"Katrin",
"Subhayan",
"Clemens",
"Thoralf"
],
"juniors": ["Orianne", "Antonia", "Sarah"]
}
class Backend:
def __init__(self, data):
self.backend_data = data
def get_all_admins(self):
return self.backend_data.get("admins")
def get_all_members(self):
return self.backend_data.get("members")
def get_all_juniors(self):
return self.backend_data.get("juniors")
class BackendAdaptor:
# Does some conversion and validation
def __init__(self, backend):
self.backend = backend
def get_all_admins(self):
return (admin for admin in self.backend.get_all_admins())
def get_all_members(self):
return (member for member in self.backend.get_all_members() if member not in self.backend.get_all_admins())
def get_all_juniors(self):
return (junior for junior in self.backend.get_all_juniors())
if __name__ == "__main__":
backend = Backend(data=backend_data)
adaptor = BackendAdaptor(backend=backend)
print(f"All admins are : {list(adaptor.get_all_admins())}")
print(f"All members are : {list(adaptor.get_all_members())}")
print(f"All juniors are : {list(adaptor.get_all_juniors())}")
So the BackendAdaptor class basically would be used to do some validation and conversion of the data that we get from the Backend .
The client should only be asked to interact with the API of the BackendAdaptor which is exactly similar to that of Backend . The adaptor class sits in middle and gets data from Backend does some validation if required and the gives back the data to client.
The issue is that the validation on the data that is getting returned from the Backend is not done for every method(For ex: there is validation done on get_all_members but not on get_all_admins and also not on get_all_juniors). The method just gives back a generator on whatever data it gets from Backend.
As is the case now i still have to implement a one liner methods for them .
Is there a way in Python to avoid this ? I am thinking in lines of magic methods like __getattribute__ ? But i have no idea on how to do this for methods.
So the best case scenario is this:
I implement the methods for which i know that i have to do some validation on Backend data
For the rest of the methods it is automatically delegated to Backend and then i just return a generator from what i get back
Any help would be greatly appreciated.
You can implement __getattr__. It is only called if a non-existing attribute is accessed. This will return some generic function with the desired functionality.
class BackendAdaptor:
def __init__(self, backend):
self.backend = backend
def __getattr__(self, name):
if not hasattr(self.backend, name):
raise AttributeError(f"'{name}' not in backend.")
return lambda: (i for i in getattr(self.backend, name)())
def get_all_members(self):
return (member for member in self.backend.get_all_members() if member not in self.backend.get_all_admins())

Use existing sql-alchemy model class as flask-restplus api.model?

I am developing a CRUD application using vue.js and vuetify as frontend (view) and python flask-resplus and sqlAlchemy as backend (controler and model).
app/main/model/person.py
from sqlalchemy import Column, Integer, String, Date
from app.main.repository.base_repository import Base
class Person(Base):
__tablename__ = 'person'
id = Column(Integer, primary_key=True)
name = Column(String)
date_of_birth = Column(Date)
def __init__(self, name=None, date_of_birth=None):
if name is not None:
self.name = name
if date_of_birth is not None:
self.date_of_birth = date_of_birth
app/main/controller/person_controller.py
from flask_restplus import Namespace, fields, Resource, cors
from flask import request
from app.main.service.person_service import PersonService
from app.main.model.person import Person
api = Namespace('persons', description='Person related operations')
persServ : PersonService = PersonService()
model = api.model('Person', {
'id': fields.Integer,
'name': fields.String,
'date_of_birth': fields.Date
})
#api.route("/persons/all")
class PersonList(Resource):
#api.marshal_with(model)
def get(self, **kwargs):
return persServ.get_persons()
# return Person(name="Test", date_of_birth=date(1984, 10, 20))
#api.route("/person/<int:id>")
class PersonReturn(Resource):
#api.marshal_with(model)
def get(self, id):
return persServ.get_person(id)
#api.route("/person/<int:id>")
class PersonUpdate(Resource):
#api.marshal_with(model)
def put(self, id):
data = request.json
#TODO marshalling currently still error
return persServ.update_person(pers=data)
#api.route("/person")
class PersonCreate(Resource):
#api.marshal_with(model)
def post(self):
data = request.json
#TODO: check why person is not correctly linked to sql alchemy when reated here, make sure that model.Person is created
# pers = Person()
# pers.name = data['name']
# persServ.create_person(pers)
return persServ.create_person_02(data['name'])
#api.route("/person/<int:id>")
class PersonDelete(Resource):
def delete(self, id):
persServ.delete_person(id)
return '', 204
Questions also after implementing a spring java backend :
1) Is it necessary to provide an api.model in person_controller.py or can I annotate properties in person.py? Could you kindly guide me to a tutorial annotating an existing object?
2) Naming conventions flask api classes: In my eyes it definitely makes sense to use PersonList and Person instead of PersonUpdate, PersonCreate to ease API documentation as described in the flask-resPlus full example: https://flask-restplus.readthedocs.io/en/stable/example.html
However, I already have an existing Person class in person.py. I am thinking about having two person classes, a Person(Resource) in person_controller.py for Post, Put and Get operations and a Person in person.py as domain specific person object. However, I do not want to cause confusion and adhere to naming and orgainization good practices. What is the suggested naming approach? Is there a good example tutorial?

Resources