My typical path is something like
/user/{user_id}/resource/{resource_id}
I have a validation method, already written in async python, like this:
async def is_allowed(user_id: int, resource_id: int) -> bool
That returns a boolean: true if the user can access the resource, false otherwise.
I want to write a middleware that calls is_allowed extracting the variables from the path.
I fiddled around but I can't find how to get them: I was expecting to get this information from request.path_params.
A somehow more complete example (edited following #Marcelo Trylesinski answer):
import logging
from fastapi import FastAPI
from starlette.requests import Request
from starlette.responses import Response
app = FastAPI()
_logger = logging.getLogger()
_logger.setLevel(logging.DEBUG)
async def is_allowed(user_id, resource_id):
_logger.error(user_id)
_logger.error(resource_id)
return True
#app.middleware('http')
async def acl(request: Request, call_next):
user_id = request.path_params.get("user_id", None)
resource_id = request.path_params.get("resource_id", None)
allowed = await is_allowed(user_id, resource_id)
if not allowed:
return Response(status_code=403)
else:
return await call_next(request)
#app.get('/user/{user_id}/resource/{resource_id}')
async def my_handler(user_id: int, resource_id: int):
return {"what": f"Doing stuff with {user_id} on {resource_id}"}
The logged values are None.
You will not be able to achieve your goal with a Middleware, because Middlewares are executed before the routing.
Therefore FastAPI/Starlette doesn't know which path it will match to and cannot populate path_params.
You will have to use a different solution, such as passing these params on a cookie, header or query arg, or using a decorator/Dependency.
Reference:
https://github.com/encode/starlette/issues/230
https://fastapi.tiangolo.com/tutorial/middleware/#middleware
Related
Seemingly simple situation:
Django model has foreign key:
class Invite(models.Model):
inviter = models.ForeignKey(User, on_delete=models.CASCADE)
...
In async context, I do:
# get invite with sync_to_async decorator, then
print(invite.inviter)
Get async's favorite error:
You cannot call this from an async context - use a thread or sync_to_async
print(sync_to_async(invite.inviter)) # -> throws the same error
Sure, I can do:
#sync_to_async
def get_inviter(self, invite):
return invite.inviter
But, this is senile, if I have to do this for every queryset property call.
Is there a sane way to handle this?
Perhaps, there is a way to do this for all calls like that at once?
Yes, resolve the extra fields using select_related:
# Good: pick the foreign_key fields using select_related
user = await Invite.objects.select_related('user').aget(key=key).user
Your other string non-foreign such as strings and ints attributes should already
exist on the model.
Won't work, (although they feel like they should)
# Error django.core.exceptions.SynchronousOnlyOperation ... use sync_to_async
user = await Model.objects.aget(key=key).user
# Error (The field is actually missing from the `_state` fields cache.
user = await sync_to_async(Invite.objects.get)(key=key).user
Other examples for research
A standard aget, followed by a foreign key inspection yields a SynchronousOnlyOperation error.
I have a string key, and a ForeignKey user to the standard user model.
class Invite(models.Model):
user = fields.user_fk()
key = fields.str_uuid()
An example with alternatives that mostly don't work:
Invite = get_model('invites.Invite')
User = get_user_model()
def _get_invite(key):
return Invite.objects.get(key=key)
async def invite_get(self, key):
# (a) works, the related field is populated on response.
user = await Invite.objects.select_related('user').aget(key=key).user
async def intermediate_examples(self, key):
# works, but is clunky.
user_id = await Invite.objects.aget(key=key).user_id
# The `user_id` (any `_id` key) exists for a FK
user = await User.objects.aget(id=user_id)
async def failure_examples(self, key):
# (b) does not work.
user = await sync_to_async(Invite.objects.get)(key=key).user
invite = await sync_to_async(Invite.objects.get)(key=key)
# (c) these are not valid, although the error may say so.
user = await invite.user
user = await sync_to_async(invite.user)
# same as the example (b)
get_invite = sync_to_async(_get_invite, thread_sensitive=True)
invite = get_invite(key)
user = invite.user # Error
# (d) Does not populate the additional model
user = await Invite.objects.aget(key=key).user # Error
print(sync_to_async(invite.inviter)) # -> throws the same error
That's because it's equivalent to:
i = invite.inviter # -> throws the error here
af = sync_to_async(i)
print(af)
The correct usage is:
f = lambda: invite.inviter
af = sync_to_async(f)
i = await af()
print(i)
# As a one-liner
print(await sync_to_async(lambda: invite.inviter)())
Is there a sane way to handle this?
Perhaps, there is a way to do this for all calls like that at once?
(Disclaimer: Not tested in production.)
With nest_asyncio, you could do this:
def do(f):
import nest_asyncio
nest_asyncio.apply()
return asyncio.run(sync_to_async(f)())
print(do(lambda: invite.inviter))
Or take it even further:
class SynchronousOnlyAttributeHandler:
def __getattribute__(self, item):
from django.core.exceptions import SynchronousOnlyOperation
try:
return super().__getattribute__(item)
except SynchronousOnlyOperation:
from asgiref.sync import sync_to_async
import asyncio
import nest_asyncio
nest_asyncio.apply()
return asyncio.run(sync_to_async(lambda: self.__getattribute__(item))())
class Invite(models.Model, AsyncUnsafeAttributeHandler):
inviter = models.ForeignKey(User, on_delete=models.CASCADE, null=True)
...
# Do this even in async context
print(invite.inviter)
Does something like this work? Instead of invite.inviter you do await async_resolve_attributes(invite, "inviter")
#sync_to_async
def async_resolve_attributes(instance, *attributes):
current_instance = instance
for attribute in attributes:
current_instance = getattr(current_instance, attribute)
resolved_attribute = current_instance
return resolved_attribute
I've met some problem with running tests using FastAPI+SQLAlchemy and PostgreSQL, which leads to lots of errors (however, it works well on SQLite). I've created a repo with MVP app and Pytest on Docker Compose testing.
The basic error is sqlalchemy.exc.InterfaceError('cannot perform operation: another operation is in progress'). This may be related to the app/DB initialization, though I checked that all the operations get performed sequentially. Also I tried to use single instance of TestClient for the all tests, but got no better results. I hope to find a solution, a correct way for testing such apps 🙏
Here are the most important parts of the code:
app.py:
app = FastAPI()
some_items = dict()
#app.on_event("startup")
async def startup():
await create_database()
# Extract some data from env, local files, or S3
some_items["pi"] = 3.1415926535
some_items["eu"] = 2.7182818284
#app.post("/{name}")
async def create_obj(name: str, request: Request):
data = await request.json()
if data.get("code") in some_items:
data["value"] = some_items[data["code"]]
async with async_session() as session:
async with session.begin():
await create_object(session, name, data)
return JSONResponse(status_code=200, content=data)
else:
return JSONResponse(status_code=404, content={})
#app.get("/{name}")
async def get_connected_register(name: str):
async with async_session() as session:
async with session.begin():
objects = await get_objects(session, name)
result = []
for obj in objects:
result.append({
"id": obj.id, "name": obj.name, **obj.data,
})
return result
tests.py:
#pytest.fixture(scope="module")
def event_loop():
loop = asyncio.get_event_loop()
yield loop
loop.close()
#pytest_asyncio.fixture(scope="module")
#pytest.mark.asyncio
async def get_db():
await delete_database()
await create_database()
#pytest.mark.parametrize("test_case", test_cases_post)
def test_post(get_db, test_case):
with TestClient(app)() as client:
response = client.post(f"/{test_case['name']}", json=test_case["data"])
assert response.status_code == test_case["res"]
#pytest.mark.parametrize("test_case", test_cases_get)
def test_get(get_db, test_case):
with TestClient(app)() as client:
response = client.get(f"/{test_case['name']}")
assert len(response.json()) == test_case["count"]
db.py:
DATABASE_URL = environ.get("DATABASE_URL", "sqlite+aiosqlite:///./test.db")
engine = create_async_engine(DATABASE_URL, future=True, echo=True)
async_session = sessionmaker(engine, expire_on_commit=False, class_=AsyncSession)
Base = declarative_base()
async def delete_database():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.drop_all)
async def create_database():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
class Model(Base):
__tablename__ = "smth"
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
data = Column(JSON, nullable=False)
idx_main = Index("name", "id")
async def create_object(db: Session, name: str, data: dict):
connection = Model(name=name, data=data)
db.add(connection)
await db.flush()
async def get_objects(db: Session, name: str):
raw_q = select(Model) \
.where(Model.name == name) \
.order_by(Model.id)
q = await db.execute(raw_q)
return q.scalars().all()
At the moment the testing code is quite coupled, so the test suite seems to work as follows:
the database is created once for all tests
the first set of tests runs and populates the database
the second set of tests runs (and will only succeed if the database is fully populated)
This has value as an end-to-end test, but I think it would work better if the whole thing were placed in a single test function.
As far as unit testing goes, it is a bit problematic. I'm not sure whether pytest-asyncio makes guarantees about test running order (there are pytest plugins that exist solely to make tests run in a deterministic order), and certainly the principle is that unit tests should be independent of each other.
The testing is coupled in another important way too - the database I/O code and the application logic are being tested simultaneously.
A practice that FastAPI encourages is to make use of dependency injection in your routes:
from fastapi import Depends, FastAPI, Request
...
def get_sessionmaker() -> Callable:
# this is a bit baroque, but x = Depends(y) assigns x = y()
# so that's why it's here
return async_session
#app.post("/{name}")
async def create_obj(name: str, request: Request, get_session = Depends(get_sessionmaker)):
data = await request.json()
if data.get("code") in some_items:
data["value"] = some_items[data["code"]]
async with get_session() as session:
async with session.begin():
await create_object(session, name, data)
return JSONResponse(status_code=200, content=data)
else:
return JSONResponse(status_code=404, content={})
When it comes to testing, FastAPI then allows you to swap out your real dependencies so that you can e.g. mock the database and test the application logic in isolation from database I/O:
from app import app, get_sessionmaker
from mocks import mock_sessionmaker
...
client = TestClient(app)
...
async def override_sessionmaker():
return mock_sessionmaker
app.dependency_overrides[get_sessionmaker] = override_sessionmaker
# now we can run some tests
This will mean that when you run your tests, whatever you put in mocks.mock_sessionmaker will give you the get_session function in your tests, rather than get_sessionmaker. We could have our mock_sessionmaker return a function called get_mock_session.
In other words, rather than with async_session() as session:, in the tests we'd have with get_mock_session() as session:.
Unfortunately this get_mock_session has to return something a little complicated (let's call it mock_session), because the application code then does an async with session.begin().
I'd be tempted to refactor the application code for simplicity, but if not then it will have to not throw errors when you call .begin, .add, and .flush on it, in this example, and those methods have to be async. But they don't have to do anything, so it's not too bad...
The FastAPI docs have an alternative example of databases + dependencies that does leave the code a little coupled, but uses SQLite strictly for the purpose of unit tests, leaving you free to do something different for an end-to-end test and in the application itself.
I am trying to add a custom decorator to secure fastapi endpoints to list of users. The incoming payload has JWT data in the header and/or cookies.
With out the decorator, fastapi_jwt_auth allows/denies to the endpoints as long as the token is valid. However when adding the custom decorator, it fails to pass the header/cookie payload to the decorator to be parsed and return the correct response if valid/invalid.
From what I could understand from this post it should be able to obtain the payload from the router function. However it only seems to create a blank class object.
from functools import wraps
from fastapi import Depends
from fastapi.responses import JSONResponse
from fastapi_jwt_auth import AuthJWT
def auth(users: Union[List[str], None, str] = None):
def decorator_auth(func):
#wraps(func)
def wrapper_auth(*args, **kwargs):
Authorize.jwt_required()
jwt_username = Authorize.get_raw_jwt()['username']
if jwt_username not in users:
return JSONResponse(
status_code=401,
content={"detail": "Unauthorized"} )
return func(*args, **kwargs)
return wrapper_auth
return decorator_auth
#app.get('/data')
def secure_data(Authorize: AuthJWT = Depends()):
Authorize.jwt_required()
return JSONResponse(
status_code=200,
content={"detail": "Payload only all with valid tokens."})
#app.get('/data1')
#auth(users=['user1', 'user3')
def secure_data_1(Authorize: AuthJWT = Depends()):
return JSONResponse(
status_code=200,
content={"detail": "Payload only for User 1 and User 3."})
#app.get('/data2')
#auth(users=['user2'])
def secure_data2(Authorize: AuthJWT = Depends()):
return JSONResponse(
status_code=200,
content={"detail": "Payload only for User 2."} )
Im trying to get the content-length of FastAPI response on the server side for logging purpose. Is this possible? Thanks.
#app.get("/foo")
async def foo(background_tasks: BackgroundTasks):
data = {"foo": "foo"}
response_content_length = get_content_length()
background_tasks.add_task(log, response_content_length )
return data
You can create your own route by inheriting APIRoute class, now you should be able to log everything, without repeating yourself.
from fastapi import FastAPI, Request, Response, Body, BackgroundTasks, APIRouter
from fastapi.routing import APIRoute
from typing import Callable, List
class ContextIncludedRoute(APIRoute):
def get_route_handler(self) -> Callable:
original_route_handler = super().get_route_handler()
async def custom_route_handler(request: Request) -> Response:
response: Response = await original_route_handler(request)
content_length = response.headers["content-length"]
print(content_length)
return response
return custom_route_handler
app = FastAPI()
router = APIRouter(route_class=ContextIncludedRoute)
#router.post("/dummy")
async def dummy():
return {"foo":"foo"}
app.include_router(router)
Trying to figure out the right mechanism to use here.
I want to modify the flask request coming in every time.
I think the request is immutable, so I am trying to figure out if this mechanism exists.
Basically, I want to append a string onto the end of the request coming in.
I can hook into the request and the right time in a before_request handler like this:
#app.before_app_request
def before_request_custom():
# Get the request
req = flask.request
method = str(req.method)
if method == "GET":
# Do stuff here
pass
But I am not sure what to actually do to add this in, and don't see a way to accomplish it...I guess i could redirect, but that seems silly in this case. Any ideas?
The request object is immutable (https://werkzeug.palletsprojects.com/en/1.0.x/wrappers/#base-wrappers), but request.args or request.form can be set from ImmutableOrderedMultiDict to just OrderedMultiDict using Subclassing on Flask (https://flask.palletsprojects.com/en/1.1.x/patterns/subclassing/). Here's an example of how you could add that filter[is_deleted]=False URL param:
from flask import Flask, request, Request
from werkzeug.datastructures import OrderedMultiDict
class MyRequest(Request):
parameter_storage_class = OrderedMultiDict
class MyApp(Flask):
def __init__(self, import_name):
super(MyApp, self).__init__(import_name)
self.before_request(self.my_before_method)
def my_before_method(self):
if "endpoint" in request.base_url:
request.args["filter[is_deleted]"] = "False"
app = MyApp(__name__)
app.request_class = MyRequest
#app.route('/endpoint/')
def endpoint():
filter = request.args.get('filter[is_deleted]')
return filter
This way you can modify request.args before you actually send the request.
How about this?
from flask import g
#app.before_request
def before_request():
# Get the request
req = flask.request
method = str(req.method)
if method == "GET":
g.my_addon = "secret sauce"
return None
Then, g.my_addon is available in every view function:
from flask import g
#app.route('/my_view')
def my_view():
if g.my_addon == "secret sauce":
print('it worked!')
Using test_request_context() you can make the trick.
Related: https://flask.palletsprojects.com/en/1.1.x/quickstart/#accessing-request-data