I'm using Pyramid (1.5.7) + traversal and following the documentation I've tried all possible ways to get the "Not found exception view" working.
from pyramid.view import notfound_view_config,forbidden_view_config, view_config
#notfound_view_config(renderer="error/not_found.jinja2")
def not_found_view(request):
request.response.status = 404
return {}
#forbidden_view_config(renderer="error/forbidden.jinja2")
def forbidden_view(request):
return {}
Using contexts:
from pyramid.view import view_config
from pyramid.httpexceptions import HTTPForbidden, HTTPUnauthorized
#view_config(context=HTTPNotFound, renderer="error/not_found.jinja2")
def not_found_view(request):
request.response.status = 404
return {}
#view_config(context=HTTPForbidden, renderer="error/forbidden.jinja2")
def forbidden_view(request):
return {}
I'm using the Scan mode, but I've tried also adding a custom function to the configuration:
def main(globals, **settings):
config = Configurator()
config.add_notfound_view(notfound)
Not luck either, all time getting the following unhandled exception:
raise HTTPNotFound(msg)
pyramid.httpexceptions.HTTPNotFound: /example-url
Ouch... My bad! I was using a tween which was preventing Pyramid to load the Exceptions:
def predispatch_factory(handler, registry):
# one-time configuration code goes here
def predispatch(request):
# code to be executed for each request before
# the actual application code goes here
response = handler(request)
# code to be executed for each request after
# the actual application code goes here
return response
return predispatch
Still I don't know why but after removing this tween all seems to work as expected.
Related
Python version 3.9, FastAPI version 0.78.0
I have a custom function that I use for application exception handling. When requests run into internal logic problems, i.e I want to send an HTTP response of 400 for some reason, I call a utility function.
#staticmethod
def raise_error(error: str, code: int) -> None:
logger.error(error)
raise HTTPException(status_code=code, detail=error)
Not a fan of this approach. So I look at
from fastapi import FastAPI, HTTPException, status
from fastapi.respones import JSONResponse
class ExceptionCustom(HTTPException):
pass
def exception_404_handler(request: Request, exc: HTTPException):
return JSONResponse(status_code=status.HTTP_404_NOT_FOUND, content={"message": "404"})
app.add_exception_handler(ExceptionCustom, exception_404_handler)
The problem I run into with the above approach is the inability to pass in the message as an argument.
Any thoughts on the whole topic?
Your custom exception can have any custom attributes that you want. Let's say you write it this way:
class ExceptionCustom(HTTPException):
pass
in your custom handler, you can do something like
def exception_404_handler(request: Request, exc: HTTPException):
return JSONResponse(status_code=status.HTTP_404_NOT_FOUND, content={"message": exc.detail})
Then, all you need to do is to raise the exception this way:
raise ExceptionCustom(status_code=404, detail='error message')
Note that you are creating a handler for this specific ExceptionCustom. If all you need is the message, you can write something more generic:
class MyHTTPException(HTTPException):
pass
def my_http_exception_handler(request: Request, exc: HTTPException):
return JSONResponse(status_code=exc.status_code, content={"message": exc.detail})
app.add_exception_handler(MyHTTPException, my_http_exception_handler)
This way you can raise any exception, with any status code and any message and have the message in your JSON response.
There's a detailed explanation on FastAPI docs
You can add custom exception handlers, and use attributes in your Exception class (i.e., class MyException(Exception) in the example below) to pass any message/variables that you would like to do so. The exception handler (i.e., #app.exception_handler(MyException) in the case below) will handle the exception as you wish and return your custom message. For more options, please have a look at this related answer as well.
Working Example
To trigger the exception in the example below, access the following URL from your browser: http://localhost:8000/something
from fastapi import FastAPI, Request, status
from fastapi.responses import JSONResponse
class MyException(Exception):
def __init__(self, name: str):
self.name = name
app = FastAPI()
#app.exception_handler(MyException)
async def my_exception_handler(request: Request, exc: MyException):
return JSONResponse(status_code=status.HTTP_404_NOT_FOUND,
content={"message": f"{exc.name} cannot be found." })
#app.get("/{name}")
def read_name(name: str):
if name == "something":
raise MyException(name=name)
return {"name": name}
I need to make an initial call to a service before I start my scraper (the initial call, gives me some cookies and headers), I decided to use InitSpider and override the init_request method to achieve this. however I also need to use start_requests to build my links and add some meta values like proxies and whatnot to that specific spider, but I'm facing a problem. whenever I override start_requests, my crawler doesn't call init_request anymore and I can not do the initialization and in order to get init_request working is to not override the start_requests method which is impossible in my case. any suggestions or possible solutions to my code:
class SomethingSpider(InitSpider):
name = 'something'
allowed_domains = ['something.something']
aod_url = "https://something?="
start_urls = ["id1","id2","id3"]
custom_settings = {
'DOWNLOAD_FAIL_ON_DATALOSS' : False,
'CONCURRENT_ITEMS': 20,
'DOWNLOAD_TIMEOUT': 10,
'CONCURRENT_REQUESTS': 3,
'COOKIES_ENABLED': True,
'CONCURRENT_REQUESTS_PER_DOMAIN': 20
}
def init_request(self):
yield scrapy.Request(url="https://something",callback=self.check_temp_cookie, meta={'proxy': 'someproxy:1111'})
def check_temp_cookie(self, response):
"""Check the response returned by a login request to see if we are
successfully logged in.
"""
if response.status == 200:
print("H2")
# Now the crawling can begin..
return self.initialized()
else:
print("H3")
# Something went wrong, we couldn't log in, so nothing happens.
def start_requests(self):
print("H4")
proxies = ["xyz:0000","abc:1111"]
for url in self.start_urls:
yield scrapy.Request(url=self.aod_url+url, callback=self.parse, meta={'proxy': random.choice(proxies)})
def parse(self, response):
#some processing happens
yield {
#some data
}
except Exception as err:
print("Connecting to...")
Spiders page (generic spiders section) on official scrapy docs doesn't have any mention of InitSpider You are trying to use.
InitSpider class from https://github.com/scrapy/scrapy/blob/2.5.0/scrapy/spiders/init.py written ~10 years ago (at that... ancient versions of scrapy start_requests method worked completely differently).
From this perspective I recommend You to not use undocumented and probably outdated InitSpider.
On current versions of scrapy required functionality can be implemented using regular Spider class:
import scrapy
class SomethingSpider(scrapy.Spider):
...
def start_requests(self):
yield scrapy.Request(url="https://something",callback=self.check_temp_cookie, meta={'proxy': 'someproxy:1111'})
def check_temp_cookie(self, response):
"""Check the response returned by a login request to see if we are
successfully logged in.
"""
if response.status == 200:
print("H2")
# Now the crawling can begin..
...
#Schedule next requests here:
for url in self.start_urls:
yield scrapy.Request(url=self.aod_url+url, callback=self.parse, ....})
else:
print("H3")
# Something went wrong, we couldn't log in, so nothing happens.
def parse(self, response):
...
If you are looking speicfically at incorporating logging in then I would reccomend you look at Using FormRequest.from_response() to simulate a user login in the scrapy docs.
Here is the spider example they give:
import scrapy
def authentication_failed(response):
# TODO: Check the contents of the response and return True if it failed
# or False if it succeeded.
pass
class LoginSpider(scrapy.Spider):
name = 'example.com'
start_urls = ['http://www.example.com/users/login.php']
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata={'username': 'john', 'password': 'secret'},
callback=self.after_login
)
def after_login(self, response):
if authentication_failed(response):
self.logger.error("Login failed")
return
# continue scraping with authenticated session...
finally, you can have a look at how too add proxies to your scrapy middleware as per this example (zyte are the guys who wrote scrapy) "How to set up a custom proxy in Scrapy?"
I am trying to catch unhandled exceptions at global level. So somewhere in main.py file I have the below:
#app.exception_handler(Exception)
async def exception_callback(request: Request, exc: Exception):
logger.error(exc.detail)
But the above method is never executed. However, if I write a custom exception and try to catch it (as shown below), it works just fine.
class MyException(Exception):
#some code
#app.exception_handler(MyException)
async def exception_callback(request: Request, exc: MyException):
logger.error(exc.detail)
I have gone through Catch exception type of Exception and process body request #575. But this bug talks about accessing request body. After seeing this bug, I feel it should be possible to catch Exception.
FastAPI version I am using is: fastapi>=0.52.0.
Thanks in advance :)
Update
There are multiple answers, I am thankful to all the readers and authors here.
I was revisiting this solution in my application. Now I see that I needed to set debug=False, default it's False, but I had it set to True in
server = FastAPI(
title=app_settings.PROJECT_NAME,
version=app_settings.VERSION,
)
It seems that I missed it when #iedmrc commented on answer given by #Kavindu Dodanduwa.
In case you want to capture all unhandled exceptions (internal server error), there's a very simple way of doing it. Documentation
from fastapi import FastAPI
from starlette.requests import Request
from starlette.responses import Response
app = FastAPI()
async def catch_exceptions_middleware(request: Request, call_next):
try:
return await call_next(request)
except Exception:
# you probably want some kind of logging here
return Response("Internal server error", status_code=500)
app.middleware('http')(catch_exceptions_middleware)
Make sure you place this middleware before everything else.
You can do something like this. It should return a json object with your custom error message also works in debugger mode.
from fastapi import FastAPI
from fastapi.responses import JSONResponse
app = FastAPI()
#app.exception_handler(Exception)
async def validation_exception_handler(request, err):
base_error_message = f"Failed to execute: {request.method}: {request.url}"
# Change here to LOGGER
return JSONResponse(status_code=400, content={"message": f"{base_error_message}. Detail: {err}"})
Adding a custom APIRoute can be also be used to handle global exceptions. The advantage of this approach is that if a http exception is raised from the custom route it will be handled by default Starlette's error handlers:
from typing import Callable
from fastapi import Request, Response, HTTPException, APIRouter, FastAPI
from fastapi.routing import APIRoute
from .logging import logger
class RouteErrorHandler(APIRoute):
"""Custom APIRoute that handles application errors and exceptions"""
def get_route_handler(self) -> Callable:
original_route_handler = super().get_route_handler()
async def custom_route_handler(request: Request) -> Response:
try:
return await original_route_handler(request)
except Exception as ex:
if isinstance(ex, HTTPException):
raise ex
logger.exception("uncaught error")
# wrap error into pretty 500 exception
raise HTTPException(status_code=500, detail=str(ex))
return custom_route_handler
router = APIRouter(route_class=RouteErrorHandler)
app = FastAPI()
app.include_router(router)
Worked for me with fastapi==0.68.1.
More on custom routes: https://fastapi.tiangolo.com/advanced/custom-request-and-route/
It is a known issue on the Fastapi and Starlette.
I am trying to capture the StarletteHTTPException globally by a following simple sample.
import uvicorn
from fastapi import FastAPI
from starlette.requests import Request
from starlette.exceptions import HTTPException as StarletteHTTPException
from starlette.responses import JSONResponse
app = FastAPI()
#app.exception_handler(StarletteHTTPException)
async def exception_callback(request: Request, exc: Exception):
print("test")
return JSONResponse({"detail": "test_error"}, status_code=500)
if __name__ == "__main__":
uvicorn.run("test:app", host="0.0.0.0", port=1111, reload=True)
It works. I open the browser and call the endpoint / and try to access http://127.0.0.1:1111/ , it will return the json {"detail":"test_error"} with HTTP code "500 Internal Server Error" .
However, when I only changed StarletteHTTPException to Exception in the #app.exception_handler,
import uvicorn
from fastapi import FastAPI
from starlette.requests import Request
from starlette.exceptions import HTTPException as StarletteHTTPException
from starlette.responses import JSONResponse
app = FastAPI()
#app.exception_handler(Exception)
async def exception_callback(request: Request, exc: Exception):
print("test")
return JSONResponse({"detail": "test_error"}, status_code=500)
if __name__ == "__main__":
uvicorn.run("test:app", host="0.0.0.0", port=1111, reload=True)
The method exception_callback could not capture the StarletteHTTPException when I accessed the http://127.0.0.1:1111/ . It reported 404 error.
The excepted behaviour should be: StarletteHTTPException error could be captured by the method exception_handler decorated by Exception because StarletteHTTPException is the child class of Exception.
However, it is a known issue reported in Fastapi and Starlette
https://github.com/tiangolo/fastapi/issues/2750
https://github.com/tiangolo/fastapi/issues/2683
https://github.com/encode/starlette/issues/1129
So we are not able to acheieve the goal currently.
First I invite to get familiar with exception base classes in python. You can read them in the document Built-in Exceptions
Secondly, read through fastApi default exception overriding behaviour Override the default exception handlers
What you must understand is that #app.exception_handler accepts any Exception or child classes derived from Exception. For example RequestValidationError is a subclass of python built in ValueError which itself a subclass of Exception.
So you must design your own exceptions or throw available exceptions with this background. I guess what went wrong is with your logger logger.error(exc.detail) by either not having a detail field or not having a proper logger configuration.
Sample code :
#app.get("/")
def read_root(response: Response):
raise ArithmeticError("Divide by zero")
#app.exception_handler(Exception)
async def validation_exception_handler(request, exc):
print(str(exc))
return PlainTextResponse("Something went wrong", status_code=400)
Output :
A stdout entry and a response with Something went wrong
I found a way to catch exceptions without the "Exception in ASGI application_" by using a middleware. Not sure if this has some other side effect but for me that works fine! #iedmrc
#app.middleware("http")
async def exception_handling(request: Request, call_next):
try:
return await call_next(request)
except Exception as exc:
log.error("Do some logging here")
return JSONResponse(status_code=500, content="some content")
I was searching for global handler for fast api for giving custome message for 429 status code i found and implemented, working fine for me
#app.exception_handler(429)
async def ratelimit_handler(request: Request, exc: Exception):
return JSONResponse({'message': "You have exceeded your request quota. Kindly try after some time.", 'status': 'failed'})
I want to understand how I can catch an HTTPException raised by flask.abort while using a test_request_context in a test for the calling method only.
# example.py
import flask
#api.route('/', methods=['POST'])
def do_stuff():
param_a = get_param()
return jsonify(a=param_a)
# check if request is json, return http error codes otherwise
def get_param():
if flask.request.is_json():
try:
data = flask.request.get_json()
a = data('param_a')
except(ValueError):
abort(400)
else:
abort(405)
# test_example.py
from app import app # where app = Flask(__name__)
from example import get_param
import flask
def test_get_param(app):
with app.test_request_context('/', data=flask.json.dumps(good_json), content_type='application/json'):
assert a == get_param()
In the get_param method above, I try to abort if the is_json() or the get_json() fail. To test this, I pass test_request_context without a content_type and, based on this blog and this answer, I tried adding a nested context manager like so:
# test_example.py
from app import app # where app = Flask(__name__)
from example import get_param
from werkzeug.exceptions import HTTPException
import flask
def test_get_param_aborts(app):
with app.test_request_context('/', data=flask.json.dumps('http://example', 'nope'), content_type=''):
with pytest.raises(HTTPException) as httperror:
get_param()
assert 405 == httperror
but I get a assert 405 == <ExceptionInfo for raises contextmanager> assertion error.
Can someone please explain this and suggest a way to test the abort in this get_param method?
Update:
Based on #tmt's answer, I modified the test. However, even though the test passes, while debugging I notice that the two assertions are never reached!
# test_example.py
from app import app # where app = Flask(__name__)
from example import get_param
from werkzeug.exceptions import HTTPException
import flask
def test_get_param_aborts(app):
with app.test_request_context('/', data=flask.json.dumps('http://example', 'nope'), content_type=''):
with pytest.raises(HTTPException) as httperror:
get_param() # <-- this line is reached
assert 405 == httperror.value.code
assert 1 ==2
httperror is an instance of ExceptionInfo which is pytest's own class that describes the exception. Once it happens, httperror would also contain value property which would be the instance of the HTTPException itself. If my memory is correct HTTPException contains code property that equals to the HTTP status code so you can use it to do the assertion:
# test_example.py
from app import app
from example import get_param
from werkzeug.exceptions import HTTPException
import flask
def test_get_param_aborts(app):
with app.test_request_context('/', data=flask.json.dumps(), content_type=''):
with pytest.raises(HTTPException) as httperror:
get_param()
assert 405 == httperror.value.code
Notes:
get_param() needs to be called within pytest.raises() context manager.
The assertion needs to be done outside of the context manager because once an exception is raised the context ends.
I don't know if pytest.raise is your typo or if it really existed in older versions of pytest. AFAIK it should be pytest.raises.
The Python3 fetch_token method in this library does not check the response status before consuming the response. If the API call it makes fails, then the response will be invalid and the script crashes. Is there something I can set so that an exception will be raised on a non-success response before the library can read the response?
import requests
from requests.auth import HTTPBasicAuth
from requests_oauthlib import OAuth2Session
from oauthlib.oauth2 import BackendApplicationClient
from oauthlib.oauth2 import OAuth2Error
AUTH_TOKEN_URL = "https://httpstat.us/500" # For testing
AUTH = HTTPBasicAuth("anID", "aSecret")
CLIENT = BackendApplicationClient(client_id="anID")
SCOPES = "retailer.orders.write"
MAX_API_RETRIES = 4
class MyApp:
def __init__(self):
"""Initialize ... and obtain initial auth token for request"""
self.client = OAuth2Session(client=CLIENT)
self.client.headers.update(
{
"Content-Type": "application/json"
}
)
self.__authenticate()
def __authenticate(self):
"""Obtain auth token."""
server_errors = 0
# This needs more work. fetch_token is not raising errors but failing
# instead.
while True:
try:
self.token = self.client.fetch_token(
token_url=AUTH_TOKEN_URL, auth=AUTH, scope=SCOPES
)
break
except (OAuth2Error, requests.exceptions.RequestException) as e:
server_errors = MyApp.__process_retry(
server_errors, e, None, MAX_API_RETRIES
)
#staticmethod
def __process_retry(errors, exception, resp, max_retries):
# Log and process retries
# ...
return errors + 1
MyApp() # Try it out
You can add a "compliance hook" that will be passed the Response object from requests before the library attempts to parse it, like so:
def raise_on_error(response):
response.raise_for_status()
return response
self.client.register_compliance_hook('access_token_response', raise_on_error)
Depending on exactly when you may get errors, you might want to do this with 'refresh_token_response' and/or 'protected_request' as well. See the docstring for the register_compliance_hook method for more info.