I have seen the documentation but i am not able to make it working.If anyone could explain me more clearly how to create a custom 403 page in pyramid,it would be great
Thanks
You can add a renderer to your #forbidden_view_config() decorator:
#forbidden_view_config(renderer='myforbiddentemplate.mako')
Or, you could probably also do some sort of custom views, too:
from pyramid.httpexceptions import (
HTTPFound,
HTTPNotFound,
HTTPForbidden,
HTTPBadRequest,
HTTPInternalServerError
)
#view_config(context=HTTPNotFound, renderer='HTTPNotFoundPage.mako')
def my_view(request):
Just decore your view with forbidden_view_config:
#view_config(route_name='login', renderer='login-page.mako')
#forbidden_view_config(renderer='login-page.mako')
def login_view(request):
return {}
from pyramid.view import forbidden_view_config
#forbidden_view_config(renderer='your_template_name')
def forbidden_view(request):
request.response.status = 403
return {}
Related
I need to make an initial call to a service before I start my scraper (the initial call, gives me some cookies and headers), I decided to use InitSpider and override the init_request method to achieve this. however I also need to use start_requests to build my links and add some meta values like proxies and whatnot to that specific spider, but I'm facing a problem. whenever I override start_requests, my crawler doesn't call init_request anymore and I can not do the initialization and in order to get init_request working is to not override the start_requests method which is impossible in my case. any suggestions or possible solutions to my code:
class SomethingSpider(InitSpider):
name = 'something'
allowed_domains = ['something.something']
aod_url = "https://something?="
start_urls = ["id1","id2","id3"]
custom_settings = {
'DOWNLOAD_FAIL_ON_DATALOSS' : False,
'CONCURRENT_ITEMS': 20,
'DOWNLOAD_TIMEOUT': 10,
'CONCURRENT_REQUESTS': 3,
'COOKIES_ENABLED': True,
'CONCURRENT_REQUESTS_PER_DOMAIN': 20
}
def init_request(self):
yield scrapy.Request(url="https://something",callback=self.check_temp_cookie, meta={'proxy': 'someproxy:1111'})
def check_temp_cookie(self, response):
"""Check the response returned by a login request to see if we are
successfully logged in.
"""
if response.status == 200:
print("H2")
# Now the crawling can begin..
return self.initialized()
else:
print("H3")
# Something went wrong, we couldn't log in, so nothing happens.
def start_requests(self):
print("H4")
proxies = ["xyz:0000","abc:1111"]
for url in self.start_urls:
yield scrapy.Request(url=self.aod_url+url, callback=self.parse, meta={'proxy': random.choice(proxies)})
def parse(self, response):
#some processing happens
yield {
#some data
}
except Exception as err:
print("Connecting to...")
Spiders page (generic spiders section) on official scrapy docs doesn't have any mention of InitSpider You are trying to use.
InitSpider class from https://github.com/scrapy/scrapy/blob/2.5.0/scrapy/spiders/init.py written ~10 years ago (at that... ancient versions of scrapy start_requests method worked completely differently).
From this perspective I recommend You to not use undocumented and probably outdated InitSpider.
On current versions of scrapy required functionality can be implemented using regular Spider class:
import scrapy
class SomethingSpider(scrapy.Spider):
...
def start_requests(self):
yield scrapy.Request(url="https://something",callback=self.check_temp_cookie, meta={'proxy': 'someproxy:1111'})
def check_temp_cookie(self, response):
"""Check the response returned by a login request to see if we are
successfully logged in.
"""
if response.status == 200:
print("H2")
# Now the crawling can begin..
...
#Schedule next requests here:
for url in self.start_urls:
yield scrapy.Request(url=self.aod_url+url, callback=self.parse, ....})
else:
print("H3")
# Something went wrong, we couldn't log in, so nothing happens.
def parse(self, response):
...
If you are looking speicfically at incorporating logging in then I would reccomend you look at Using FormRequest.from_response() to simulate a user login in the scrapy docs.
Here is the spider example they give:
import scrapy
def authentication_failed(response):
# TODO: Check the contents of the response and return True if it failed
# or False if it succeeded.
pass
class LoginSpider(scrapy.Spider):
name = 'example.com'
start_urls = ['http://www.example.com/users/login.php']
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata={'username': 'john', 'password': 'secret'},
callback=self.after_login
)
def after_login(self, response):
if authentication_failed(response):
self.logger.error("Login failed")
return
# continue scraping with authenticated session...
finally, you can have a look at how too add proxies to your scrapy middleware as per this example (zyte are the guys who wrote scrapy) "How to set up a custom proxy in Scrapy?"
Trying to figure out the right mechanism to use here.
I want to modify the flask request coming in every time.
I think the request is immutable, so I am trying to figure out if this mechanism exists.
Basically, I want to append a string onto the end of the request coming in.
I can hook into the request and the right time in a before_request handler like this:
#app.before_app_request
def before_request_custom():
# Get the request
req = flask.request
method = str(req.method)
if method == "GET":
# Do stuff here
pass
But I am not sure what to actually do to add this in, and don't see a way to accomplish it...I guess i could redirect, but that seems silly in this case. Any ideas?
The request object is immutable (https://werkzeug.palletsprojects.com/en/1.0.x/wrappers/#base-wrappers), but request.args or request.form can be set from ImmutableOrderedMultiDict to just OrderedMultiDict using Subclassing on Flask (https://flask.palletsprojects.com/en/1.1.x/patterns/subclassing/). Here's an example of how you could add that filter[is_deleted]=False URL param:
from flask import Flask, request, Request
from werkzeug.datastructures import OrderedMultiDict
class MyRequest(Request):
parameter_storage_class = OrderedMultiDict
class MyApp(Flask):
def __init__(self, import_name):
super(MyApp, self).__init__(import_name)
self.before_request(self.my_before_method)
def my_before_method(self):
if "endpoint" in request.base_url:
request.args["filter[is_deleted]"] = "False"
app = MyApp(__name__)
app.request_class = MyRequest
#app.route('/endpoint/')
def endpoint():
filter = request.args.get('filter[is_deleted]')
return filter
This way you can modify request.args before you actually send the request.
How about this?
from flask import g
#app.before_request
def before_request():
# Get the request
req = flask.request
method = str(req.method)
if method == "GET":
g.my_addon = "secret sauce"
return None
Then, g.my_addon is available in every view function:
from flask import g
#app.route('/my_view')
def my_view():
if g.my_addon == "secret sauce":
print('it worked!')
Using test_request_context() you can make the trick.
Related: https://flask.palletsprojects.com/en/1.1.x/quickstart/#accessing-request-data
The documentation example for a simple restful api is:
from flask import Flask, request
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
todos = {}
class TodoSimple(Resource):
def get(self, todo_id):
return {todo_id: todos[todo_id]}
def put(self, todo_id):
todos[todo_id] = request.form['data']
return {todo_id: todos[todo_id]}
api.add_resource(TodoSimple, '/<string:todo_id>')
if __name__ == '__main__':
app.run(host="0.0.0.0",port="80",debug=True)
However, suppose I made a put request with a nested dictionary,ie {'data':{'fruit':'orange'}}. The TodoSimple would have request.form.to_dict() = {'data':'fruit'}. How can I work with the full nested dictionary?
You should probably use Schemas to achieve this goal. Take a good look at this first example of marshmallow docs:
https://marshmallow.readthedocs.io/en/3.0/
As flask-restful docs says:
The whole request parser part of Flask-RESTful is slated for removal
and will be replaced by documentation on how to integrate with other
packages that do the input/output stuff better (such as marshmallow).
I am trying to use aiohttp-jinja2 with app.add_subapp(). Unfortunately I am not able to figure out how to use {{ url('named_url') }} in case of subapps.
I followed the docs and now I am using it as following:
main.py
app.router.add_route('GET', '/', Index)
app.add_subapp(r'/api/v1/todo', todos_app)
app['todos_app'] = todos_app
index_handler.py
import aiohttp_jinja2
from aiohttp import web
class Index(web.View):
"""Index page for the server."""
#aiohttp_jinja2.template('index.html')
async def get(self):
"""Return a simple page with urls."""
todos_app = self.request.app['todos_app']
return {
'new_todo_url': todos_app.router['new_todo'].url_for(),
}
index.html
<li class="list-group-item">New Todo</li>
I was wondering if there is a more efficient/better way to achieve this, by just using url('subapp_named_resource') ?
I'm using Pyramid (1.5.7) + traversal and following the documentation I've tried all possible ways to get the "Not found exception view" working.
from pyramid.view import notfound_view_config,forbidden_view_config, view_config
#notfound_view_config(renderer="error/not_found.jinja2")
def not_found_view(request):
request.response.status = 404
return {}
#forbidden_view_config(renderer="error/forbidden.jinja2")
def forbidden_view(request):
return {}
Using contexts:
from pyramid.view import view_config
from pyramid.httpexceptions import HTTPForbidden, HTTPUnauthorized
#view_config(context=HTTPNotFound, renderer="error/not_found.jinja2")
def not_found_view(request):
request.response.status = 404
return {}
#view_config(context=HTTPForbidden, renderer="error/forbidden.jinja2")
def forbidden_view(request):
return {}
I'm using the Scan mode, but I've tried also adding a custom function to the configuration:
def main(globals, **settings):
config = Configurator()
config.add_notfound_view(notfound)
Not luck either, all time getting the following unhandled exception:
raise HTTPNotFound(msg)
pyramid.httpexceptions.HTTPNotFound: /example-url
Ouch... My bad! I was using a tween which was preventing Pyramid to load the Exceptions:
def predispatch_factory(handler, registry):
# one-time configuration code goes here
def predispatch(request):
# code to be executed for each request before
# the actual application code goes here
response = handler(request)
# code to be executed for each request after
# the actual application code goes here
return response
return predispatch
Still I don't know why but after removing this tween all seems to work as expected.