I am trying to use aiohttp-jinja2 with app.add_subapp(). Unfortunately I am not able to figure out how to use {{ url('named_url') }} in case of subapps.
I followed the docs and now I am using it as following:
main.py
app.router.add_route('GET', '/', Index)
app.add_subapp(r'/api/v1/todo', todos_app)
app['todos_app'] = todos_app
index_handler.py
import aiohttp_jinja2
from aiohttp import web
class Index(web.View):
"""Index page for the server."""
#aiohttp_jinja2.template('index.html')
async def get(self):
"""Return a simple page with urls."""
todos_app = self.request.app['todos_app']
return {
'new_todo_url': todos_app.router['new_todo'].url_for(),
}
index.html
<li class="list-group-item">New Todo</li>
I was wondering if there is a more efficient/better way to achieve this, by just using url('subapp_named_resource') ?
Related
I am trying to scrape a website with multiple pages with the same url using scrapy-playwright.
the following script returned only the data of the second page and did not continue to the rest of the pages.
can anyone suggest how I can fix it?
import scrapy
from scrapy_playwright.page import PageMethod
from scrapy.crawler import CrawlerProcess
class AwesomeSpideree(scrapy.Spider):
name = "awesome"
def start_requests(self):
# GET request
yield scrapy.Request(
url=f"https://www.cia.gov/the-world-factbook/countries/" ,
callback = self.parse,
meta=dict(
playwright = True,
playwright_include_page = True,
playwright_page_methods = {
"click" : PageMethod('click',selector = 'xpath=//div[#class="pagination-controls col-lg-6"]//span[#class="pagination__arrow-right"]'),
"screenshot": PageMethod("screenshot", path=f"step1.png", full_page=True)
},
)
)
async def parse(self, response):
page = response.meta["playwright_page"]
await page.close()
print("-"*80)
CountryLst = response.xpath("//div[#class='col-lg-9']")
for Country in CountryLst:
yield {
"country_link": Country.xpath(".//a/#href").get()
}
I see you are trying to fetch URLs of countries from above mentioned URL.
if you inspect the Network tab you can see there is one request to one JSON data API. You can fetch all countries URL's from this url
after that if you still want scrap more data from scraped URL's then you can easily scrap because that data is static so there will be no need to use playwright.
Have a good day :)
I have the following tree:
apps
templatetags
pyos.py
templates
index.html
In pyos.py I have the following code:
import os
from django import template
register = template.Library()
#register.simple_tag
def listdir(path):
d = os.listdir
return d(path)
def test():
pass
Then in index.html, inside templates I have the following already working:
{%extends 'principal.html'%}
<div>
{%load pyos%}
{%listdir 'C:/'%}
</div>
I need to pass some static path as argument, is there any way to do something like that?
<div>
{%load pyos%}
{%load static%}
{%listdir {%static 'img/imagesFolder%}%}
</div>
Since i am not able to login to https://www.duif.nl/login, i tried many different methods like selenium, which i successfully logged in, but didnt manage to start crawling.
Now i tried my luck with scrapy-splash, but i cant login :(
If i render the loginpage with splash, i see following picture:
Well, there should be a loginform, like username and password, but scrapy cant see it?
Im sitting here like a week in front of that loginform and losing my will to live..
My last question didnt even get one answer, now i try it again.
here is the html code of the login-form:
When i login manual, i get redirected to "/login?returnUrl=", where i only have these form_data:
My Code
# -*- coding: utf-8 -*-
import scrapy
from scrapy_splash import SplashRequest
from scrapy.spiders import CrawlSpider, Rule
from ..items import ScrapysplashItem
from scrapy.http import FormRequest, Request
import csv
class DuifSplash(CrawlSpider):
name = "duifsplash"
allowed_domains = ['duif.nl']
login_page = 'https://www.duif.nl/login'
with open('duifonlylinks.csv', 'r') as f:
reader = csv.DictReader(f)
start_urls = [items['Link'] for items in reader]
def start_requests(self):
yield SplashRequest(
url=self.login_page,
callback=self.parse,
dont_filter=True
)
def parse(self, response):
return FormRequest.from_response(
response,
formdata={
'username' : 'not real',
'password' : 'login data',
}, callback=self.after_login)
def after_login(self, response):
accview = response.xpath('//div[#class="c-accountbox clearfix js-match-height"]/h3')
if accview:
print('success')
else:
print(':(')
for url in self.start_urls:
yield response.follow(url=url, callback=self.parse_page)
def parse_page(self, response):
productpage = response.xpath('//div[#class="product-details col-md-12"]')
if not productpage:
print('No productlink', response.url)
for a in productpage:
items = ScrapysplashItem()
items['SKU'] = response.xpath('//p[#class="desc"]/text()').get()
items['Title'] = response.xpath('//h1[#class="product-title"]/text()').get()
items['Link'] = response.url
items['Images'] = response.xpath('//div[#class="inner"]/img/#src').getall()
items['Stock'] = response.xpath('//div[#class="desc"]/ul/li/em/text()').getall()
items['Desc'] = response.xpath('//div[#class="item"]/p/text()').getall()
items['Title_small'] = response.xpath('//div[#class="left"]/p/text()').get()
items['Price'] = response.xpath('//div[#class="price"]/span/text()').get()
yield items
In my "prework", i crawled every internal link and saved it to a .csv-File, where i analyse which of the links are product links and which are not.
Now i wonder, if i open a link of my csv, it opens an authenticated session or not?
I cant find no cookies, this is also strange to me
UPDATE
I managed to login successfully :-) now i only need to know where the cookies are stored
Lua Script
LUA_SCRIPT = """
function main(splash, args)
splash:init_cookies(splash.args.cookies),
splash:go("https://www.duif.nl/login"),
splash:wait(0.5),
local title = splash.evaljs("document.title"),
return {
title=title,
cookies = splash:get_cookies(),
},
end
"""
I don't think using Splash here is the way to go, as even with a normal Request the form is there: response.xpath('//form[#id="login-form"]')
There are multiple forms available on the page, so you have to specify which form you want to base yourself on to make a FormRequest.from_response. Best specify the clickdata as well (so it goes to 'Login', not to 'forgot password'). In summary it would look something like this:
req = FormRequest.from_response(
response,
formid='login-form',
formdata={
'username' : 'not real',
'password' : 'login data'},
clickdata={'type': 'submit'}
)
If you don't use Splash, you don't have to worry about passing cookies - this is taken care of by Scrapy. Just make sure you don't put COOKIES_ENABLED=False in your settings.py
There is a simple flask app which writes statistics-table from db to a page. How can I plot plotly.express chart on this page?
Code for chart that I want to integrate to a flask app: (took from https://plotly.com/python/time-series/)
# Using plotly.express
import plotly.express as px
import pandas as pd
df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/...')
fig = px.line(df, x='Date', y='AAPL.High')
fig.show()
There need to be more answers floating around out there that actually show how to do this without using dash. I will share my working example of using plotly.express and flask. I'll cut out most of the data work and figure building to just show what you need to do.
Imports needed
You'll need these in addition to your usual px and flask imports.
from plotly import utils
from json import dumps
Short Explanation: JSON and Plotly.js are key
I use pandas to get a dataframe in a function called get_data and get a scatter plot with lines connected with a function called get_lfig. The only important thing to note here is that get_lfig is returning a figure generated from px.scatter() but it can be any figure. Now the trick here is to turn your figure into a JSON and use it on the template side somewhere, you don't do much in python.
Example 1: Create the fig, JSON and pass it to the template
Altogether it looks something like this
from flask import Flask, render_template
from extensions import get_data, get_lfig
from plotly import utils
from json import dumps
app = Flask(__name__)
#app.route('/')
def home():
# function that get data, private and public sets
days=2
priv_data, pub_data= get_data(hours=24*days)
# function that returns a px.scatter figure
all_pub_fig = get_lfig(pub_data)
# turn the figure into a JSON then pass it to the template
all_pub_json = dumps(all_pub_fig, cls=utils.PlotlyJSONEncoder)
return render_template('graph.html', pub_lines_JSON=all_pub_json)
if __name__ == '__main__':
app.run(debug=True, port=8080, host='0.0.0.0')
On the template side, you want to include the src for plotly.js somewhere and then just use the Plotly.plot() function to populate your graph in a div.
<!-- somewhere up top -->
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
...
<h3>Line Graph Representation</h3>
<!-- scatter plot goes in this div -->
<div id='all-pub-graph'></div>
<script type="text/javascript">
// here is where the JSON gets plugged in via JS
var the_pubs_graph = {{pub_lines_JSON | safe}};
// you target the graph div id in the first arg,
// put your graph in the second, and set the third as {}
Plotly.plot("all-pub-graph", the_pubs_graph, {});
</script>
Example 2: Create the JSON, send it to JS fetch request
Knowing you just want a JSON of your figure, you can take this further and handle it in all JS without having to pass it to the template directly.
Here I handle a post request from on-page selectors that make an api call every time they're changed. The get_lfig function now returns a JSON of the figure instead of a figure object.
#app.route('/get-graphs', methods=['POST'])
def get_graphs():
if request.method == 'POST':
# This whole block is just form handling and data stuff
form = dict(request.form)
agg_func = form['agg_func']
days = int(form['days'])
interval = int(form['interval'])
if (len(agg_func) > 0) and (interval != 0):
pub, priv = get_data(hours=24*days, interval=interval, agg_func=agg_func)
else:
pub, priv = get_data(hours=24*days)
# get JSON figures from the data
pub_f, priv_f = get_lfig(pub), get_lfig(priv)
return {'public': pub_f, 'private': priv_f}
On the template side I use an event listener attached to a form so every time I make a change the graphs get updated. I still need the JS function to be able to find the url, so I pass it to the function using url_for() since utils.js isn't being rendered and can't take advantage of that same template functionality.
<div id="selectors">
<form id="graph-selectors" method="post" onchange="selector_changed('{{url_for('get_graphs')}}')">
<label for="day-selector">Days</label>
<select name="days" id="day-selector">
{% for opt in range(1,10) %}
<option value={{opt}}>{{opt}}</option>
{% endfor %}
</select>
<label for="function-selector">Aggregate Func</label>
<select name="agg_func" id="function-selector">
{% for opt in ['','mean','sum','min','max','std','count'] %}
<option value="{{opt}}">{{opt}}</option>
{% endfor %}
</select>
<label for="interval-selector">Aggregate Interval</label>
<select name="interval" id="interval-selector">
{% for opt in [0, 5, 10, 30, 60] %}
<option value={{opt}}>{{opt}} minutes</option>
{% endfor %}
</select>
</form>
</div>
<div id="content">
<div id="graph-section"></div>
</div>
<script src="{{ url_for('static', filename='utils.js') }}"></script>
then finally, the selector_changed function that makes an API call is stored in my utils.js and looks like this
// I just use this so it clears existing graphs between changes
const clearChildren = (parent) => {
while (parent.lastChild) {
parent.removeChild(parent.lastChild);
}
}
async function selector_changed(gUrl) {
// get the form data
var form_data = new FormData(document.querySelector('form#graph-selectors'))
// send it to get_graphs()
let response = await fetch(gUrl, {
method: "POST",
body: form_data
});
// get_graphs() returns the figure's JSON
let graphJSONs = await response.json();
// declare and clear the target area graphs will go in
var target_area = document.getElementById('graph-section');
clearChildren(target_area);
// create the public server graph section
var pub_area = document.createElement('div');
var pub_header = document.createElement('h3');
var pub_graph = JSON.parse(graphJSONs['public']);
pub_header.textContent = "Public Servers";
pub_area.id = 'public-graphs';
// create the private server graph section
var priv_area = document.createElement('div');
var priv_header = document.createElement('h3');
var priv_graph = JSON.parse(graphJSONs['private']);
priv_header.textContent = "Internal Servers";
priv_area.id = 'private-graphs';
// add everything to the page
target_area.append(pub_area); // start w/the divs
target_area.append(priv_area);
Plotly.plot(pub_area.id, pub_graph, {}); // then add the graphs
Plotly.plot(priv_area.id, priv_graph, {});
pub_area.prepend(pub_header); // then add the headers
priv_area.prepend(priv_header);
}
This is a lot of code, but I wanted to show two ways to handle this which are:
Creating a JSON and passing it to the template directly, and
Handling it as an API call that responds to fetch requests.
The second option is faster and you don't have to refresh the entire page every time, the first option was just showing how to do it with as little code as possible. Either way there should be enough here to modify to your needs and for my future reference. (:
i think to do this is a little more complicated to just call a fig.show(), take a fast look in the ploty lib i found this packet import dash_html_components as html with this you can return a html with your chart to put in web site,
from flask import Flask
app = Flask(__name__)
import dash_core_components as dcc
import dash_html_components as html
import plotly.express as px
import pandas as pd
#app.route('/chart')
def chart():
df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/...')
fig = px.line(df, x='Date', y='AAPL.High')
return html.Div([dcc.Graph(figure=fig)])
I am new to vue.js and django also,
Need some help on below issue.
I have created an api in which I am trying to fetch the data from one DB table and send the same to vue.js page to have check on it.
I am getting error "{"detail":"Method \"GET\" not allowed."}"
I tried it in multiple ways like sending the data in JSON form or as a array but still getting the same issuse
Model.py below are the models for deserializing and for the table I am using to fetch the data
def deserialize_Issue(issue):
"""Deserialize user instance to JSON."""
return {id': issue.id, 'IName': issue.IName, 'IDesc': issue.IDesc, 'ITeam': issue.ITeam,'IStat': issue.IStat, 'IChat': issue.IChat, 'Iuri': issue.Iuri, 'Ibridge': issue.Ibridge, 'Ibridgedetails': issue.Ibridgedetails }
class Issue(TrackableDateModel):
IName=models.CharField(max_length=500)
IDesc=models.CharField(max_length=1000)
ITeam =models.CharField(max_length=500)
IStat=models.CharField(max_length=250)
IChat=models.CharField(max_length=2,default='NA')
Iuri=models.URLField(default='NA')
Ibridge=models.CharField(max_length=2,default='NA')
Ibridgedetails=models.CharField(max_length=20,default='NA')
def __str__(self):
return self.IName, self.IStat
IN my view.py file I have created the calls and the get method to fetch the data from the table.
class IssuesDetailsView(APIView):
def get(self, request, *args, **kwargs):
DETAILS='Issue.objects.all()'
serializer=deserialize_Issue(DETAILS,many=True)
return Response ({'Issues': serializer.data})
abc.vue in vue.js file trying to acees the get the data
fetchIssueDetails () {
$.get(`http://127.0.0.1:8000/api/chats/`, (data) => {
this.Issues = data.Issues
})
.fail((response) => {
alert(response.responseText)
})
}
I want to read the table data in the vue.js file in for loop
Here's the urls.py you've provided me in the comments:
from django.contrib import admin
from django.urls import path
from . import views
urlpatterns = [
path('chats/', views.ChatSessionView.as_view()),
path('chats/<uri>/', views.ChatSessionView.as_view()),
path('chats/<uri>/messages/', views.ChatSessionMessageView.as_view()),
]
Notice how the url /api/chats/ is tied to views.ChatSessionView, not IssuesDetailsView. Yet in your Javascript you are making the request to /api/chats/.
So one way to resolve your issue is
path('chats/', views.IssuesDetailsView.as_view()),