Python Bottle framework form tutorials always dead end by spitting out raw text of data obtained from a POST method. But that’s not how web sites work. I want to cycle right back to the same HTML/URL page to interactively display the number of items just added to a cart then let the user add more items as needed. I tried global variables, a dictionary, or Bottle environmental variables to keep track of my cart item count between page reloads, and all work fine on a localhost server but act sporadically when placed on Namecheap. I redirect back to the original URL either by an HTML meta refresh, by the Bottle redirect command, or simply to again call the Python function Bottle routes to that URL. For all of these variations Namecheap with a cPanel Python 3.7 installation the cart item count display jumps around up and down as I try to add more items in an interactive cycle. At least one of the versions does keep a running tally but most results jump back down to a small number or just 1 again. The code below is my passenger_wsgi.py file.
I just changed from Bottle to Flask with essentially the same code but in Flask flavor and it’s doing the exact same thing from cPanel-based Python on Namecheap.
Are multiple threads being used that can’t contain the same global or even system environmental variable values over time?
import bottle
from bottle import route, request, post, debug
cart_count = 0
#route('/ItemA')
def add():
global cart_count
debug(True)
return f'''
<html>
<head>
<title>Velkommen</title>
</head>
<body>
<center>
<p>{cart_count} ITEMS</p>
<form action="/ItemA" method="post">
<input name="item_count" type="text" size="1" value="1"/>
$120
<input type="submit">
</form>
</center>
</body>
</html>
'''
#post('/ItemA')
def do_add():
global cart_count
debug(True)
item_count = int(request.forms.get('item_count'))
cart_count += item_count
return add()
# passenger hook
def application(environ, start_response):
return bottle.app().wsgi(environ,start_response)
UPDATE: the Flask debugger won’t load any page on iPhone browsers so I returned to Bottle and got it to work via cookies as long as my returning to the same item addition page was done not by merely calling the internal Python function that creates that page but by using a Bottle redirect for otherwise the cookies lagged one user step behind meaning it lost the first step altogether.
Screenshot
import bottle
from bottle import route, run, static_file, template, request, get, post, debug, error, response, redirect
bugs = True
#route('/ItemA')
def addA():
debug(bugs)
if not request.cookies.a_count:
a_count = 0
response.set_cookie('a_count', str(a_count))
else:
a_count = request.cookies.get('a_count')
return f'''
<p style="color:#047CFC;">USA ONLY • 🛒 {str(a_count)} ITEMS $325
<button class='button'>VIEW CART</button>
FREE SHIPPING
<img src = "IMG_4248.JPG" alt = "ALT" height = "500" width = "500"/>
<form action='/ItemA' method="post">
<input name="item_count" type="text" size="1" value="1"/>
$120
<input class='button' value="ADD TO CART" type="submit"/>
'''
#post('/ItemA')
def do_add():
debug(bugs)
a_count = int(request.cookies.get('a_count'))
a_count += int(request.forms.get('item_count'))
response.set_cookie('a_count', str(a_count))
return redirect('/ItemA')
# passenger hook
def application(environ, start_response):
return bottle.app().wsgi(environ,start_response)
Related
my brain crashed.
I'm trying to get the ID of a span if specific text matches using BeautifulSoup, this because i need a number from the ID but the ID changes every time when searching for a new product but the product (CORRECT). Purpose of this is because when i have the number, 11 in this case, i can add it in another part of the code to scrape the information i need.
Example:
<span id="random-text-10-random-again">IGNORE</span>,
<span id="random-text-11-random-again">CORRECT</span>,
<span id="random-text-12-random-again">IGNORE</span>
Been reading documentation but i never seem to get right or not even remotely close. I'm aware how to pull the text (CORRECT) if i know the ID but not reversed.
Find_all() span items with required text and then get the id attribute and split() the attribute value with -
from bs4 import BeautifulSoup
html='''<span id="random-text-10-random-again">IGNORE</span>
<span id="random-text-11-random-again">CORRECT</span>
<span id="random-text-12-random-again">IGNORE</span>'''
soup=BeautifulSoup(html,'html.parser')
for item in soup.find_all('span',text='CORRECT'):
print(item['id'].split('-')[2])
It will print:
11
I prefer to use :contains to target the innerText by a specified value. Available for bs4 4.7.1+
from bs4 import BeautifulSoup as bs
html = '''
<span id="random-text-10-random-again">IGNORE</span>,
<span id="random-text-11-random-again">CORRECT</span>,
<span id="random-text-12-random-again">IGNORE</span>'''
soup = bs(html, 'lxml')
target = soup.select_one('span:contains("CORRECT")[id]')
if target is None:
print("Not found")
else:
print(target['id'].split('-')[2])
One of the final steps in my project is to get the price of a product , i got everything i need except the price.
Source :
<div class="prices">
<div class="price">
<div class="P01 tooltip"><span>Product 1</span></div>€<div class="encoded" data-price="bzMzlXaZjkxLjUxNA==">151.4</div>
</div>
<div class="price">
<div class="Po1plus tooltip"><span>Product 1 +</span></div>€<div class="encoded" data-price="MGMSKJDFsTcxLjU0NA==">184.4</div>
</div>
what i need to get is after the
==">
I don't know if there is some protection from the encoded part, but the clostest i get is returnig this <div class="encoded" data-price="bzMzlXaZjkxLjUxNA=="></div>
Don't know if is relevant i'm using "html.parser" for the parsing
PS. i'm not trying to hack anything, this is just a personal project to help me learn.
Edit: if when parsing the test i get no price, the other methods can get it without a different parser ?
EDIT2 :
this is my code :
page_soup = soup(pagehtml, "html.parser")
pricebox = page_soup.findAll("div",{ "id":"stationList"})
links = pricebox[0].findAll("a",)
det = links[0].findAll("div",)
det[7].text
#or
det[7].get_text()
the result is ''
With Regex
I suppose there are ways to do this using beautifulsoup, anyway here is one approach using regex
import regex
# Assume 'source_code' is the source code posted in the question
prices = regex.findall(r'(?<=data\-price[\=\"\w]+\>)[\d\.]+(?=\<\/div)', source_code)
# ['151.4', '184.4']
# or
[float(p) for p in prices]
# [151.4, 184.4]
Here is a short explanation of the regular expression:
[\d\.]+ is what we are actually searching: \d means digits, \. denotes the period and the two combined in the square brackets with the + means we want to find at least one digit/period
The brackets before/after further specify what has to precede/succeed a potential match
(?<=data\-price[\=\"\w]+\>) means before any potential match there must be data-price...> where ... is at least one of the symbols A-z0-9="
Finally, (?=\<\/div) means after any match must be followed by </div
With lxml
Here is an approach using the module lxml
import lxml.html
tree = lxml.html.fromstring(source_code)
[float(p.text_content()) for p in tree.find_class('encoded')]
# [151.4, 184.4]
"html.parser" works fine as a parser for your problem. As you are able to get this <div class="encoded" data-price="bzMzlXaZjkxLjUxNA=="></div> on your own that means you only need prices now and for that you can use get_text() which is an inbuilt function present in BeautifulSoup.
This function returns whatever the text is in between the tags.
Syntax of get_text() :tag_name.get_text()
Solution to your problem :
from bs4 import BeautifulSoup
data ='''
<div class="prices">
<div class="price">
<div class="P01 tooltip"><span>Product 1</span></div>€<div class="encoded" data-price="bzMzlXaZjkxLjUxNA==">151.4</div>
</div>
<div class="price">
<div class="Po1plus tooltip"><span>Product 1 +</span></div>€<div class="encoded" data-price="MGMSKJDFsTcxLjU0NA==">184.4</div>
</div>
'''
soup = BeautifulSoup(data,"html.parser")
# Searching for all the div tags with class:encoded
a = soup.findAll ('div', {'class' : 'encoded'})
# Using list comprehension to get the price out of the tags
prices = [price.get_text() for price in a]
print(prices)
Output
['151.4', '184.4']
Hope you get what you are looking for. :)
I'm trying to make sending naver email automation python scripts, and one thing that I can't do is upload file without SendKeys method(even have not to input tags(elements). and I searched in google there have only uploaded a file with input tags. plz, help me.
I'm tried to code by sendkeys but the error is selenium.common.exceptions,ElementNotInteractableException: Message: element not interactable
# Attachment
if attach:
attach = config.attachment
self.driver.find_element_by_xpath('//[#id="write_drag_here"]').send_keys('path in here')
print('attachment activated')
time.sleep(10)
else:
print('attachment deactivated')
How can I code this HTML
*HTML CODE:**
<a href="#" id="AddButton_fla" class="utx_btn_b _c1(mwAttach|clickUpload)
_stopDefault" onclick="ccr(this,'wrh.attach',event)" style="display: none;">
<span class="utx_inrbtn">내 PC</span></a>
<a href="#" id="AddButton_html5" class="utx_btn_b
_click(mwAttach|clickHtml5Upload) _stopDefault"
onclick="ccr(this,'wh5.attach',event)" style="display: inline-block;">
<span
class="utx_inrbtn">내 PC</span></a>
Note: Feel free to ask me again if you can't understand anything.
Thanks
If a script is interfering with it, one way I typically do it is to use screen coordinates to interact with the Element as a user would by using a macro program like AppRobotic, and simulate mouse movements and use SendKeys only after the Element is activated on the screen:
import win32com.client
x = win32com.client.Dispatch("AppRobotic.API")
from selenium import webdriver
# navigate to Instagram
driver = webdriver.Firefox()
driver.get('https://www.AttachmentWebsite.com')
# sleep 1 second
x.Wait(1000)
# Move mouse to coordinates 250, 500 on the screen
x.MoveCursor(250,500)
x.MouseLeftClick
# sleep 1 second
x.Wait(1000)
x.Type("my attachment")
i am new to web scraping and got an issue
I am using BeautifulSoup for scraping a webpage. I want to get nodes which have text in it.
I tried that using get_text() method like this
soup = BeautifulSoup(open('FAQ3.html'), "html.parser")
body = soup.find('body')
for i in body:
if type(i) != bs4.element.Comment and type(i)!= bs4.element.NavigableString :
if i.get_text():
print(i)
but get_text is giving node even if its child have text in it,
sample html:
<div>
<div id="header">
<script src="./FAQ3_files/header-home.js"></script>
</div>
<div>
<div>
this node contain text
</div>
</div>
</div>
while checking topmost div itself, it is returning the whole node as the innermost had text in it,
how to iterate over all nodes and filter only the nodes which actually have text in it?
I used depth-first search for this, it solved my use case
def get_text_bs4(self, soup, leaf):
if soup.name is not None:
if soup.string != None and soup.name != 'script':
if soup.text not in leaf:
leaf[soup.text] = soup
for child in soup.children:
self.get_text_bs4(child, leaf)
return leaf
This question already has answers here:
Converting Flask form data to JSON only gets first value
(2 answers)
Closed 4 years ago.
I need to capture multiple select form vlaue (generated from a MongoDB collection ) and POST via a Flask route to another MongoDB collection: recipes
Below is the relevant form div:
<form action="{{ url_for('insert_recipe') }}" method="POST" class="col s12">
...
<div class="input-field col s6 l6">
<i class="material-icons prefix">warning</i>
<select multiple id="allergen_name" name="allergenlist">
<option value="" disabled selected>Choose allergens</option>
{% for allergen in allergens %}
<option value="{{allergen.allergen_name}}">{{allergen.allergen_name}}</option>
{% endfor %}
</select>
</div>
</div>
...
</form>
I want to capture the selected options and POST them via Flask:
# Get all recipes
#app.route('/get_recipes')
def get_recipes():
return render_template("recipes.html",
recipes=mongo.db.recipes.find())
# Render HTML form
#app.route('/add_recipe')
def add_recipe():
return render_template('addrecipe.html',
users=mongo.db.users.find(),
allergens=mongo.db.allergens.find(),
cuisines=mongo.db.cuisine.find(),)
# Send the form
#app.route('/insert_recipe', methods=['POST'])
def insert_recipe():
recipes = mongo.db.recipes
recipes.insert(request.form.to_dict())
return redirect(url_for('get_recipes'))
However, only the first selected option is being captured and sent.
Any help would be appreciated.
EDIT:
When looking at: http://werkzeug.pocoo.org/docs/0.12/datastructures/#werkzeug.datastructures.MultiDict.to_dict
... relaised that I need to set to_dict(flat=false) in order to have all values of dict returned.
See EDIT above, the correct way is to:
# Send the form
#app.route('/insert_recipe', methods=['POST'])
def insert_recipe():
recipes = mongo.db.recipes
recipes.insert_one(request.form.to_dict(flat=False))
return redirect(url_for('get_recipes'))
Also, just found a duplicate as notified by #davidism :
Converting Flask form data to JSON only gets first value