Flask Assign different ID to different website visitor at the same time and write their input information into json file - multithreading

I am using FLASK to create a series of websites for doing a survey, I want to run the server once and invite multiple participants to help me fill the form on the webpage.
1.
I am aiming for assigning a unique ID (currently using python UUID) to different visitors and save their input info into different JSON files. (eg,. A's id is 00001, after he submiting the form, the info will be saved in the file named 00001.json, B's info will be saved in 00002.json). For now, I only create a UUID when the server start running, every visitor's information will be rewritten in the same file, (what I want is n visitors will produce n files).
2.
I want to show different survey questions to different visitors.
I am quite confused about if it is related to Multi-thread? Session? Cookie? Flask-login.
Hope someone can guide me on this. Thanks a lot!
assignmentID = str(uuid.uuid1())
jsonFileName = "upload/" + assignmentID + ".json"
jsonSurvey = "upload/" + assignmentID + "_survey.json"
#it ideally should randomly select different info/question no. to different user
Idx = random.sample(range(0,10), 5)
#app.route("/")
def index():
return render_template("surveyPage.html", data=Idx)
# write input information from webpage to JSON files, each visitor ideally should have their own JSON file.
#app.route("/record")
def recordData():
if request.method == 'POST':
print("READING FORM")
with open(jsonSurvey, 'w') as f:
json.dump(request.form, f)
f.close()
if __name__ == "__main__":
app.config['JSON_SORT_KEYS'] = False
app.run(debug = True)
# app.config["TEMPLATES_AUTO_RELOAD"] = True

You are defining assignmentID, jsonFileName, jsonSurvey and Idx outside of the request handlers which means they will be set once: when the server starts. jsonSurvey will therefore have the same value for each request which means that every new survey you store will overwrite the previous one.
The solution is to move the definition into the request handlers so that you genereate new values on each request.
#app.route("/")
def index():
Idx = random.sample(range(0,10), 5)
return render_template("surveyPage.html", data=Idx)
#app.route("/record")
def recordData():
if request.method == 'POST':
assignmentID = str(uuid.uuid1())
jsonFileName = "upload/" + assignmentID + ".json"
jsonSurvey = "upload/" + assignmentID + "_survey.json"
print("READING FORM")
with open(jsonSurvey, 'w') as f:
json.dump(request.form, f)
f.close()

Related

How to set daily update limit for django model(db)?

Hi I'm having one Django web app where My model A contains 1000 records and I want to set a daily update limit for example if users are updating that data by uploading a .xls file then it should count how many records updated and once it is more than 500 then the user should get an Error message(Is there Any easy way to implement this at file processing level).
Can Anyone help me how to implement this (Is there an SQLite parameter that I can mention in settings.py)
below is my upload code. (Here I have tried to store the count value and comparing it with half of the record but again how to reset it after 12/certain time/hrs?)
def CTA_upload(request):
try:
if request.method == 'POST':
movie_resource = CTAResource()
##we will get data in movie_resources####
dataset = Dataset()
new_movie = request.FILES['file']
if not new_movie.name.endswith('xls'):
messages.info(request, 'Sorry Wrong File Format.Please Upload valid format')
return render(request, 'apple/uploadinfosys.html')
messages.info(request, 'Uploading Data Line by Line...')
imported_data = dataset.load(new_movie.read(), format='xls')
count = 1
for data in imported_data:
value = CTA(
data[0],
data[1],
data[2],
data[3],
data[4],
data[5],
data[6],
data[7],
data[8],
)
count = count + 1
value.save()
# messages.info(request, count)
# time.sleep(1)
messages.info(request, 'File Uploaded Successfully...')
except:
messages.info(request,
'Same Email ID has been observed more than once.Except that other records has been added../nPlease Make sure Email field should be unique.')
return render(request, 'app/cta.html')

How do I efficiency create big forms - FlaskForms

So I have an update form end point, which is very large, I've tried a few things to make it dynamically created to condense my code but I cant seem to figure it out.
#customers.route("/<int:customer_id>/update_customer", methods=['GET', 'POST'])
# not programmatic - needs work - I do not like this
def update_customer(customer_id):
post = Post.query.get_or_404(customer_id)
if post.author != current_user:
abort(403)
form = CustomerForm()
if form.validate_on_submit():
post.tpi_name = form.tpi_name.data
post.tpi_ref = form.tpi_ref.data
post.company_name = form.company_name.data
post.company_type = form.company_type.data
post.company_reg = form.company_reg.data
post.sic_code = form.sic_code.data
post.vat_number = form.vat_number.data
db.session.commit()
flash("That customer's record has been updated!", 'success')
return redirect(url_for('customers.view_customer', customer_id=post.id))
elif request.method == 'GET':
form.tpi_name.data = post.tpi_name
form.tpi_ref.data = post.tpi_ref
form.company_name.data = post.company_name
form.company_type.data = post.company_type
form.company_reg.data = post.company_reg
form.sic_code.data = post.sic_code
form.vat_number.data = post.vat_number
return render_template('customers/customers.html', form=form, username=current_user.username,image_file=current_user.image_file, email=current_user.email)
So say if we had 100+ fields in the form, how would I change this code so I don't have to explicitly state each field.
Based on pjcunningham's answer, and looking into wtform docs, I found this piece of code:
def edit_profile(request):
user = User.objects.get(pk=request.session['userid'])
form = EditProfileForm(request.POST, obj=user)
if request.POST and form.validate():
form.populate_obj(user)
user.save()
return redirect('/home')
return render_to_response('edit_profile.html', form=form)
Based on that, I guess you want to write this code:
if form.validate_on_submit():
form.populate_obj(post)
db.session.commit()
flash("That customer's record has been updated!", 'success')
return redirect(url_for('customers.view_customer', customer_id=post.id))
For the second part of your code (the "GET" branch), it depends on your intentions. You can populate your form with data from object when you create it:
form = CustomerForm(obj=post)

How can I reduce the number of requests and use only one?

My program does this:
Get the XML from my website
Run all the URLs
Get data from my web page (SKU, name, title, price, etc.) with requests
Get the lowest price from another website, by comparing the price with the same SKU with requests.
I'm using with lots of requests, on each def:
def get_Price (SKU):
check ='https://www.XXX='+SKU
r = requests.get(check)
html = requests.get(r.url)
bsObj = BeautifulSoup(html.content,'html.parser')
return Price
def get_StoreName (SKU):
check ='https://XXX?keyword='+SKU
r = requests.get(check)
html = requests.get(r.url)
bsObj = BeautifulSoup(html.content,'html.parser')
return storeName
def get_h1Tag (u):
html = requests.get(u)
bsObj = BeautifulSoup(html.content,'xml')
h1 = bsObj.find('h1',attrs={'itemprop':'name'}).get_text()
return h1
How can I reduce the number of requests or connections to the URL - and use with one request or one connection throughout the whole program ?
I assume this is a script with a group of methods you call in a particular order.
If so, this is a good use case for a dict. I would write a function that memorizes calls to URLs.
You can then reuse this function across your other functions:
requests_cache = {}
def get_url (url, format_parser):
if url not in requests_cache:
r = requests.get(url)
html = requests.get(r.url)
requests_cache[url] = BeautifulSoup(html.content, format_parser)
return requests_cache[url]
def get_Price (makat):
url = 'https://www.zap.co.il/search.aspx?keyword='+makat
bsObj = get_url(url, 'html.parser')
# your code to find the price
return zapPrice
def get_zapStoreName (makat):
url = 'https://www.zap.co.il/search.aspx?keyword='+makat
bsObj = get_url(url, 'html.parser')
# your code to find the store name
return storeName
def get_h1Tag (u):
bsObj = get_url(u, 'xml')
h1 = bsObj.find('h1',attrs={'itemprop':'name'}).get_text()
return h1
If you want to avoid a global variable, you can also set requests_cache as attribute of get_url or as a default argument in the definition. The latter would also allow you to bypass the cache by passing an empty dict.
Again, the assumption here is that you are running this code as a script periodically. In that case, the requests_cache will get cleared every time you run the program.
However, if this is part of a larger program, you would want to 'expire' the cache on a regular basis, otherwise you would get the same results every time.
This is a good use case for the requests-cache library. Example:
from requests_cache import CachedSession
# Save cached responses in a SQLite file (scraper_cache.sqlite), and expire after 6 minutes
session = CachedSession('scraper_cache.sqlite', expire_after=360)
def get_Price (SKU):
check ='https://www.XXX='+SKU
r = session.get(check)
html = session.get(r.url)
bsObj = BeautifulSoup(html.content,'html.parser')
return Price
def get_StoreName (SKU):
check ='https://XXX?keyword='+SKU
r = session.get(check)
html = session.get(r.url)
bsObj = BeautifulSoup(html.content,'html.parser')
return storeName
def get_h1Tag (u):
html = session.get(u)
bsObj = BeautifulSoup(html.content,'xml')
h1 = bsObj.find('h1',attrs={'itemprop':'name'}).get_text()
return h1
Aside: with or without requests-cache, using sessions is good practice whenever you're making repeated calls to the same host, since it uses connection pooling: https://docs.python-requests.org/en/latest/user/advanced/#session-objects

results of sqlite query not displayed in flask web app

I'm attempting to learn flask, so decided to follow this tutorial:
https://www.blog.pythonlibrary.org/2017/12/14/flask-101-adding-editing-and-displaying-data/
I just updated my main function with the below:
#app.route('/results')
def search_results(search):
results = []
search_string = search.data['search']
if search.data['search'] == '':
qry = db_session.query(Album)
results = qry.all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
table = Results(results)
table.border = True
return render_template('results.html', table=table)
but when I add an album to the DB and try to query it back using search option it says no results. The DB file was created correctly and I have exactly the same code as in the tutorial up to this point.
The only change I made was adding from tables import Results. Full main.py below. Can you please give me some guidance about where to look for the culprit? Like I said, just learning, so any suggestions re resources in a friendly laid out way would be much appreciated (beginner programmer).
from app import app
from db_setup import init_db, db_session
from forms import MusicSearchForm, AlbumForm
from flask import flash, render_template, request, redirect
from models import Album, Artist
from tables import Results
init_db()
def save_changes(album, form, new=False):
"""
Save the changes to the database
"""
# Get data from form and assign it to the correct attributes
# of the SQLAlchemy table object
artist = Artist()
artist.name = form.artist.data
album.artist = artist
album.title = form.title.data
album.release_date = form.release_date.data
album.publisher = form.publisher.data
album.media_type = form.media_type.data
if new:
# Add the new album to the database
db_session.add(album)
# commit the data to the database
db_session.commit()
#app.route('/', methods=['GET', 'POST'])
def index():
search = MusicSearchForm(request.form)
if request.method == 'POST':
return search_results(search)
return render_template('index.html', form=search)
#app.route('/results')
def search_results(search):
results = []
search_string = search.data['search']
if search.data['search'] == '':
qry = db_session.query(Album)
results = qry.all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
table = Results(results)
table.border = True
return render_template('results.html', table=table)
#app.route('/new_album', methods=['GET', 'POST'])
def new_album():
"""
Add a new album
"""
form = AlbumForm(request.form)
if request.method == 'POST' and form.validate():
# save the album
album = Album()
save_changes(album, form, new=True)
flash('Album created successfully!')
return redirect('/')
return render_template('new_album.html', form=form)
if __name__ == '__main__':
app.run()
No doubt you have already peppered your source code with print() statements and found nothing illuminating. Cached rows in the DB model might be the aspect that is hard to understand, here, and logging sqlite calls would shed light on that.
Use this:
import logging
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
It's noisy, but it will show when rows hit the backend DB and when they are retrieved.
Get in the habit of repeatedly issuing debug queries like this, so you know for sure what has been persisted:
$ echo 'select * from album;' | sqlite3 music.db
For repeatable testing, it can be convenient to copy the database file to a backup location, and then cp that frozen snapshot on top of the active file before each test run. It's important that the running flask app be restarted after such copying. Setting FLASK_DEBUG=1 can help with that.
Also, jeverling suggests using SQLAlchemyDebugPanel.

get numeric IDs of list of public Facebook pages from page name

I have a list of public Facebook pages (e.g. media outlets) of which I need the page ids (numeric) but only have the name. Since I have a lot of pages, I can't find all of them manually, so how would I process that manually?
As an example, my pages are:
pages = ['financialtimes', 'dailytelegraph', 'theguardian']
And I think I need to do something like this:
url = "https://graph.facebook.com/v2.4/" + 'financialtimes' + '/?access_token=' + 'my_app_id' + 'my_app_secret'
print(url)
except I need that for a lot of pages. How do I do this?
You can just set up a function and pass the items of your list to that function, e.g.
# FB access
app_id = "my_app_id"
app_secret = "my_app_secret"
access_token = app_id + "|" + app_secret
#define function
def FacebookIDs(page_name, access_token=access_token):
""" get page's numeric information """
# construct the URL string
base = "https://graph.facebook.com/v2.4"
node = "/" + str(page_name)
parameters = "/?access_token=%s" % access_token
url = base + node + parameters
# retrieve data
with urllib.request.urlopen(url) as url:
data = json.loads(url.read().decode())
print(data)
#return data # if you want the data not to be printed but to be the input for another function etc.
# define list
pages = ['financialtimes', 'dailytelegraph', 'theguardian']
# iterate over list
for page in pages: FacebookIDs(page, access_token)

Resources