Hi I'm having one Django web app where My model A contains 1000 records and I want to set a daily update limit for example if users are updating that data by uploading a .xls file then it should count how many records updated and once it is more than 500 then the user should get an Error message(Is there Any easy way to implement this at file processing level).
Can Anyone help me how to implement this (Is there an SQLite parameter that I can mention in settings.py)
below is my upload code. (Here I have tried to store the count value and comparing it with half of the record but again how to reset it after 12/certain time/hrs?)
def CTA_upload(request):
try:
if request.method == 'POST':
movie_resource = CTAResource()
##we will get data in movie_resources####
dataset = Dataset()
new_movie = request.FILES['file']
if not new_movie.name.endswith('xls'):
messages.info(request, 'Sorry Wrong File Format.Please Upload valid format')
return render(request, 'apple/uploadinfosys.html')
messages.info(request, 'Uploading Data Line by Line...')
imported_data = dataset.load(new_movie.read(), format='xls')
count = 1
for data in imported_data:
value = CTA(
data[0],
data[1],
data[2],
data[3],
data[4],
data[5],
data[6],
data[7],
data[8],
)
count = count + 1
value.save()
# messages.info(request, count)
# time.sleep(1)
messages.info(request, 'File Uploaded Successfully...')
except:
messages.info(request,
'Same Email ID has been observed more than once.Except that other records has been added../nPlease Make sure Email field should be unique.')
return render(request, 'app/cta.html')
Related
I'm working on a Notes app project and I want to implement a new feature in my API that allows users to move a post to trash (if the user doesn't restore the post after a certain time it will be permanently deleted.),
Given two tables: Posts and Deleted, I want my endpoint "./posts/totrash/{id}" to insert a post in the deleted table and then delete this post from the table Posts:
Here is my code
(edit: the table Posts is "page")
#router.delete("/posts/totrash/{id}", status_code=status.HTTP_204_NO_CONTENT)
def del_post(id: int, current_user=Depends(Oathou2.get_current_user)):
user_id=current_user.id
cur.execute("""select user_id from page where id = %s""", (str(id),))
post_owner=cur.fetchone()
if post_owner is None:
raise HTTPException(status_code=status.HTTP_204_NO_CONTENT, detail="the post with the id : {id} doesn't exist")
if int(user_id) != int(post_owner["user_id"]):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="you are not the owner of"
"this post")
else:
cur.execute("""select * from page where id=(%s)""", (str(id),))
note = cur.fetchone()
cur.execute("""insert into deleted(name, content, id, published , user_id , created_at)
values(%s, %s, %s, %s, %s, %s) returning * """,
(note["name"], note["content"], note["id"], note["published"], note["user_id"], note["created_at"])
)
# If a comment this following line, the insert will work
cur.execute("""delete from page where id = %s""", (str(id),))
conn.commit()
return Response(status_code=status.HTTP_204_NO_CONTENT)
Now the selected post is deleted from the table Posts but when I check the table deleted I don't find the post which means It hasn't been inserted.
EDIT
#this is how I create the conn (connection)
class Database:
def connection(self):
while True:
try:
conn = psycopg2.connect(host=settings.host, database=settings.db_name, user=settings.db_user_name,
password=settings.db_password, cursor_factory=RealDictCursor )
print("connected")
break
except Exception as error:
print("connection faild")
print(error)
time.sleep(2)
return conn
database=Database()
In routes files:
from .dtbase import database
conn= database.connection()
cur = conn.cursor()
The approach that I used wasn't well structured.
what I did in the end is using only one table Posts and a column named deleted so if thr note is not deleted yet the value of deleted is false. When I delete the note I switch the value to true.
I have tried to fix this issue for a while.
I have a model where there are logs entered by users and saved in Log Model. Also there is ActiveSession Model which has a ManytoMany relationship with Log Model.
In My views I am getting the POST and saving them in the Log Model. The next step is my issue where I am trying to add unqiue Logs inside the ActiveSession Model. By unique I mean by searching a field called 'log_order' in the Log Model.
The desired outcome is to:
1: Get the POST form and save the Log details in the Log Model
2: Search if the new saved Log has been previously added the many to many relation log under Active Session Model
3: If there is no Log in the ActiveSession Model to add the new Log in the Post to the activesession
4: If there are logs previously saved in ActiveSession to search for the log_order related to these logs
5: If there is no same log_order to add the new Log in the Post to the activesession
6: If the log_order is existing inside the activesession logs to delete this specific log and to add the new Log in the Post to the activesession
Models.py:
class Log(models.Model):
...................................
log_order = models.IntegerField(validators=[MinValueValidator(1)],blank=True, null=True)
date = models.DateTimeField(...........)
class LogForm(forms.Form):
.............................
log_order = forms.IntegerField()
class ActiveSession(models.Model):
log = models.ManyToManyField(Log)
..................................
Views.py:
def addlog(request, id):
active_session = ActiveSession.objects.get(id=ActiveSession.objects.last().id)
if request.method == 'POST':
form = LogForm(request.POST)
if form.is_valid():
data = Log()
data.log_order = request.POST.get('log_order')
data.log_repetitions = form.cleaned_data['log_repetitions']
data.log_weight = form.cleaned_data['log_weight']
data.save()
if active_session.log.values():
for i in active_session.log.values():
existing = i['log_order']
new = data.log_order
if int(existing) == int(new):
active_session.delete(active_session.log)
active_session.log.add(data)
active_session.save()
else:
active_session.log.add(data)
else:
active_session.log.add(data)
I have tried to save the data and update() but nothing is changing the dictionary the same exisiting outcome is printed and the new data is not replacing the existing ones. My objective is replace and avoid duplicating.
Update: I am also thought of if the keys is i['log_order'] is equal to data.log_order to delete the whole dictionary and add the new data whichever is easier
Trial:
With the below trial I got 'ManyRelatedManager' object is not subscriptable. I am not sure if I am going the wrong direction.
if active_session.log.values():
for i in range(len(active_session.log.values())):
new = data.log_order
if active_session.log[i]['log_order'] == new:
active_session.save()
else:
active_session.log.add(data)
else:
active_session.log.add(data)
Here is the current query:
<QuerySet [{'log_order': 1}, {'log_order': 1}]>
here is the requested query:
<QuerySet [{'log_order': 1}]>
I have tried and tested your code and found some issues with it.
Below, there is the code snippet I tested and it works.
I am not completely sure what is the purpose for the code, but I think you might find the method update_or_create useful (also works on related managers such as your log). Here is a link to Django docs.
def addlog(request, id):
if request.method == 'POST':
# Moved this part to the POST section of the view. It is sufficient to simplify
# the query in this way. Also, you probably want to check if there are sessions
# in the DB. Otherwise, the code would break. This returns a 404 in case there
# are absolutely no sessions.
active_session = ActiveSession.objects.last()
if active_session is None:
return Http404()
form = LogForm(request.POST)
if form.is_valid():
data = Log()
# Use form.cleaned_data instead of log_order. Form validates the data, POST dict doesn't.
data.log_order = form.cleaned_data['log_order']
# Ignored because it's missing from the model code
# data.log_repetitions = form.cleaned_data['log_repetitions']
# data.log_weight = form.cleaned_data['log_weight']
data.save()
# Use Django ORM querysets instead of values()
# See https://www.django-antipatterns.com/antipattern/over-use-of-values.html
active_session_logs = active_session.log.filter(log_order=data.log_order)
if active_session_logs.exists():
# Suggestion: try to use update() instead of a combination of delete & add.
# It reduces two SQL queries to one.
active_session_logs.delete()
active_session.log.add(data)
# After using RelatedManager.add(), no need to call save(), the changes
# are persisted in the DB simply by using add().
# Always redirect after successfully processing the form.
return redirect(reverse('test_73764999:add-log', args=(data.id,)))
else:
# If there are errors, re-render the same form with errors
return render(request, 'test_73764999/form.html', {
'form': form, 'id': id,
})
else:
# If this is a GET request, render an empty form
form = LogForm()
return render(request, 'test_73764999/form.html', {
'form': form, 'id': id,
})
I am using FLASK to create a series of websites for doing a survey, I want to run the server once and invite multiple participants to help me fill the form on the webpage.
1.
I am aiming for assigning a unique ID (currently using python UUID) to different visitors and save their input info into different JSON files. (eg,. A's id is 00001, after he submiting the form, the info will be saved in the file named 00001.json, B's info will be saved in 00002.json). For now, I only create a UUID when the server start running, every visitor's information will be rewritten in the same file, (what I want is n visitors will produce n files).
2.
I want to show different survey questions to different visitors.
I am quite confused about if it is related to Multi-thread? Session? Cookie? Flask-login.
Hope someone can guide me on this. Thanks a lot!
assignmentID = str(uuid.uuid1())
jsonFileName = "upload/" + assignmentID + ".json"
jsonSurvey = "upload/" + assignmentID + "_survey.json"
#it ideally should randomly select different info/question no. to different user
Idx = random.sample(range(0,10), 5)
#app.route("/")
def index():
return render_template("surveyPage.html", data=Idx)
# write input information from webpage to JSON files, each visitor ideally should have their own JSON file.
#app.route("/record")
def recordData():
if request.method == 'POST':
print("READING FORM")
with open(jsonSurvey, 'w') as f:
json.dump(request.form, f)
f.close()
if __name__ == "__main__":
app.config['JSON_SORT_KEYS'] = False
app.run(debug = True)
# app.config["TEMPLATES_AUTO_RELOAD"] = True
You are defining assignmentID, jsonFileName, jsonSurvey and Idx outside of the request handlers which means they will be set once: when the server starts. jsonSurvey will therefore have the same value for each request which means that every new survey you store will overwrite the previous one.
The solution is to move the definition into the request handlers so that you genereate new values on each request.
#app.route("/")
def index():
Idx = random.sample(range(0,10), 5)
return render_template("surveyPage.html", data=Idx)
#app.route("/record")
def recordData():
if request.method == 'POST':
assignmentID = str(uuid.uuid1())
jsonFileName = "upload/" + assignmentID + ".json"
jsonSurvey = "upload/" + assignmentID + "_survey.json"
print("READING FORM")
with open(jsonSurvey, 'w') as f:
json.dump(request.form, f)
f.close()
I'm attempting to learn flask, so decided to follow this tutorial:
https://www.blog.pythonlibrary.org/2017/12/14/flask-101-adding-editing-and-displaying-data/
I just updated my main function with the below:
#app.route('/results')
def search_results(search):
results = []
search_string = search.data['search']
if search.data['search'] == '':
qry = db_session.query(Album)
results = qry.all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
table = Results(results)
table.border = True
return render_template('results.html', table=table)
but when I add an album to the DB and try to query it back using search option it says no results. The DB file was created correctly and I have exactly the same code as in the tutorial up to this point.
The only change I made was adding from tables import Results. Full main.py below. Can you please give me some guidance about where to look for the culprit? Like I said, just learning, so any suggestions re resources in a friendly laid out way would be much appreciated (beginner programmer).
from app import app
from db_setup import init_db, db_session
from forms import MusicSearchForm, AlbumForm
from flask import flash, render_template, request, redirect
from models import Album, Artist
from tables import Results
init_db()
def save_changes(album, form, new=False):
"""
Save the changes to the database
"""
# Get data from form and assign it to the correct attributes
# of the SQLAlchemy table object
artist = Artist()
artist.name = form.artist.data
album.artist = artist
album.title = form.title.data
album.release_date = form.release_date.data
album.publisher = form.publisher.data
album.media_type = form.media_type.data
if new:
# Add the new album to the database
db_session.add(album)
# commit the data to the database
db_session.commit()
#app.route('/', methods=['GET', 'POST'])
def index():
search = MusicSearchForm(request.form)
if request.method == 'POST':
return search_results(search)
return render_template('index.html', form=search)
#app.route('/results')
def search_results(search):
results = []
search_string = search.data['search']
if search.data['search'] == '':
qry = db_session.query(Album)
results = qry.all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
table = Results(results)
table.border = True
return render_template('results.html', table=table)
#app.route('/new_album', methods=['GET', 'POST'])
def new_album():
"""
Add a new album
"""
form = AlbumForm(request.form)
if request.method == 'POST' and form.validate():
# save the album
album = Album()
save_changes(album, form, new=True)
flash('Album created successfully!')
return redirect('/')
return render_template('new_album.html', form=form)
if __name__ == '__main__':
app.run()
No doubt you have already peppered your source code with print() statements and found nothing illuminating. Cached rows in the DB model might be the aspect that is hard to understand, here, and logging sqlite calls would shed light on that.
Use this:
import logging
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
It's noisy, but it will show when rows hit the backend DB and when they are retrieved.
Get in the habit of repeatedly issuing debug queries like this, so you know for sure what has been persisted:
$ echo 'select * from album;' | sqlite3 music.db
For repeatable testing, it can be convenient to copy the database file to a backup location, and then cp that frozen snapshot on top of the active file before each test run. It's important that the running flask app be restarted after such copying. Setting FLASK_DEBUG=1 can help with that.
Also, jeverling suggests using SQLAlchemyDebugPanel.
I have been using psycopg2 to manage items in my PostgreSQL database. Recently someone suggested that I could improve my database transactions by using asyncio and asyncpg in my code. I have looked around Stack Overflow and read though the documentation for examples. I have been able to create tables and insert records, but I haven't been able to get the execution feedback that I desire.
For example in my psycopg2 code, I can verify that a table exists or doesn't exist prior to inserting records.
def table_exists(self, verify_table_existence, name):
'''Verifies the existence of a table within the PostgreSQL database'''
try:
self.cursor.execute(verify_table_existence, name)
answer = self.cursor.fetchone()[0]
if answer == True:
print('The table - {} - exists'.format(name))
return True
else:
print ('The table - {} - does NOT exist'.format(name))
return False
except Exception as error:
logger.info('An error has occurred while trying to verify the existence of the table {}'.format(name))
logger.info('Error message: {}').format(error)
sys.exit(1)
I haven't been able to get the same feedback using asyncpg. How do I accomplish this?
import asyncpg
import asyncio
async def main():
conn = await asyncpg.connect('postgresql://postgres:mypassword#localhost:5432/mydatabase')
answer = await conn.fetch('''
SELECT EXISTS (
SELECT 1
FROM pg_tables
WHERE schemaname = 'public'
AND tablename = 'test01'
); ''')
await conn.close()
#####################
# the fetch returns
# [<Record exists=True>]
# but prints 'The table does NOT exist'
#####################
if answer == True:
print('The table exists')
else:
print('The table does NOT exist')
asyncio.get_event_loop().run_until_complete(main())
You used fetchone()[0] with psycopg2, but just fetch(...) with asyncpg. The former will retrieve the first column of the first row, while the latter will retrieve a whole list of rows. Being a list, it doesn't compare as equal to True.
To fetch a single value from a single row, use something like answer = await conn.fetchval(...).