After hours of debugging, and because my organization does not have a lot of Python expertise, I am turning to this community for help.
I am trying to follow this tutorial with the goal of committing some data to the database. Although no errors get reported, I am also not saving any rows. What am I doing wrong?
When trying to commit using the db2Session, I get:
Transaction must be committed using the transaction manager.
But nowhere in the tutorial, do I see the transaction manager being used. I thought that this manager is bound using zope.sqlalchemy? Yet, nothing is happening otherwise. Help again would be really appreciated!
I have the following setup in my main function in a Pyramid App:
from sqlalchemy import engine_from_config
from .models import db1Session, db2Session
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
db1_engine = engine_from_config(settings, 'db1.')
db2_engine = engine_from_config(settings, 'db2.')
db1Session.configure(bind=db1_engine)
db2Session.configure(bind=db2_engine)
In .models/__init__py, I have:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import (scoped_session, sessionmaker)
from zope.sqlalchemy import ZopeTransactionExtension
db1Session = scoped_session(sessionmaker(
extension=ZopeTransactionExtension()))
db2Session =
scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()
In ./model/db2.py I have:
class PlateWellResult(Base):
__tablename__ = 'SomeTable'
__table_args__ = {"schema": 'some_schema'}
id = Column("ID", Integer, primary_key=True)
plate_id = Column("PlateID", Integer)
hit_group_id = Column("HitID", Integer, ForeignKey(
'some_schema.HitGroupID.ID'))
well_loc = Column("WellLocation", String)
The relevant bits of my saving function look like this. ./lib/db2_api.py:
def save_selected_rows(input_data, selected_rows, hit_group_id):
""" Wrapper method for saving selected rows """
# Assume I have all the right data below.
new_hit_row = PlateWellResult(
plate_id=master_plate_id,
hit_group_id=hit_group_id,
well_loc=selected_df_row.masterWellLocation)
db1Session.add(new_hit_row)
# When I try the row below:
# db2Session.commit()
# I get: Transaction must be committed using the transaction manager
# If I cancel the line above, nothing gets committed.
return 'Save successful.'
That function is called from my viewer:
#view_config(route_name='some_routename', renderer='json',
permission='create_hit_group')
def save_to_hitgroup(self):
""" Backend to AJAX call to save selected rows to a hit_group """
try:
# Assume that all values were checked and all the right
# parameters are passed
status = save_selected_rows(some_result,
selected_rows_list,
hitgroup_id)
json_resp = json.dumps({'errors': [],
'status': status})
return json_resp
except Exception as e:
json_resp = json.dumps({'errors': ['Error during saving. {'
'0}'.format(e)],
'status': []})
return json_resp
The comments above are good. I just wanted to summarize here.
The transaction manager is begun/committed/aborted by pyramid_tm. If you aren't using that then it's likely the issue.
You are also squashing possible database exceptions which need to be conveyed to the transaction manager. You can do this via transaction.abort() in the exception handler.
Related
I'm debugging existing code. I'm trying to find out the intention of the obviously wrong access to .dicts of a peewee Model in the warning statement in MyDbBackend.store and how I could correct that.
I guess that the warning message should add more detailed output to the model which could not be saved. However, the .dicts attribute exists in orm.BaseQuery class, only.
The output message is currently not very helpful. I want to provide an improved warning message given that the i.save fails. With "improved" i mean to provide some meta informations about the record which failed to be saved.
So, how can i obtain the BaseQuery from the model and what would .dicts output, then? Would that information be useful in the context of the warning message?
import peewee as orm
database = orm.Proxy()
class ModelBase(orm.Model):
class Meta:
database = database
class MyModel(ModelBase):
dtfield = orm.DateTimeField(null=True)
intfield = orm.IntegerField(null=True)
floatfield = orm.FloatField(null=True)
class MyDbBackend:
def __init__(self, database):
self.db = database
self.records = [] # Holds objects derived from ModelBase
[...]
def store(self):
with self.db.atomic():
for i in self.records:
try:
i.save()
except Exception as e:
logger.warning("could not save record: {}".format(i.dicts()))
raise e
self.clear()
->
logger.warning("could not save record: {}".format(i.dicts()))
AttributeError: 'MyModel' object has no attribute 'dicts'
I guess that the original code was meant to make use of playhouse.shortcuts.model_to_dict.
This is the only idea I have why the original code uses i.dict().
Perhaps some misunderstanding.
import peewee as orm
from playhouse.shortcuts import model_to_dict
[...]
logger.warning(f"Model dict: {model_to_dict(i, recurse = True, max_depth = 2)}")
[...]
I'd like to test a POST route that processes a non-trivial form (by working with flask.request.form). I didn't really find a good tutorial for this somehow as most pass json data rather than form (or is it the same?).
I tried to write the code in the following way:
import pytest
import app #app.app is the Flask app
#pytest.fixture
def client():
app.app.config['TESTING'] = True
with app.app.test_client() as client:
with app.app.app_context():
yield client
def test_route_webapp_post(client):
form = app.forms.ImputeForm.make_form(data_dict=app.data_dictionary.data_dict,
numeric_fields=app.binaries_dict['numeric_mappers'].keys(),
recordname2description=app.binaries_dict['recordname2description'])
rv = client.post('/web_app',form=form)
assert rv.status_code==200
The form is generated dynamically and I don't always know ahead of time what are the fields:
from flask_wtf import FlaskForm
from wtforms import SelectField, DecimalField, BooleanField
class ImputeForm(FlaskForm):
#classmethod
def make_form(cls, data_dict, numeric_fields, recordname2description, request_form=None):
for key in numeric_fields:
setattr(cls, key, DecimalField(id=key, label=recordname2description[key].split('(')[0]))
setattr(cls, 'mask_' + key, BooleanField(label='mask_' + key))
for key in data_dict:
setattr(cls, key, SelectField(id=key, label=recordname2description[key],
choices=[(-1, 'None selected')]+list(data_dict[key].items())))
setattr(cls, 'mask_' + key, BooleanField(label='mask_' + key))
instance = cls(request_form)
return instance
But this doesn't really work as I can't make a form inside the test case and get
E RuntimeError: Working outside of request context.
E
E This typically means that you attempted to use functionality that needed
E an active HTTP request. Consult the documentation on testing for
E information about how to avoid this problem.
So what is the proper approach to testing my form (in particular I am ok with sending an empty one)?
The correct way is to create a python dictionary and pass it as "data", not to try to create a form.
In particular case this involved making a new function:
def make_from_data( data_dict, numeric_fields):
data = dict()
for key in numeric_fields:
data[key]='234'
data['mask_' + key]='y'
for key in data_dict:
data[key]=-1
data['mask_' + key]='y'
return data
and passing it as follows:
def test_route_webapp_post(client):
data = make_from_data(data_dict=app.data_dictionary.data_dict,
numeric_fields=app.binaries_dict['numeric_mappers'].keys())
rv = client.post('/web_app',data=data)
assert rv.status_code==200
I'm attempting to learn flask, so decided to follow this tutorial:
https://www.blog.pythonlibrary.org/2017/12/14/flask-101-adding-editing-and-displaying-data/
I just updated my main function with the below:
#app.route('/results')
def search_results(search):
results = []
search_string = search.data['search']
if search.data['search'] == '':
qry = db_session.query(Album)
results = qry.all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
table = Results(results)
table.border = True
return render_template('results.html', table=table)
but when I add an album to the DB and try to query it back using search option it says no results. The DB file was created correctly and I have exactly the same code as in the tutorial up to this point.
The only change I made was adding from tables import Results. Full main.py below. Can you please give me some guidance about where to look for the culprit? Like I said, just learning, so any suggestions re resources in a friendly laid out way would be much appreciated (beginner programmer).
from app import app
from db_setup import init_db, db_session
from forms import MusicSearchForm, AlbumForm
from flask import flash, render_template, request, redirect
from models import Album, Artist
from tables import Results
init_db()
def save_changes(album, form, new=False):
"""
Save the changes to the database
"""
# Get data from form and assign it to the correct attributes
# of the SQLAlchemy table object
artist = Artist()
artist.name = form.artist.data
album.artist = artist
album.title = form.title.data
album.release_date = form.release_date.data
album.publisher = form.publisher.data
album.media_type = form.media_type.data
if new:
# Add the new album to the database
db_session.add(album)
# commit the data to the database
db_session.commit()
#app.route('/', methods=['GET', 'POST'])
def index():
search = MusicSearchForm(request.form)
if request.method == 'POST':
return search_results(search)
return render_template('index.html', form=search)
#app.route('/results')
def search_results(search):
results = []
search_string = search.data['search']
if search.data['search'] == '':
qry = db_session.query(Album)
results = qry.all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
table = Results(results)
table.border = True
return render_template('results.html', table=table)
#app.route('/new_album', methods=['GET', 'POST'])
def new_album():
"""
Add a new album
"""
form = AlbumForm(request.form)
if request.method == 'POST' and form.validate():
# save the album
album = Album()
save_changes(album, form, new=True)
flash('Album created successfully!')
return redirect('/')
return render_template('new_album.html', form=form)
if __name__ == '__main__':
app.run()
No doubt you have already peppered your source code with print() statements and found nothing illuminating. Cached rows in the DB model might be the aspect that is hard to understand, here, and logging sqlite calls would shed light on that.
Use this:
import logging
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
It's noisy, but it will show when rows hit the backend DB and when they are retrieved.
Get in the habit of repeatedly issuing debug queries like this, so you know for sure what has been persisted:
$ echo 'select * from album;' | sqlite3 music.db
For repeatable testing, it can be convenient to copy the database file to a backup location, and then cp that frozen snapshot on top of the active file before each test run. It's important that the running flask app be restarted after such copying. Setting FLASK_DEBUG=1 can help with that.
Also, jeverling suggests using SQLAlchemyDebugPanel.
I'm trying to learn how to create python-based back-ends from some existing data that i have collected. I've come to realize that i definitely want to use sqlalchemy and that flask seems like a good library to go with it. My problem is that even after many hours of reading the sqlalchemy docs and browsing various answers on stackexchange i still don't understand how i can reshape data from an existing table into an object with a completely different structure.
The transformation i want to do is very concrete. I want to go from this structure in my MariaDB table:
Columns: company_name, date, indicators(1...23)
To this json output generated from a serialized class object:
{
"company_name[1]":
{
"indicator_name[1]":
{
"date[1]": "indicator_name[1].value[1]",
"date[2]": "indicator_name[1].value[2]",
"date[3]": "indicator_name[1].value[3]",
"date[4]": "indicator_name[1].value[4]",
"date[5]": "indicator_name[1].value[5]"
},
"indicator_name[2]":
{
"date[1]": "indicator_name[2].value[1]",
"date[2]": "indicator_name[2].value[2]",
"date[3]": "indicator_name[2].value[3]",
"date[4]": "indicator_name[2].value[4]",
"date[5]": "indicator_name[2].value[5]"
},
I found a great tutorial with which i can output the entire table record by record but the structure is not what i want, and i don't think creating the desired structure on the front-end makes sense in this case.
Here is the code that outputs the entire table to json record by record:
from flask import Flask, jsonify
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import PrimaryKeyConstraint
from sqlalchemy import orm
from sqlalchemy import select, func
from sqlalchemy import Column, Integer, String, ForeignKey
from flask_marshmallow import Marshmallow
import decimal
import flask.json
class MyJSONEncoder(flask.json.JSONEncoder): # Enables decimal queries for the API
def default(self, obj):
if isinstance(obj, decimal.Decimal):
# Convert decimal instances to strings.
return str(obj)
return super(MyJSONEncoder, self).default(obj)
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql+pymysql://USER:PASS#localhost:3306/kl_balance_sheets'
app.json_encoder = MyJSONEncoder
db = SQLAlchemy(app)
ma = Marshmallow(app)
# Bind declarative base to engine
db.Model.metadata.reflect(db.engine)
class CompanyData(db.Model):
__table__ = db.Model.metadata.tables['kl_balance_sheets']
class CompanyDataSchema(ma.ModelSchema):
class Meta:
model = CompanyData
#app.route('/')
def index():
company_data = CompanyData.query.all()
company_data_schema = CompanyDataSchema(many=True)
output = company_data_schema.dump(company_data).data
return jsonify({'company_data' : output})
if __name__ == '__main__':
app.run(debug=True)
My main question i guess is: How do i edit this code to produce the desired json?
What i think i should do is to create a custom constructor and then feed that into the index function but i can't figure out how to concretely do that. The two options i've come across are:
#orm.reconstructor
def init_on_load(self):
#do custom stuff
or:
class Foo(db.Model):
# ...
def __init__(**kwargs):
super(Foo, self).__init__(**kwargs)
# do custom stuff
To me this seems like a basic operation any flask-marshmallow user would be doing regularly. Could someone please explain how sql data is normally inserted into an object with a new structure and then serialized? In my case, do i need to change things mainly on the metadata, object or marshmallow level? I'm surprised i can't find some good examples of this.
I am trying to build a database driver for Peewee and i'm having trouble getting the save() method to fill in the primary key/id for objects. Here's some sample code:
from datetime import date
from peewee import BooleanField
from peewee import CharField
from peewee import DateField
from peewee import ForeignKeyField
from peewee import IntegerField
from peewee import Model
from SQLRelay import PySQLRDB
from sqlrelay_ext import SQLRelayDatabase
DB = SQLRelayDatabase('test2', host='<host>', user='<un>', password='<pwd>')
class BaseModel(Model):
class Meta:
database = DB
class Person(BaseModel):
name = CharField()
birthday = DateField()
is_relative = BooleanField()
class Pet(BaseModel):
owner = ForeignKeyField(Person, backref='pets')
name = CharField()
animal_type = CharField()
DB.connect()
Person.create_table(safe=False)
Pet.create_table(safe=False)
uncle_bob = Person(name='Bob', birthday=date(1960, 1, 15), is_relative=True)
uncle_bob.save() # bob is now stored in the database
print('Uncle Bob id: {}'.format(uncle_bob.id))
print('Uncle Bob _pk: {}'.format(uncle_bob._pk))
Both uncle_bob.id and uncle_bob._pk are None after .save(). From digging into the peewee.py code, it seems that the _WriteQuery.execute() method is supposed to set the _pk attribute, but that isn't happening. My best guess is that the cursor implementation isn't acting properly. Does anyone have more insight than this that can maybe help me track down this problem?
Thanks!
Edit to answer:
For SQL Server, the following code allows you to return the last inserted id:
def last_insert_id(self, cursor, query_type=None):
try:
cursor.execute('SELECT SCOPE_IDENTITY()')
result = cursor.fetchone()
return result[0]
except (IndexError, KeyError, TypeError):
pass
In your SQLRelayDatabase implementation, you will probably need to correctly implement the last_insert_id() method. For python db-api 2.0 drivers, this typically looks like cursor.lastrowid.
The default implementation is:
def last_insert_id(self, cursor, query_type=None):
return cursor.lastrowid
Where cursor is the cursor object used to execute the insert query.
Databases like Postgresql do not implement this -- instead you execute an INSERT...RETURNING query, so the Postgres implementation is a bit different. The postgres implementation ensures that your insert query includes a RETURNING clause, and then grabs the id returned.
Depending on your DB and the underlying DB-driver, you'll need to pull that last insert id out somehow. Peewee should handle the rest assuming last_insert_id() is implemented.