I trying to create a system with models that can be used within the flask's app context and by external jobs.
I have created a database.py with the following:
from sqlalchemy import MetaData, Column, Integer, String, Float, DateTime
from sqlalchemy.ext.declarative import declarative_base
metadata = MetaData()
Base = declarative_base(metadata=metadata)
class Stock(Base):
__tablename__ = 'Stock'
id = Column(Integer, primary_key=True)
ticker = Column(String(10), nullable=False)
time_stamp = Column(DateTime, nullable=False)
open = Column(Float, nullable=False)
high = Column(Float, nullable=False)
low = Column(Float, nullable=False)
close = Column(Float, nullable=False)
volume = Column(Integer, nullable=False)
price = Column(Float, nullable=True)
I have also then created a flask_db.py file:
from flask_sqlalchemy import SQLAlchemy
from database import metadata
db = SQLAlchemy(metadata=metadata)
This is then used in my app.py
from flask import Flask, render_template
from flask_db import db
import os
# create the flask app
app = Flask(__name__, template_folder="dist/", static_folder="dist/", static_url_path="")
# config
app.config.from_object(os.environ['APP_SETTINGS'])
# initialise db
db.init_app(app)
#app.route("/")
def index():
"""
renders basic index page
:return: template
"""
return render_template("index.html")
# app entry point
if __name__ == "__main__":
app.run()
When I run a file called create_db.py:
from flask_db import db
db.create_all()
I get the following error:
RuntimeError: No application found. Either work inside a view function or push an application context. See http://flask-sqlalchemy.pocoo.org/contexts/.
I am not really sure why no application is found.
The db extension will only get initialized when init_app is called, and the app context will only get created when the app is run. But because you're importing from your flask_db module, neither of those things happens - the code in the app module never gets run by the Python interpreter.
What you probably need to do is something like this:
from app import app, db
app_ctx = app.app_context()
ctx.push()
db.create_all()
ctx.pop()
I would think that should work. But I strongly suggest you look into using the application factory pattern to create your app, and using click commands (with the with_appcontext decorator) to add to Flask's cli interface, instead of using one-off scripts. Also, it's probably worth using Flask-Migrate for proper database migrations.
EDIT: in your app module, you probably also need to tell Flask-SQLAlchemy about your custom base model:
db = SQLAlchemy(metadata=metadata, model_class=Base)
Related
I just started learning Flask. I can't figure out how to initialize a mysql database. Do I need to use some version of Alchemy, can I not use mysql modules only instead? I'm getting confused as I google looking any docs on using flask and mysql and I get many hits that include slqalchemy, flask-mysql, flask-mysqldb, flask-mysql-connector. Each example is different and I am going cross-eyed.
With so many examples I suspect my syntax is mixed up. I finally was able to display schemas with following command so I assume am close:
# msql_connector.py
from application import mysql
print(mysql)
conn = mysql.connect()
cursor = conn.cursor()
cursor.execute('''SHOW SCHEMAS;''')
rv = cursor.fetchall()
for db in rv:
print(db)
However I now want to initialize the db or just be able to write values to it as per below. I am not sure if I actually do need to initialize first or whether I can define the model and add data during runtime.
My error:
AttributeError: 'MySQL' object has no attribute 'Model'
My main file:
# __init__.py
from flask import Flask
from config import Config
from flaskext.mysql import MySQL
app = Flask(__name__)
app.config.from_object(Config)
mysql = MySQL()
mysql.init_app(app)
from application import routes
Next my DB definitions:
# models.py
import flask
from application import mysql
class User(mysql.Model):
user_id = mysql.IntField(unique=True )
first_name = mysql.StringField( max_length = 50 )
last_name = mysql.StringField( max_length = 50 )
email = mysql.StringField( max_length = 50 )
password = mysql.StringField( max_length = 50 )
def __init__(self, user_id, first_name, last_name, email, password):
self.user_id = user_id
self.first_name = first_name
self.last_name = last_name
self.email = email
self.password = password
My configuration:
# config.py
import os
from flask_mysql_connector import MySQL
class Config(object):
SECRET_KEY = os.environ.get('SECRET_KEY') or 'djfdsjdsj4skldjess85'
MYSQL_DATABASE_HOST = 'localhost'
MYSQL_DATABASE_USER = 'flasker'
MYSQL_DATABASE_PASSWORD = '**********'
MYSQL_DATABASE_DB = 'UTA_Enrollment'
and last but not least:
import json
from application import app
from application import mysql
from flask import render_template, request, Response
from application.models import User, Course, Enrollment
#app.route("/user")
def user():
conn = mysql.connect()
cursor = conn.cursor()
cursor.execute('''SHOW SCHEMAS;''')
rv = cursor.fetchall()
User(user_id=1, first_name="John", last_name="Doh", email="jd#domain.com", password="abc123")
User(user_id=2, first_name="Jane", last_name="Doh", email="jad#domain.com", password="abc123")
users = User.objects.all()
return render_template("user.html", users=users)
So when I go to http://url/user I would like to be able to write the above data into the database.
My specs:
Python 3.9.6
click==8.1.3
Flask==2.2.2
Flask-MySQL==1.5.2
flask-mysql-connector==1.1.0
Flask-WTF==1.0.1
importlib-metadata==6.0.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
mysql-connector-python==8.0.31
numpy==1.24.1
pandas==1.5.2
protobuf==3.20.1
PyMySQL==1.0.2
python-dateutil==2.8.2
python-dotenv==0.21.0
pytz==2022.7
six==1.16.0
Werkzeug==2.2.2
WTForms==3.0.1
zipp==3.11.0
So to recap:
Do I have to use some form of Alchemy to run this type of syntax: title = mysql.StringField( max_length = 100 )
am I only limited to running executes as in cursor.execute('''SHOW SCHEMAS;''')
do I need to initialize the database (including creating the the tables and fields first?
So I couldnt find the elegant solution for MySQL modules. I went round and round to python-mysql, python-mysql-connector, and flask-msyql. It did turn out that using flask-sqlalchemy was fairly easy to get it to do what I needed.
pip install Flask-SQLAlchemy
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
class Config(object):
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://..."
SQLALCHEMY_TRACK_MODIFICATIONS = False
class Course(db.Model):
course_id = db.Column( db.String(10), primary_key=True )
title = db.Column( db.String(100), nullable=False )
description = db.Column( db.String(255), nullable=False )
credits = db.Column( db.String(255), nullable=False )
term = db.Column( db.String(25), nullable=False )
I am working on a proyect that works with these libraries (among others)
postgresql 10.5
pandas 1.1.1
psycopg2 2.7.5
pytest 5.0.1
python 3.7.1
I am trying to run pytest on a library we have created that uses sqlalchemy to access a postgresql database.
I want to re-create a database to test the methods that we have being using for few months.
I have tried different things but without success.
I know that postgresql cannot create an in memory database, which is why I am trying to use sqlite3 for that purpose. .
The method I am trying to run test over is:
DatabaseHelper.py
class DatabaseHelper(object):
"""
Helps accessing database.
"""
def __init__(self):
pass
# ...
def create_engine(self, host_dns, dbname, port, user, password):
"""
:param host_dns: dns route to the database
:param dbname: name of the database to access to
:param port: number or port of the database
:param user: name of the user to access de database
:param password: password to connect to the database
"""
self.host = host_dns
self.dbname = dbname
self.port = port
self.user = user
self.password = password
self.connection_str = f'postgresql://{self.user}:{self.password}#{self.host}:{self.port}/{self.dbname}'
self.engine = create_engine(self.connection_str)
# session_factory = sessionmaker(bind=self.engine)
Session = sessionmaker(bind=self.engine)
self.session = Session()
# print("Agora objectec created ok")
# ...
def read_db_to_df(self, **kwargs):
""" Reads a database and transforms into a pandas.DataFrame """
try:
default_reading_sql_args = {'con': self.session.connection()}
reading_sql_args = utils.merge_two_dicts(default_reading_sql_args, kwargs)
df = pd.read_sql(**reading_sql_args)
return df
except SQLAlchemyError as e:
# self.logger.error("Error reading db to df")
# self.logger.error(str(e).replace("\n", ""))
print(e)
return -20
MY_test_before_test.py
from sqlalchemy import MetaData, Column, Table, ForeignKey, select, PrimaryKeyConstraint, Index
from sqlalchemy import Integer, String
from sqlalchemy import create_engine
from sqlalchemy.schema import CreateTable, DropTable
from sqlalchemy.ext.declarative.api import DeclarativeMeta
from agora_db.agora_helper import AgoraHelper
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.sql.functions import current_timestamp, current_user
engine = create_engine('sqlite:///:memory:')
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()
#This is how the database models look like:
class Schema(object):
""" A simple schema to provide a common argument for all tables """
__table_args__ = {"schema": "ifsmtd"}
class AbstractTable(object):
"""
A class that sets how the class is represents its objects as string.
"""
def __repr__(self):
""" Returns the object representation in string fromat in a way that can be used to reconstruct the object.
This returns an 'official' string representation of the object.
:Example:
"""
from sqlalchemy.inspection import inspect
table_inst = inspect(self)
table_name = type(self).__name__
column_key_value = [(c_attr.key, getattr(self, c_attr.key)) for c_attr in table_inst.mapper.column_attrs]
fields_str = ", ".join(["%s='%s'" % (c[0], c[1]) for c in column_key_value])
return "<" + table_name + "(" + fields_str + ")>"
class MyBasicTable(Base, Schema, AbstractTable):
__tablename__ = "mybasic_table"
timest_mov = Column(TIMESTAMP, primary_key=True, nullable=False, server_default=current_timestamp())
id_info = Column(String, primary_key=True, nullable=False)
id_wf = Column(String, primary_key=True, nullable=False)
process_name = Column(String, primary_key=True, nullable=False)
error_type = Column(String, primary_key=True, nullable=False)
resolution_status = Column(Boolean)
aud_timest_umo = Column(TIMESTAMP, server_default=current_timestamp())
aud_id_user = Column(String, server_default=current_user())
__table_args__ = (
PrimaryKeyConstraint('timest_mov', 'id_info', 'id_wf', 'process_name', 'error_type', name='pk_idx_mybasic_table'),
Index('pk_idx_mybasic_table', 'timest_mov', 'id_info', 'id_wf', 'process_name', 'error_type', unique=True),
{"schema": "ifsmtd"}
)
dbhelper = DatabaseHelper()
dbhelper.engine = engine
dbhelper.session = session
query = session.query(MyBasicTable.timest_mov.label("timest_mov"),
MyBasicTable.id_info .label("id_info "),
MyBasicTable.id_wf.label("id_wf"),
MyBasicTable.process_name.label("process_name"),
MyBasicTable.error_type.label("error_type"),
MyBasicTable.resolution_status.label("resolution_status")
)\
.distinct(MyBasicTable.id_jira.label("id_jira"))
df = dbhelper.read_db_to_df(sql=query.statement)
print(df)
The Error I get is:
(sqlite3.OperationalError) no such table: ifsmtd.mybasic_table
How could I do the test.
Looking at the code, there seems to be missing a call to Base.metadata.create_all(engine). This would create the initial database schema. After this call the tables are empty and you need to populate them.
Add the above statement just before using the database, but after defining the tables.
Base.metadata.create_all(engine) # This will create the schema!
dbhelper = DatabaseHelper()
On the use of sqlite: I have also gone that route and bumped into the fact that by default sqlite does not check foreign key constraints (it can be enabled!). There may be more differences!
I have an old question looking for a fresh answer. I've tried the recipes presented in the similar but somewhat aged question "Start a flask application in a seperate thread", and some other similar solutions found in other posts.
The long and short of it is, I need to start a flask application in a 'background' thread, such that a wxPython GUI can run in the foreground. The solutions presented here seem to no longer have the desired effect. The flask app starts and the GUI never runs.
My suspicion is, the existing answers are out of date. That said, I'm open to the possibility that I've mangled something else that's hosing it up, please have a peek and advise accordingly.
Thanks for your eyeballs and brain cycles :)
My code follows.
#!/usr/bin/env python
"""
integrator.py (the app)
"""
import wx
from pubsub import pub
from flask import Flask
from flask_graphql import GraphQLView
from models import db_session
from schema import schema
from models import engine, db_session, Base, Idiom
flaskapp = Flask(__name__)
flaskapp.debug = True
flaskapp.add_url_rule(
'/graphql',
view_func=GraphQLView.as_view(
'graphql',
schema=schema,
graphiql=True
)
)
flaskapp.run(threaded=True,use_reloader=False)
#flaskapp.teardown_appcontext
def shutdown_session(exception=None):
db_session.remove()
class IntegratorTarget(wx.TextDropTarget):
def __init__(self, object):
wx.DropTarget.__init__(self)
self.object = object
def OnDropText(self, x, y, data):
print(f"<publish>{data}</publish>")
pub.sendMessage('default', arg1=data)
return True
class IntegratorFrame(wx.Frame):
def __init__(self, parent, title):
super(IntegratorFrame, self).__init__(parent, title = title,size = wx.DisplaySize())
self.panel = wx.Panel(self)
box = wx.BoxSizer(wx.HORIZONTAL)
dropTarget = IntegratorTarget(self.panel)
self.panel.SetDropTarget(dropTarget)
pub.subscribe(self.catcher, 'default')
self.panel.SetSizer(box)
self.panel.Fit()
self.Centre()
self.Show(True)
def catcher(self,arg1):
data = arg1
print(f"<subscribed>{data}</subscribed>\n\n")
return
ex = wx.App()
Base.metadata.create_all(bind=engine)
IntegratorFrame(None,'Praxis:Integrator')
ex.MainLoop()
-- eof --
""" models.py """
from sqlalchemy import *
from sqlalchemy.orm import (scoped_session, sessionmaker, relationship, backref)
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///.praxis/lexicon/unbound.db3', convert_unicode=True)
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
Base = declarative_base()
# We will need this for querying
Base.query = db_session.query_property()
class Idiom(Base):
__tablename__ = "idiomae"
id = Column(Integer, primary_key=True)
src = Column(String) # the text of the drag/paste operation
taxonomy = Column(String) # the type of resource referenced in the drag/paste operation
localblob = Column(String) # local path to media referenced in 'src'
timestamp = Column(DateTime) # date and time of capture
-- eof --
""" schema.py """
import graphene
from graphene import relay
from graphene_sqlalchemy import SQLAlchemyObjectType, SQLAlchemyConnectionField
from models import db_session, Idiom as IdiomaticModel
class Idiom(SQLAlchemyObjectType):
class Meta:
model = IdiomaticModel
interfaces = (relay.Node, )
class Query(graphene.ObjectType):
node = relay.Node.Field()
# Allows sorting over multiple columns, by default over the primary key
all_idioms = SQLAlchemyConnectionField(Idiom.connection)
# Disable sorting over this field
# all_departments = SQLAlchemyConnectionField(Department.connection, sort=None)
schema = graphene.Schema(query=Query)
I see where you tell Flask to be multi-threaded, but I don't see where you're starting up the Flask app in a thread. I expected to see something like
app = Flask(__name__)
# any extra configuration
def webserver():
app.run(use_reloader=False)
web_thread = threading.Thread(target=webserver)
web_thread.start()
... continue on with the main thread
I have a working example you can crib from here. Note the need to use appropriate locking of any data structures shared between the primary thread and the threads running Flask.
I've written a few spiders that pull similar data from different sources. I've also written a pipeline that allows this data to be put in a database. I want to be able to use the same code for multiple spiders to output to different tables, named dynamically from the spider name.
Here is the pipeline.py code:
class DbPipeline(object):
def __init__(self):
"""
Initialises database connection and sessionmaker.
Creates table if it doesn't exist.
"""
engine = db_connect()
create_output_table(engine)
self.Session = sessionmaker(bind=engine)
def process_item(self, item, spider):
"""
Saves scraped products in database
"""
exists = self.check_item_exists(item)
if not exists:
session = self.Session()
product = Products(**item)
try:
session.add(product)
session.commit()
except:
session.rollback()
raise
finally:
session.close()
return item
def check_item_exists(self,item):
session = self.Session()
product = Products(**item)
result = session.query(Products).filter(Products.title == item['title']).first()
return result is not None
And here is the model.py file:
DeclarativeBase = declarative_base()
def create_output_table(engine):
DeclarativeBase.metadata.create_all(engine)
def db_connect():
"""
Connects to database from settings defined in settings.py
Returns an sqlalchemy engine instance
"""
return create_engine(URL(**settings.DATABASE))
class Products(DeclarativeBase):
"""Sqlalchemy table model"""
__tablename__ = "name"
id = Column(Integer, primary_key=True)
title = Column('title', String(200))
price = Column('price', String(10), nullable=True)
url = Column('url', String(200), nullable=True)
What i'm trying to do is get the __tablename__ variable to be the same as the spider name, which I can easily do in the process_item function as it is passed a spider object and can use spider.name and assign it to a class variable, however the function will run after the table is created/defined. How can I go about getting the spider name outside of the process_item function in the pipelines.py file?
Edit: I've tried the solutions listed in How to access scrapy settings from item Pipeline however access to the 'settings' doesn't give me access to the attributes assigned to the current spider running. I need to dynamically get the name of the spider based on what spider is running the pipelines. Thanks
It's pretty easy to get current spider name in your create_output_table:
class DbPipeline(object):
#classmethod
def from_crawler(cls, crawler):
return cls(crawler.spider.name)
def __init__(self, spider_name):
"""
Initializes database connection and sessionmaker.
Creates deals table.
"""
engine = db_connect()
create_output_table(engine, spider_name)
......
and (in models.py):
def create_output_table(engine, spider_name):
# now you have your spider_name
DeclarativeBase.metadata.create_all(engine)
The problem here is that Scrapy process your models.py file before your pipelines.py. So you need to find a way to generate your SQLAlchemy model later. You can use this thread as a starting point: Dynamically setting __tablename__ for sharding in SQLAlchemy?
I'm following the Flask-SQLAlchemy tutorial. I have Flask 0.9, sqlalchemy 0.7.8 and flask-sqlalchemy 0.16 on python 2.6. (and I work with eclipse)
(The tuto is here : http://packages.python.org/Flask-SQLAlchemy/models.html)
I have 2 classes : a man and a wallet. There is a 1-1 relationship. (Each man has his own wallet)
class Man(db.Model):
sid = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), unique=False)
wallet = db.relationship('Wallet', backref='man', lazy='dynamic', uselist=False)
def __init__(self, wallet):
self.wallet = wallet
class Wallet(db.Model):
sid = db.Column(db.Integer, primary_key=True)
account = db.Column(db.Integer)
manId = db.Column(db.Integer, db.ForeignKey('man.sid'))
def __init__(self, account):
self.account = account
In my "main" module, I create my database :
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:PATH'
db = SQLAlchemy(app)
In this very same module, I try to attach a Wallet to a Man :
if __name__ == "__main__":
db.create_all()
w1 = Wallet(132)
w2 = Wallet(18)
db.session.add(w1)
db.session.add(w2)
db.session.commit()
man1 = Man(w1)
db.session.add(man1)
db.session.commit()
But I get this error :
TypeError: 'Wallet' object is not iterable
I fail to understand why such an error appears. What is the right way of adding a mapped object ?
PS : I've been on the SQLAlchemy tutorial and I believe that they would declare things differently :
class Man(db.Model):
sid = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), unique=False)
wallet = db.relationship('Wallet', backref='man', lazy='dynamic', uselist=False)
manId = db.Column(db.Integer, db.ForeignKey('man.sid'))
def __init__(self, wallet):
self.wallet = wallet
class Wallet(db.Model):
sid = db.Column(db.Integer, primary_key=True)
account = db.Column(db.Integer)
def __init__(self, account):
self.account = account
Which tutorial should I trust ?
Thank you very much !
I fail to understand why such an error appears. What is the right way of adding a mapped object ?
Notice that when you configure you wallet relationship you use lazy="dynamic" option. This way you are setting up a dynamic relationship. As it is designed to be used with large collections it doesn't really makes much sense to use it with one-to-one relationship.
At the same time it alters the way you can assign to your scalar relationship, i.e. you cannot assing your single object directly:
self.wallet = wallet
but you must use an iterable
self.wallet = [wallet]
So you have two solutions here: either assign collection of one element as shown above or better yet stop using dynamic collections for this relationship.