I'm trying to learn how to create python-based back-ends from some existing data that i have collected. I've come to realize that i definitely want to use sqlalchemy and that flask seems like a good library to go with it. My problem is that even after many hours of reading the sqlalchemy docs and browsing various answers on stackexchange i still don't understand how i can reshape data from an existing table into an object with a completely different structure.
The transformation i want to do is very concrete. I want to go from this structure in my MariaDB table:
Columns: company_name, date, indicators(1...23)
To this json output generated from a serialized class object:
{
"company_name[1]":
{
"indicator_name[1]":
{
"date[1]": "indicator_name[1].value[1]",
"date[2]": "indicator_name[1].value[2]",
"date[3]": "indicator_name[1].value[3]",
"date[4]": "indicator_name[1].value[4]",
"date[5]": "indicator_name[1].value[5]"
},
"indicator_name[2]":
{
"date[1]": "indicator_name[2].value[1]",
"date[2]": "indicator_name[2].value[2]",
"date[3]": "indicator_name[2].value[3]",
"date[4]": "indicator_name[2].value[4]",
"date[5]": "indicator_name[2].value[5]"
},
I found a great tutorial with which i can output the entire table record by record but the structure is not what i want, and i don't think creating the desired structure on the front-end makes sense in this case.
Here is the code that outputs the entire table to json record by record:
from flask import Flask, jsonify
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import PrimaryKeyConstraint
from sqlalchemy import orm
from sqlalchemy import select, func
from sqlalchemy import Column, Integer, String, ForeignKey
from flask_marshmallow import Marshmallow
import decimal
import flask.json
class MyJSONEncoder(flask.json.JSONEncoder): # Enables decimal queries for the API
def default(self, obj):
if isinstance(obj, decimal.Decimal):
# Convert decimal instances to strings.
return str(obj)
return super(MyJSONEncoder, self).default(obj)
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql+pymysql://USER:PASS#localhost:3306/kl_balance_sheets'
app.json_encoder = MyJSONEncoder
db = SQLAlchemy(app)
ma = Marshmallow(app)
# Bind declarative base to engine
db.Model.metadata.reflect(db.engine)
class CompanyData(db.Model):
__table__ = db.Model.metadata.tables['kl_balance_sheets']
class CompanyDataSchema(ma.ModelSchema):
class Meta:
model = CompanyData
#app.route('/')
def index():
company_data = CompanyData.query.all()
company_data_schema = CompanyDataSchema(many=True)
output = company_data_schema.dump(company_data).data
return jsonify({'company_data' : output})
if __name__ == '__main__':
app.run(debug=True)
My main question i guess is: How do i edit this code to produce the desired json?
What i think i should do is to create a custom constructor and then feed that into the index function but i can't figure out how to concretely do that. The two options i've come across are:
#orm.reconstructor
def init_on_load(self):
#do custom stuff
or:
class Foo(db.Model):
# ...
def __init__(**kwargs):
super(Foo, self).__init__(**kwargs)
# do custom stuff
To me this seems like a basic operation any flask-marshmallow user would be doing regularly. Could someone please explain how sql data is normally inserted into an object with a new structure and then serialized? In my case, do i need to change things mainly on the metadata, object or marshmallow level? I'm surprised i can't find some good examples of this.
Related
I'm debugging existing code. I'm trying to find out the intention of the obviously wrong access to .dicts of a peewee Model in the warning statement in MyDbBackend.store and how I could correct that.
I guess that the warning message should add more detailed output to the model which could not be saved. However, the .dicts attribute exists in orm.BaseQuery class, only.
The output message is currently not very helpful. I want to provide an improved warning message given that the i.save fails. With "improved" i mean to provide some meta informations about the record which failed to be saved.
So, how can i obtain the BaseQuery from the model and what would .dicts output, then? Would that information be useful in the context of the warning message?
import peewee as orm
database = orm.Proxy()
class ModelBase(orm.Model):
class Meta:
database = database
class MyModel(ModelBase):
dtfield = orm.DateTimeField(null=True)
intfield = orm.IntegerField(null=True)
floatfield = orm.FloatField(null=True)
class MyDbBackend:
def __init__(self, database):
self.db = database
self.records = [] # Holds objects derived from ModelBase
[...]
def store(self):
with self.db.atomic():
for i in self.records:
try:
i.save()
except Exception as e:
logger.warning("could not save record: {}".format(i.dicts()))
raise e
self.clear()
->
logger.warning("could not save record: {}".format(i.dicts()))
AttributeError: 'MyModel' object has no attribute 'dicts'
I guess that the original code was meant to make use of playhouse.shortcuts.model_to_dict.
This is the only idea I have why the original code uses i.dict().
Perhaps some misunderstanding.
import peewee as orm
from playhouse.shortcuts import model_to_dict
[...]
logger.warning(f"Model dict: {model_to_dict(i, recurse = True, max_depth = 2)}")
[...]
from flask import Flask, jsonify, request
from flask_restful import Api, Resource
import jsonpickle
app = Flask(__name__)
api = Api(app)
# creating an empty dictionary and initializing user id to 0.. will increment everytime a person makes a POST request
user_dict = {}
user_id = 0
# Define a class and pass it a Resource. These methods require an ID
class User(Resource):
#staticmethod
def get(path_user_id):
return jsonify(jsonpickle.encode(user_dict.get(path_user_id, "This user does not exist")))
When I boot up the server, I go to visit the /users/1 endpoint. Since the dictionary is empty, it doesn't exist. I get thrown a KeyError, so my temporary solution was to change my dictionary accessor from user_dict[path_user_id] to .get(path_user_id, "This user does not exist"). Is there a better way to handle this? I'm not sure if this is useful or not, but my dictionary consists of integer keys which map to a "Person" class which contains information about the person (name, age, address, etc)
A 404 status code represents "Resource not found", which perfectly suits your use-case
from flask import abort
...
def get(path_user_id):
if path_user_id not in user_dict:
abort(404)
...
I am creating a rest-api in python flask-restful that will take the string argument( 'Retailer' for example) and automatically generate the crud template for it on attribute id. Retailer has some more attributes that I have migrated using ORM. Now I need to take the class name 'Retailer' which is of type str and convert it into a resource and auto generate crud. My code is below.
if __name__ != '__main__':
from flask import Flask,jsonify
from flask_restful import Resource, Api, reqparse
import parse_json
import create_models
app = Flask(__name__)
api = Api(app)
temp = parse_json.class_name
def get(self,id):
#code to be written here
vars()[temp] = type(temp,(Resource,),{'get' : get})
api.add_resource(vars()[temp], '/{0}/<int:id>'.format(temp))
app.run(port = 5000)
I have created only the get method on this resource and dont know what to write in it to get all the attributes by its id. Please help !
i i well understand, for example to retrieve data from your db according to a selected id
cur = con.cursor()
cur.execute("SELECT * FROM dataset where id='"+id+"'")
data = cur.fetchall()
I am trying to build a database driver for Peewee and i'm having trouble getting the save() method to fill in the primary key/id for objects. Here's some sample code:
from datetime import date
from peewee import BooleanField
from peewee import CharField
from peewee import DateField
from peewee import ForeignKeyField
from peewee import IntegerField
from peewee import Model
from SQLRelay import PySQLRDB
from sqlrelay_ext import SQLRelayDatabase
DB = SQLRelayDatabase('test2', host='<host>', user='<un>', password='<pwd>')
class BaseModel(Model):
class Meta:
database = DB
class Person(BaseModel):
name = CharField()
birthday = DateField()
is_relative = BooleanField()
class Pet(BaseModel):
owner = ForeignKeyField(Person, backref='pets')
name = CharField()
animal_type = CharField()
DB.connect()
Person.create_table(safe=False)
Pet.create_table(safe=False)
uncle_bob = Person(name='Bob', birthday=date(1960, 1, 15), is_relative=True)
uncle_bob.save() # bob is now stored in the database
print('Uncle Bob id: {}'.format(uncle_bob.id))
print('Uncle Bob _pk: {}'.format(uncle_bob._pk))
Both uncle_bob.id and uncle_bob._pk are None after .save(). From digging into the peewee.py code, it seems that the _WriteQuery.execute() method is supposed to set the _pk attribute, but that isn't happening. My best guess is that the cursor implementation isn't acting properly. Does anyone have more insight than this that can maybe help me track down this problem?
Thanks!
Edit to answer:
For SQL Server, the following code allows you to return the last inserted id:
def last_insert_id(self, cursor, query_type=None):
try:
cursor.execute('SELECT SCOPE_IDENTITY()')
result = cursor.fetchone()
return result[0]
except (IndexError, KeyError, TypeError):
pass
In your SQLRelayDatabase implementation, you will probably need to correctly implement the last_insert_id() method. For python db-api 2.0 drivers, this typically looks like cursor.lastrowid.
The default implementation is:
def last_insert_id(self, cursor, query_type=None):
return cursor.lastrowid
Where cursor is the cursor object used to execute the insert query.
Databases like Postgresql do not implement this -- instead you execute an INSERT...RETURNING query, so the Postgres implementation is a bit different. The postgres implementation ensures that your insert query includes a RETURNING clause, and then grabs the id returned.
Depending on your DB and the underlying DB-driver, you'll need to pull that last insert id out somehow. Peewee should handle the rest assuming last_insert_id() is implemented.
I've got a bunch of GIS tables in my model that I created in flaskSQLAlchemy. Each of these models has a 'geom' field which is a WKB object.
Which need to be JSON serialized into WKT or geojson, So that The API GET call would work.
I tried to use geoalchemy2 functions, but I'm stuck.
I use a flask marshmallow/marshmallow-sqlalchemy combo, and I tried something like the following, with no luck.
from geoalchemy2 import functions
from marshmallow import fields
class WKTSerializationField(fields.Field):
def _serialize(self, value, attr, obj):
if value is None:
return value
else:
if type(value).__name__ == 'WKBElement':
return functions.ST_AsEWKT(value)
else:
return None
class GISModelTableSchema(ma.ModelSchema):
class Meta:
model = GISModelTable
geom = WKTSerializationField(attribute="geom")
Please provide a code example if you can, how to serialize/deserialize a field in marshmallow alchemy. Or any answer is welcomed at this point.
Try to use marshmallow-sqlalchemy 'fields.Method()' and in the method use another method 'to_shape' from geoalchemy2.shape package. This will help you with serialization issue.
#!schemas.py
from marshmallow import fields
from marshmallow_sqlalchemy import ModelSchema
from geoalchemy2.shape import to_shape
from .models import YourModel
class YourModelSchema(ModelSchema):
your_geom_field = fields.Method("geom_to_dict")
#staticmethod
def geom_to_dict(obj):
point = to_shape(obj.your_geom_field)
return {
lat: point.y,
lon: point.x
}
class Meta:
model = YourModel
exclude = ("your_geom_field")
this migh help you with serialization, for desirialization you may read more detailed in geoalchemy2 api reference
Try to code all required fields by yourself, you may get more specific serialization in a form you want