I have an association table that contains relationships between two other SQLAlchemy models that I would like to delete:
class ItemCategories(db.Model):
id = Column(Integer, primary_key=True)
item_id = Column(Integer, ForeignKey("item.id"))
category_id = Column(Integer, ForeignKey("category.id"))
# ... other fields
The old syntax was to use something like:
db.session.query(ItemCategories).filter_by(category_id=5).filter(ItemCategories.name="Shelved").delete()
But with the newer syntax, I tried:
db.session.execute(db.select(ItemCategories).filter_by(category_id=5).filter(ItemCategories.name="Shelved").delete())
But this errored with:
AttributeError: 'Select' object has no attribute 'delete'
Flask-SQLAlchemy suggests doing:
db.session.delete(Model Object)
But this only deletes a single row, and I would like to delete multiple rows at once. I know I can loop through all the rows and do a session delete one-by-one, but would prefer a bulk delete instead like with the session.query line.
Is there a way to do multiple deletes with db.session.execute()?
Related
I have a couple of tables with the following many-to-many relationship
def TableOne(db.Model):
__tablename__ = "table_one"
id = db.Column(db.Integer, primary_key=True)
table_twos = db.relationship(
"TableTwo", secondary=relationship_table, lazy="subquery"
)
# Some other attributes
def TableTwo(db.Model):
__tablename__ = "table_two"
id = db.Column(db.Integer, primary_key=True)
relationship_table = db.Table(
"relationship_table",
db.Column("table_one_id", db.Integer, db.ForeignKey("table_one.id"), primary_key=True),
db.Column(
"table_two_id",
db.Integer,
db.ForeignKey("table_two.id"),
primary_key=True,
),
)
Normally, I've worked on small projects, and I can delete all the relationships as follows
tables = db.session.query(TableOne).all()
for t in tables:
t.table_twos = []
db.session.flush()
db.session.commit()
table_twos = db.session.query(TableTwo).all()
for t in table_twos:
db.session.delete(t)
db.session.flush()
db.session.commit()
However, since I am working with millions of rows I can't load them all into memory. If I try to just delete all the TableTwo rows, it gives me an error about foreign keys.
How can I delete all of the relationships at once and then delete the TableTwo rows all at once?
Thank you
If you want to delete all rows in a table, it's much faster to use TRUNCATE, that simply trash the table files on disk instead of deleting every row one by one. It will also reclaim disk space, unlike DELETE which will only create free space in the table file.
If there are foreign keys:
You can TRUNCATE the referencing table (TableTwo).
But you cannot truncate the referenced table because that would break the foreign key references. But if want to delete all rows in the referencing and referenced table, just truncate both:
TRUNCATE table1, table2;
If the referencing and referenced table are listed in the same truncate command, it will work. Do not use two independent TRUNCATE commands, or postgres will refuse to break your foreign keys (as it should!).
Note if you want row deletions in table1 to also delete the referencing rows in table2, you must set your foreign key to "ON DELETE CASCADE". Then you could use DELETE on table1, but to delete all rows TRUNCATE is much faster.
I am working on an app using python3 and SqlAlchemy for SQLite3 database management. I have some tables that have a Many to Many relationship. I've created an association table to handle this relationship.
Class Machine(Base):
__tablename__ 'machine'
machine_ID = Column(Integer, primary_key=True)
etc...
Class Options(Base):
__tableName__ 'options'
options_ID = Column(Integer, primary_key=True)
etc...
The association table
Machine_Options = table('machine_options', Base.metadata,
Column('machine_FK', Integer, ForeignKey('machine.machine_ID'),
primary_key=True),
Column('options_FK',Integer, ForeignKey('options.options_ID'),
primary_key=True))
All the items for the Machine and Options are inserted independently. When I want to associate a machine with an option I use an append query which works very well.
My problem is when I want to break this association between a machine and an option. I have tried a direct row deletion from the association table using a FILTER() clause on the machine_FK and the options_FK but SqlAlchemy gives me an error informing me that 'Machine_Options' table has no field 'machine_FK'.
I have tried to remove the row from 'Machine_Options' indirectly using joins with the machine and options table but received another error that I can not delete or update using joins.
I am looking for the code to only delete a row from the association table without affecting the original machine or options table.
So far my internet search has been fruitless.
The answer to my problem is to use myparent.children.remove(somechild)
The association is made using machine.children.append(option)
Using the same code as the 'append' and substituting 'remove' unmakes the association
The code:
def removeOption(machineKey, OptionKey):
session = connectToDatabase()
machineData = session.query(Machine).filter(Machine.machine_ID == machineKey).one()
optionData = session.query(Options).filter(Options. options_ID == OptionKey).one()
machineData.children.remove(optionData)
session.add(machineData)
session.commit()
session.close()
How might one use SqlAlchemy dynamic schema translation with foreign key relationships using the declarative ORM? Below is an example, of what I am trying to do for a multi-tenant DB (each 'user' mapping to a schema). I am not sure how one goes about setting ForeignKey constraints in SQLAlchemy
Assuming the following table declarations:
class BaseMixin():
__table_args__ = {'schema':'dynamic'}
class Image(Base, BaseMixin):
__tablename__ == 'image'
id = Column(Integer, primary_key=True, autoincrement=True)
class Point(Base, BaseMixin):
__tablename__ == 'point'
fk = Column(Integer, ForeignKey("image.id", ondelete="CASCADE"))
The Point table has a FK relationship to the Image table. I am getting the following error when I attempt to query these tables:
NoReferencedTableError: Foreign key associated with column 'point.fk' could not find table 'image' with which to generate a foreign key to target column 'id'
The session is parametrized with the following:
session.connection(execution_options={
"schema_translate_map": {"dynamic": "my_custom_schema"}})
From the docs, it looks like the schema_translate_map update on the session should be replacing dynamic with the user's schema. Oddly, the table instantiation is happening as expected. The error is popping up when attempting to query.
You may need to specify the schema in the ForeignKey i.e. update this:
fk = Column(Integer, ForeignKey("image.id", ondelete="CASCADE"))
to:
fk = Column(Integer, ForeignKey("dynamic.image.id", ondelete="CASCADE"))
I have been having some trouble with my understanding of how foreign keys work in sqlite3.
Im trying to get the userid (james) in one table userstuff to appear as foreign key in my otherstuff table. Yet when I query it returns None.
So far I have tried:
Enabling foreign key support
Rewriting a test script (that is being discussed here) to isolate issue
I have re-written some code after finding issues in how I had initially written it
After some research I have come across joins but I do not think this is the solution as my current query is an alternative to joins as far as I am aware
Code
import sqlite3 as sq
class DATAB:
def __init__(self):
self.conn = sq.connect("Atest.db")
self.conn.execute("pragma foreign_keys")
self.c = self.conn.cursor()
self.createtable()
self.defaultdata()
self.show_details() # NOTE DEFAULT DATA ALREADY RAN
def createtable(self):
self.c.execute("CREATE TABLE IF NOT EXISTS userstuff("
"userid TEXT NOT NULL PRIMARY KEY,"
" password TEXT)")
self.c.execute("CREATE TABLE IF NOT EXISTS otherstuff("
"anotherid TEXT NOT NULL PRIMARY KEY,"
"password TEXT,"
"user_id TEXT REFERENCES userstuff(userid))")
def defaultdata(self):
self.c.execute("INSERT INTO userstuff (userid, password) VALUES (?, ?)", ('james', 'password'))
self.c.execute("INSERT INTO otherstuff (anotherid, password, user_id) VALUES (?, ?, ?)",('aname', 'password', 'james'))
self.conn.commit()
def show_details(self):
self.c.execute("SELECT user_id FROM otherstuff, userstuff WHERE userstuff.userid=james AND userstuff.userid=otherstuff.user_id")
print(self.c.fetchall())
self.conn.commit()
-----NOTE CODE BELOW THIS IS FROM NEW FILE---------
import test2 as ts
x = ts.DATAB()
Many thanks
A foreign key constraint is just that, a constraint.
This means that it prevents you from inserting data that would violate the constraint; in this case, it would prevent you from inserting a non-NULL user_id value that does not exist in the parent table.
By default, foreign key constraints allow NULL values. If you want to prevent userstuff rows without a parent row, add a NOT NULL constraint to the user_id column.
In any case, a constraint does not magically generate data (and the database cannot know which ID you want). If you want to reference a specific row of the parent table, you have to insert its ID.
I have a sqlite3 database accessing it with SQLAlchemy in python3.
I want to add a new and drop an old column with the database-migation tool alembic. Simple example:
class Model(_Base):
__tablename__ = 'Model'
_oid = Column('oid', sa.Integer, primary_key=True)
_number_int = sa.Column('number_int', sa.Integer)
Should be after migration like this:
class Model(_Base):
__tablename__ = 'Model'
_oid = Column('oid', sa.Integer, primary_key=True)
_number_str = sa.Column('number_str', sa.String(length=30))
The relevant point here is that there is data in _number_int that should be converted into _number_str like this:
number_conv = {1: 'one', 2: 'two', 3: 'three'}
_number_str = number_conv[_number_int]
Is there an alembic way to take care of that? It means if alembic itself take care of cases like that in its concept/design?
I want to know If I can use alembic tools for that or if I have to do my own extra code for that.
Of course the original data is a little bit more complex to convert. This is just an example here.
Here is alembic operation reference. There is a method called bulk_insert() for bulk inserting content, but nothing for migrating existing content. It seems alembic doesn't have it built-in. But you can implement data migration yourself.
One possible approach is described in the article "Migrating content with alembic". You need to define intermediate table inside your migration file, which contains both columns (number_int and number_str):
import sqlalchemy as sa
model_helper = sa.Table(
'Model',
sa.MetaData(),
sa.Column('oid', sa.Integer, primary_key=True),
sa.Column('number_int', sa.Integer),
sa.Column('number_str', sa.String(length=30)),
)
And use this intermediate table to migrate data from old column to the new one:
from alembic import op
def upgrade():
# add the new column first
op.add_column(
'Model',
sa.Column(
'number_str',
sa.String(length=30),
nullable=True
)
)
# build a quick link for the current connection of alembic
connection = op.get_bind()
# at this state right now, the old column is not deleted and the
# new columns are present already. So now is the time to run the
# content migration. We use the connection to grab all data from
# the table, convert each number and update the row, which is
# identified by its id
number_conv = {1: 'one', 2: 'two', 3: 'three'}
for item in connection.execute(model_helper.select()):
connection.execute(
model_helper.update().where(
model_helper.c.id == item.id
).values(
number_str=number_conv[item.number_int]
)
)
# now that all data is migrated we can just drop the old column
# without having lost any data
op.drop_column('Model', 'number_int')
This approach is a bit noisy (you need to define table manually), but it works.