SQLite foreign key, on delete cascade, and SQLITE_CONSTRAINT - node.js

Overview:
I have a parent / child table relationship where the child may contain 2:n records with FK's back to the parent. When attempting to delete from the parent, I get a SQLITE_CONSTRAINT error. This is unexpected as I have FK's enabled, have the child registered with ON DELETE CASCADE, and a new enough SQLite version.
However: My child table originally did not have ON DELETE CASCADE. I added (and enabled FK's) after data had been added to parent/child. From there, I renamed the original child & created a new table with the constraint, and finally moved to the new table.
Table layout as follows:
CREATE TABLE IF NOT EXISTS message (
message_id INTEGER PRIMARY KEY,
area_tag VARCHAR NOT NULL,
message_uuid VARCHAR(36) NOT NULL,
reply_to_message_id INTEGER,
to_user_name VARCHAR NOT NULL,
from_user_name VARCHAR NOT NULL,
subject, /* FTS # message_fts */
message, /* FTS # message_fts */
modified_timestamp DATETIME NOT NULL,
view_count INTEGER NOT NULL DEFAULT 0,
UNIQUE(message_uuid)
);
CREATE INDEX IF NOT EXISTS message_by_area_tag_index
ON message (area_tag);
CREATE VIRTUAL TABLE IF NOT EXISTS message_fts USING fts4 (
content="message",
subject,
message
);
CREATE TRIGGER IF NOT EXISTS message_before_update BEFORE UPDATE ON message BEGIN
DELETE FROM message_fts WHERE docid=old.rowid;
END;
CREATE TRIGGER IF NOT EXISTS message_before_delete BEFORE DELETE ON message BEGIN
DELETE FROM message_fts WHERE docid=old.rowid;
END;
CREATE TRIGGER IF NOT EXISTS message_after_update AFTER UPDATE ON message BEGIN
INSERT INTO message_fts(docid, subject, message) VALUES(new.rowid, new.subject, new.message);
END;
CREATE TRIGGER IF NOT EXISTS message_after_insert AFTER INSERT ON message BEGIN
INSERT INTO message_fts(docid, subject, message) VALUES(new.rowid, new.subject, new.message);
END;
CREATE TABLE IF NOT EXISTS message_meta (
message_id INTEGER NOT NULL,
meta_category INTEGER NOT NULL,
meta_name VARCHAR NOT NULL,
meta_value VARCHAR NOT NULL,
UNIQUE(message_id, meta_category, meta_name, meta_value),
FOREIGN KEY(message_id) REFERENCES message(message_id) ON DELETE CASCADE
);
At startup, directly after attaching to the DB's I ensure FK's are enabled:
PRAGMA foreign_keys = ON;
Other details:
SQLite version: 3.7.17
Access: node-sqlite3
Exact error: Error: SQLITE_CONSTRAINT: FOREIGN KEY constraint failed
Is this caused by the fact that I later added the constraint? (See Update 1)
How do I fix this without losing data?
Update 1:
I can confirm that only select messages (I believe, messages that were in message before ON DELETE CASCADE as added to message_meta) cause the constraint error. Others delete just fine and properly take out associated message_meta records.

Answering my own question -- after some hours of trying various things I was able to find the issue(s):
When I originally added the ON DELETE CASCADE clause, I did so by renaming the original message_meta table to message_meta_backup, creating a new table with the clause, then moving the data into it: SELECT * FROM message_meta_backup INSERT INTO message_meta;. What I did not do was to drop the backup table.
Due to #1 or something related, something internal to my database became corrupted or confused.
What I tried (that did not work):
REINDEX;
Simply dropping the backup table: DROP TABLE message_meta_backup;
...and various other things I forget :)
What DID work:
What finally ended up working was a combination of dropping the backup table and completely rebuilding the database using the sqlite3 shell's .drop command:
> sqlite3 db/message.sqlite3
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> drop table message_meta_backup;
sqlite> .quit
> sqlite3 db/message.sqlite3 ".dump" >> message_dump.sql
rm db/message.sqlite3
> cat message_dump.sql | sqlite3 db/message.sqlite3
I'm now able to DELETE FROM message ... and have it properly cascade the delete to message_meta without the nasty error:
sqlite> DELETE FROM message WHERE message_id IN(SELECT message_id FROM message WHERE area_tag='some_area' ORDER BY message_id desc limit -1 offset 200);
sqlite>
(no error given!)

Related

Node.js and Oracle DB select query getting empty array in rows

const result = await connection.execute(
`SELECT * from no_example `, [], { maxRows: 1000 } // bind value for :id
);
but in result i always get empty rows
If you are inserting rows in another tool, or another program. Make sure that you COMMIT the data:
SQL> create table t (c number);
Table created.
SQL> insert into t (c) values (1);
1 row created.
SQL> commit;
Commit complete.
If you are insert using Node.js, look at the autoCommit attribute and connection.commit() function. Also see the node-oracledb documentation on Transaction Management.
Unrelated to your problem, but you almost certainly shouldn't be using maxRows. By default node-oracledb will return all rows. If you don't want all, then add some kind of WHERE clause or row-limiting clause to your query. If you expect a big number of rows, then use a result set so you can access consecutive batches of rows.

Execute stored procedure in Azure Data Platform - Post SQL Scripts

Based on the documentation below,
https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database
There is a feature to run post SQL script. Would it be possible to run stored procedure from there?
I have tried, it does not seem to be working and currently investigating.
Thanks for your information in advance.
I created a test to prove that the stored procedure can be called in the Post SQL scripts.
I created two tables:
CREATE TABLE [dbo].[emp](
id int IDENTITY(1,1),
[name] [nvarchar](max) NULL,
[age] [nvarchar](max) NULL
)
CREATE TABLE [dbo].[emp_stage](
id int,
[name] [nvarchar](max) NULL,
[age] [nvarchar](max) NULL
)
I created a sotred procedure.
create PROCEDURE [dbo].[spMergeEmpData]
AS
BEGIN
SET IDENTITY_INSERT dbo.emp ON
MERGE [dbo].[emp] AS target
USING [dbo].[emp_stage] AS source
ON (target.[id] = source.[id])
WHEN MATCHED THEN
UPDATE SET name = source.name,
age = source.age
WHEN NOT matched THEN
INSERT (id, name, age)
VALUES (source.id, source.name, source.age);
TRUNCATE TABLE [dbo].[emp_stage]
END
I will copy the csv file into my Azure SQL staging table [dbo].[emp_stage], then use stored porcedure [dbo].[spMergeEmpData] to transfer data from [dbo].[emp_stage] to [dbo].[emp].
Enter the stored procedure name exec [dbo].[spMergeEmpData] in the Post SQL scripts field.
I successfully debugged.
I can see the data are all in TABLE [dbo].[emp].

Sqlite3 delete by primary key

I am using Node with node-sqlite3. I cannot seem to delete rows by passing their primary key, but I can delete rows by passing some other column value.
Works
this.db.run("DELETE FROM MyTable WHERE Name = 'Bob'");
Does nothing
this.db.run("DELETE FROM MyTable WHERE _id = 3);
I have specified _id as my autoincrement / primary key. I can prove this by doing a PRAGMA and showing the debugger's results here, which clearly indicates _id as the pk
PRAGMA call
Debugger
I really don't know what to do as I get no errors from the db.run call

Bad distributed join plan: result table shard keys do not match

We are very new to memsql/mysql and we are trying to play around with a memsql installation.
It is installed on a CentOS7 virtual machine and we are running version 5.1.0 of MemSQL.
We are receiving the error from one of the queries we are attempting:
ERROR 1889 (HY000): Bad distributed join plan: result table shard keys do not match. Please contact MemSQL support at support#memsql.com.
On one of our queries
We have two tables:
CREATE TABLE `MyObjects` (
`Id` INT NOT NULL AUTO_INCREMENT,
`Name` VARCHAR(128) NOT NULL,
`Description` VARCHAR(256) NULL,
`Boolean` BIT NOT NULL,
`Int8` TINYINT NOT NULL,
`Int16` SMALLINT NOT NULL,
`Int32` MEDIUMINT NOT NULL,
`Int64` INT NOT NULL,
`Float` DOUBLE NOT NULL,
`DateCreated` TIMESTAMP NOT NULL,
SHARD KEY (`Id`),
PRIMARY KEY (`Id`)
);
CREATE TABLE `MyObjectDetails` (
`MyObjectId` INT,
`Int32` MEDIUMINT NOT NULL,
SHARD KEY (`MyObjectId`),
INDEX (`MyObjectId`)
);
And here is the SQL we are executing and getting the error.
memsql> SELECT mo.`Id`,mo.`Name`,mo.`Description`,mo.`Boolean`,mo.`Int8`,mo.`Int16`,
mo.`Int32`,mo.`Int64`,mo.`Float`,mo.`DateCreated`,mods.`MyObjectId`,
mods.`Int32` FROM
( SELECT
mo.`Id`,mo.`Name`,mo.`Description`,mo.`Boolean`,mo.`Int8`,
mo.`Int16`,mo.`Int32`,mo.`Int64`,mo.`Float`,mo.`DateCreated`
FROM `MyObjects` mo LIMIT 10 ) AS mo
LEFT JOIN `MyObjectDetails` mods ON mo.`Id` = mods.`MyObjectId` ORDER BY `Name` DESC;
ERROR 1889 (HY000): Bad distributed join plan: result table shard keys do not match. Please contact MemSQL support at support#memsql.com.
Does anyone know why we are receiving this error, and if there is a possible change we can make to help alleviate this issue?
The one thing we do know is it has something to do with the inner select as if I pull it out and do the join it works, however we only get 10 total rows from the join. What we are attempting is getting the top 10 from the main table and include all of the details from the right.
We also tried changing the MyObjectDetails table to have an empty SHARD KEY, but that resulted in the same error.
SHARD KEY()
We also added an auto-incrementing Id column to the details table and put the shard on that column, and yet still received the same error.
Thanks in advance for any help.
UPDATE:
I contacted MemSQL through email (huge props to their customer service by the way -- very fast response time, less than a couple hours)
But from what Mike stated I changed the table to be a REFERENCE table and removed the SHARD KEY part of the create table statement. Once I did this, I was able to run the queries. I am not 100% sure on what ramifications this will have but it fixed my issue at hand. Thanks
CREATE REFERENCE TABLE `MyObjects` (
`Id` INT NOT NULL AUTO_INCREMENT,
`Name` VARCHAR(128) NOT NULL,
`Description` VARCHAR(256) NULL,
`Boolean` BIT NOT NULL,
`Int8` TINYINT NOT NULL,
`Int16` SMALLINT NOT NULL,
`Int32` MEDIUMINT NOT NULL,
`Int64` INT NOT NULL,
`Float` DOUBLE NOT NULL,
`DateCreated` TIMESTAMP NOT NULL,
PRIMARY KEY (`Id`)
);
Thanks to Mike Gallegos for looking into this, adding a summary of his answer here:
The error message here is bad, but the reason for the error is that MemSQL does not currently support a distributed left join where the left side (the Limit subquery in this case) has a LIMIT operator. If you cannot rewrite the query to do the limit after the join, then you could change the MyObjects table to a reference table to work around the issue.

Inserting/Updating sqlite table from python program

I have a sqlite3 table as shown below
Record(WordID INTEGER PRIMARY KEY, Word TEXT, Wordcount INTEGER, Docfrequency REAL).
I want to create and insert data into this table if the table not exists else I want to update the table in such a way that only 'Wordcount' column get updated on the basis(Reference) of data in the column 'Word'. I am trying to execute this from a python program like
import sqlite3
conn = sqlite3.connect("mydatabase")
c = conn.cursor()
#Create table
c.execute("CREATE TABLE IF NOT EXISTS Record(WordID INTEGER PRIMARY KEY, Words TEXT, Wordcount INTEGER, Docfrequency REAL)")
#Update table
c.execute("UPDATE TABLE IF EXISTS Record")
#Insert a row of data
c.execute("INSERT INTO Record values (1,'wait', 9, 10.0)")
c.execute("INSERT INTO Record values (2,'Hai', 5, 6.0)")
#Updating data
c.execute("UPDATE Record SET Wordcount='%d' WHERE Words='%s'" %(11,'wait') )
But I can't update the table. On running the program I am getting the error message as
c.execute("UPDATE TABLE IF EXISTS Record")
sqlite3.OperationalError: near "TABLE": syntax error
How should I write the code to update the table ?
Your SQL query for UPDATE is invalid - see the documentation.
Also, I don't understand why you'd want to check for the table's existence when updating, given that just before that you're creating it if it doesn't exist.
If your goal is to update an entry if it exists or insert it if it doesn't, you might do it either by:
First doing an UPDATE and checking the number of rows updated. If 0, you know the record didn't exist and you should INSERT instead.
First doing an INSERT - if there's an error related to constraint violation, you know the entry already existed and you should do an UPDATE instead.

Resources