I have a sqlite3 database where the first column is the id and set as primary key with auto increment. I'm trying to insert the values from my python dictionary as such:
value = {'host': [], 'drive': [], 'percent': []}
soup = bs(contents, 'html.parser')
for name in soup.find_all("td", class_="qqp0_c0"):
hostname = name.parent.find('td').get_text()
drive = name.parent.find('td', class_="qqp0_c1").get_text()
used_percent = name.parent.find('td', class_="qqp0_c5").get_text()
value['host'].append(hostname)
value['drive'].append(drive)
value['percent'].append(used_percent)
#cur.executemany("INSERT INTO scrap VALUES (?, ?, ?)", hostname, drive, used_percent)
cur.execute("INSERT INTO scrap VALUES (?, ?, ?);", value)
I keep getting errors, my latest error seems to imply it needs an id value:
cur.execute("INSERT INTO scrap VALUES (?, ?, ?);", value)
sqlite3.OperationalError: table scrap has 4 columns but 3 values were supplied
Do I need to supply an id number?
This is the db schema:
CREATE TABLE scrap (
id INTEGER PRIMARY KEY AUTOINCREMENT,
hostname VARCHAR(255),
drive VARCHAR(255),
perc VARCHAR(255)
);
If the id column is auto-incrementing you don't need to supply a value for it, but you do need to "tell" the database that you aren't inserting it. Note that in order to bind a dictionary, you need to specify the placeholders by name:
cur.execute("INSERT INTO scrap (hostname, drive, perc) VALUES (:host, :drive, :percent);", value)
EDIT:
Following up on the discussion from the comments - the value dictionary should map placeholder names to their intended values, not list containing them:
soup = bs(contents, 'html.parser')
for name in soup.find_all("td", class_="qqp0_c0"):
hostname = name.parent.find('td').get_text()
drive = name.parent.find('td', class_="qqp0_c1").get_text()
used_percent = name.parent.find('td', class_="qqp0_c5").get_text()
value = {'host': hostname, 'drive': drive, 'percent': used_percent}
cur.execute("INSERT INTO scrap (hostname, drive, perc) VALUES (:host, :drive, :percent);", value)
Related
Having a dict of structured_files variable as below;
structured_files = {'158795_1635509614542_20211029151335135_27.1943_38.4319.jpg': ['158795', '1635509614542', '20211029151335135', '27.1943', '38.4319'], '2773170_1635255851988_20211026164412576_27.1836_38.4777.jpg': ['2773170', '1635255851988', '20211026164412576', '27.1836', '38.4777'], '2777171_1635330525915_20211027132846547_27.1875_38.4720.jpg': ['2777171', '1635330525915', '20211027132846547', '27.1875', '38.4720'], '2779392_1635150026780_20211025112027410_27.1627_38.4724.jpg': ['2779392', '1635150026780', '20211025112027410', '27.1627', '38.4724'], '3724797_1634893471399_20211022120432057_27.1334_38.5426.jpg': ['3724797', '1634893471399', '20211022120432057', '27.1334', '38.5426'], '5218186_1635161250087_20211025142730572_27.0694_38.4943.jpg': ['5218186', '1635161250087', '20211025142730572', '27.0694', '38.4943'], '5347212_1634987405913_20211023141006566_27.0454_38.5138.jpg': ['5347212', '1634987405913', '20211023141006566', '27.0454', '38.5138'], '8174846_1635335281021_20211027144801672_27.1039_38.3572.jpg': ['8174846', '1635335281021', '20211027144801672', '27.1039', '38.3572'], '9179383_1635321573332_20211027105933968_27.1251_38.3576.jpg': ['9179383', '1635321573332', '20211027105933968', '27.1251', '38.3576']}
Trying to insert in to mysql db code is follows;
For Creation;
Qcreate1 = "CREATE TABLE data_11 (id_key LONGTEXT NOT NULL, name VARCHAR(50) NOT NULL, date VARCHAR(50) NOT NULL, sample VARCHAR(50) NOT NULL, coord_y VARCHAR(50) NOT NULL, coord_x VARCHAR(50) NOT NULL, created datetime NOT NULL, key_id int PRIMARY KEY NOT NULL AUTO_INCREMENT )"
For Inserting;
cols = list(structured_files.keys())
vals = list(structured_files.values())
Q1 = f"INSERT INTO data_11 {cols} values {vals}"
mycursor.execute(Q1)
It gives the error : mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '['158795_1635509614542_20211029151335135_27.1943_38.4319.jpg', '2773170_16352...' at line 1
Why is that ?
Okay, I tested this code to do what you want:
Q1 = "INSERT INTO data_11 (id_key, name, date, sample, coord_y, coord_x) VALUES (%s, %s, %s, %s, %s, %s)"
for id_key, values in structured_files.items():
cursor.execute(Q1, [id_key] + values)
cnx.commit()
Tips:
Write the column names explicitly in the INSERT statement. Learn the syntax for INSERT:
INSERT INTO <table> (<columns>) VALUES (<values>)
Your id_key, the key in the dict, isn't a column name. It's one of the values, so it belongs in the VALUES clause.
This is the syntax for inserting one row. Once you get used to this, you can explore multi-row INSERT syntax.
Don't use f-strings to interpolate variables into the INSERT string, because that risks SQL injection flaws. Leave the values as placeholders as I showed, then pass the values separately as a list in the second argument to cursor.execute().
The problem now is that I can only enter one record. No errors are recorded. It just takes the first record from one database and puts in the other database. I am trying to create a machine usable database from the user interface database. I will try to transfer around 100 records once it is working. I would appreciate in comments or suggestions. Thank you!
import sqlite3
sql = 'INSERT INTO heavenStream (scene, cascade, enclosure, sensor, streamer, dither) VALUES (?, ?, ?, ?, ?, ?)'
def dropTable(crs,conn):
crs.execute("DROP TABLE IF EXISTS heavenStream")
def createTable(crs,conn):
sql ='''CREATE TABLE heavenStream(
id INTEGER PRIMARY KEY AUTOINCREMENT,
scene TEXT,
cascade TEXT,
enclosure TEXT,
sensor TEXT,
streamer TEXT,
dither TEXT,
timeStream TEXT,
streamTime TEXT
)'''
crs.execute(sql)
print("Table created successfully........")
def insert_one(conn, crs):
crs.execute("SELECT * FROM animalStream")
for row in crs:
scene = row[1]
cascade = row[2]
enclosure = row[3]
sensor = row[4]
streamer = row[5]
dither = row[6]
print(f"{row[1]} {row[2]} {row[3]} {row[4]} {row[5]} {row[6]}")
try:
crs.execute(sql, (scene, cascade, enclosure,
sensor,streamer,dither))
except sqlite3.IntegrityError as err:
print('sqlite error: ', err.args[0]) # column name is
not unique
conn.commit()
def main():
conn = sqlite3.connect("/home/harry/interface/wildlife.db")
crs = conn.cursor()
dropTable(crs,conn)
createTable(crs,conn)
insert_one(conn, crs)
# conn.commit()
conn.close()
print('done')
main()
The user interface database has had records deleted. There is one record with an id of 64 and the rest are in the 90's.
The cursor (crs) changes here
crs.execute(sql, (scene, cascade, enclosure,sensor,streamer,dither))
after the first insert. Therefore, there are "no more rows to fetch" in the orginal crs.
One solution would be to instantiate another cursor for the insert. Another solution would be to fetchall() the rows into a variable and iterate over that variable as with:
rows = crs.execute("SELECT * FROM animalStream").fetchall()
for row in rows:
I'm trying to create a code piece that inserts an object I've created to store data in a very specific way into an SQL table as a blob type, and it keeps giving me an ' sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.' error.
Has any of you encountered something similar before? Do you have any ideas how to deal with it?
conn = sqlite3.connect('my_database.db')
c = conn.cursor()
params = (self.question_id, i) #i is the object in question
c.execute('''
INSERT INTO '''+self.current_test_name+''' VALUES (?, ?)
''',params)
conn.commit()
conn.close()
For starters, this would be a more appropriate execute statement as it is way cleaner:
c.execute("INSERT INTO "+self.current_test_name+" VALUES (?, ?)", (self.question_id, i))
You are also missing the table you are inserting into (or the columns if self.current_test_name is the table name.)
Also, Is the column in the database setup to handle the data type for the provided input for self.question_id and i? (Not expecting TEXT when you provided INT?)
Example of a working script to insert into a table that has 2 columns named test and test2:
import sqlite3
conn = sqlite3.connect('my_database.db')
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS test(test INT, test2 INT)")
conn.commit()
for i in range(10):
params = (i, i) # i is the object in question
c.execute("INSERT INTO test (test, test2) VALUES (?, ?)", params)
conn.commit()
conn.close()
I made new column with NULL values called 'id' in table. Now I want to add data to it from list. It holds about 130k elements.
I tried with insert, it returned error:
conn = create_connection(xml_db)
cursor = conn.cursor()
with conn:
cursor.execute("ALTER TABLE xml_table ADD COLUMN id integer")
for data in ssetId:
cursor.execute("INSERT INTO xml_table(id) VALUES (?)", (data,))
conn.commit()
I also tried with update:
conn = create_connection(xml_db)
cursor = conn.cursor()
with conn:
cursor.execute("ALTER TABLE xml_table ADD COLUMN id INTEGER")
for data in ssetId:
cursor.execute("UPDATE xml_table SET ('id' = ?)", (data,))
conn.commit()
What is incorrect here ?
EDIT for clarification.
The table was already existing, filled with data. I want to add column 'id' with custom values to it.
Heres an example similar to yours which may be useful.
import sqlite3
conn = sqlite3.connect("xml.db")
cursor = conn.cursor()
with conn:
# for testing purposes, remove this or else the table gets dropped whenever the file is loaded
cursor.execute("drop table if exists xml_table")
# create table with some other field
cursor.execute("create table if not exists xml_table (other_field integer not null)")
for other_data in range(5):
cursor.execute("INSERT INTO xml_table (other_field) VALUES (?)", (other_data,))
# add id field
cursor.execute("ALTER TABLE xml_table ADD COLUMN id integer")
# make sure the table exists
res = cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
print("Table Name: {}".format(res.fetchone()[0]))
# add data to the table
for data in range(5):
cursor.execute("UPDATE xml_table SET id = ? WHERE other_field = ?", (data, data))
# if you must insert an id, you must specify a other_field value as well, since other_field must be not null
cursor.execute("insert into xml_table (id, other_field) VALUES (? ,?)", (100, 105))
# make sure data exists
res = cursor.execute("SELECT id, other_field FROM xml_table")
for id_result in res:
print(id_result)
conn.commit()
conn.close()
As I stated in the comment below, since one of your rows has a NOT NULL constraint on it, no rows can exist in the table that have that column NULL. In the example above other_field is specified NOT NULL, therefore there can be no rows that have NULL values in the column other_field. Any deviation from this would be an IntegrityError.
Output:
Table Name: xml_table
(0, 0)
(1, 1)
(2, 2)
(3, 3)
(4, 4)
(100, 105)
I have been having some trouble with my understanding of how foreign keys work in sqlite3.
Im trying to get the userid (james) in one table userstuff to appear as foreign key in my otherstuff table. Yet when I query it returns None.
So far I have tried:
Enabling foreign key support
Rewriting a test script (that is being discussed here) to isolate issue
I have re-written some code after finding issues in how I had initially written it
After some research I have come across joins but I do not think this is the solution as my current query is an alternative to joins as far as I am aware
Code
import sqlite3 as sq
class DATAB:
def __init__(self):
self.conn = sq.connect("Atest.db")
self.conn.execute("pragma foreign_keys")
self.c = self.conn.cursor()
self.createtable()
self.defaultdata()
self.show_details() # NOTE DEFAULT DATA ALREADY RAN
def createtable(self):
self.c.execute("CREATE TABLE IF NOT EXISTS userstuff("
"userid TEXT NOT NULL PRIMARY KEY,"
" password TEXT)")
self.c.execute("CREATE TABLE IF NOT EXISTS otherstuff("
"anotherid TEXT NOT NULL PRIMARY KEY,"
"password TEXT,"
"user_id TEXT REFERENCES userstuff(userid))")
def defaultdata(self):
self.c.execute("INSERT INTO userstuff (userid, password) VALUES (?, ?)", ('james', 'password'))
self.c.execute("INSERT INTO otherstuff (anotherid, password, user_id) VALUES (?, ?, ?)",('aname', 'password', 'james'))
self.conn.commit()
def show_details(self):
self.c.execute("SELECT user_id FROM otherstuff, userstuff WHERE userstuff.userid=james AND userstuff.userid=otherstuff.user_id")
print(self.c.fetchall())
self.conn.commit()
-----NOTE CODE BELOW THIS IS FROM NEW FILE---------
import test2 as ts
x = ts.DATAB()
Many thanks
A foreign key constraint is just that, a constraint.
This means that it prevents you from inserting data that would violate the constraint; in this case, it would prevent you from inserting a non-NULL user_id value that does not exist in the parent table.
By default, foreign key constraints allow NULL values. If you want to prevent userstuff rows without a parent row, add a NOT NULL constraint to the user_id column.
In any case, a constraint does not magically generate data (and the database cannot know which ID you want). If you want to reference a specific row of the parent table, you have to insert its ID.