Having a dict of structured_files variable as below;
structured_files = {'158795_1635509614542_20211029151335135_27.1943_38.4319.jpg': ['158795', '1635509614542', '20211029151335135', '27.1943', '38.4319'], '2773170_1635255851988_20211026164412576_27.1836_38.4777.jpg': ['2773170', '1635255851988', '20211026164412576', '27.1836', '38.4777'], '2777171_1635330525915_20211027132846547_27.1875_38.4720.jpg': ['2777171', '1635330525915', '20211027132846547', '27.1875', '38.4720'], '2779392_1635150026780_20211025112027410_27.1627_38.4724.jpg': ['2779392', '1635150026780', '20211025112027410', '27.1627', '38.4724'], '3724797_1634893471399_20211022120432057_27.1334_38.5426.jpg': ['3724797', '1634893471399', '20211022120432057', '27.1334', '38.5426'], '5218186_1635161250087_20211025142730572_27.0694_38.4943.jpg': ['5218186', '1635161250087', '20211025142730572', '27.0694', '38.4943'], '5347212_1634987405913_20211023141006566_27.0454_38.5138.jpg': ['5347212', '1634987405913', '20211023141006566', '27.0454', '38.5138'], '8174846_1635335281021_20211027144801672_27.1039_38.3572.jpg': ['8174846', '1635335281021', '20211027144801672', '27.1039', '38.3572'], '9179383_1635321573332_20211027105933968_27.1251_38.3576.jpg': ['9179383', '1635321573332', '20211027105933968', '27.1251', '38.3576']}
Trying to insert in to mysql db code is follows;
For Creation;
Qcreate1 = "CREATE TABLE data_11 (id_key LONGTEXT NOT NULL, name VARCHAR(50) NOT NULL, date VARCHAR(50) NOT NULL, sample VARCHAR(50) NOT NULL, coord_y VARCHAR(50) NOT NULL, coord_x VARCHAR(50) NOT NULL, created datetime NOT NULL, key_id int PRIMARY KEY NOT NULL AUTO_INCREMENT )"
For Inserting;
cols = list(structured_files.keys())
vals = list(structured_files.values())
Q1 = f"INSERT INTO data_11 {cols} values {vals}"
mycursor.execute(Q1)
It gives the error : mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '['158795_1635509614542_20211029151335135_27.1943_38.4319.jpg', '2773170_16352...' at line 1
Why is that ?
Okay, I tested this code to do what you want:
Q1 = "INSERT INTO data_11 (id_key, name, date, sample, coord_y, coord_x) VALUES (%s, %s, %s, %s, %s, %s)"
for id_key, values in structured_files.items():
cursor.execute(Q1, [id_key] + values)
cnx.commit()
Tips:
Write the column names explicitly in the INSERT statement. Learn the syntax for INSERT:
INSERT INTO <table> (<columns>) VALUES (<values>)
Your id_key, the key in the dict, isn't a column name. It's one of the values, so it belongs in the VALUES clause.
This is the syntax for inserting one row. Once you get used to this, you can explore multi-row INSERT syntax.
Don't use f-strings to interpolate variables into the INSERT string, because that risks SQL injection flaws. Leave the values as placeholders as I showed, then pass the values separately as a list in the second argument to cursor.execute().
Related
I'm using pg package in node.js and have created my table schema:
CREATE TABLE clients_steps (
id SERIAL PRIMARY KEY,
client_id INTEGER NOT NULL
REFERENCES clients ON DELETE CASCADE,
step_id INTEGER NOT NULL
REFERENCES steps ON DELETE CASCADE,
update_date DATE NOT NULL DEFAULT CURRENT_DATE
);
Now I'm trying to seed my database. But it kept throwing errors at me.
My seed file:
### this works fine ###
INSERT INTO clients_steps (client_id, step_id, update_date)
VALUES (1, 2, '2023-02-02');
### this is not working ###
INSERT INTO clients_steps (client_id, step_id, update_date)
VALUES (1, 1);
And the error is:
ERROR: INSERT has more target columns than expressions
LINE 1: INSERT INTO clients_steps (client_id,step_id,update_date)
I think the reason is it can't insert the default value into the column.
So how should I set up the default value for a DATE datatype? I don't need the time stamp, only the date will be required.
Thank you so much!
You have two options to turn your second insert into valid SQL:
omit the column for which you want the default value from the list:
INSERT INTO clients_steps (client_id, step_id)
VALUES (1, 1);
use the special value DEFAULT:
INSERT INTO clients_steps (client_id, step_id, update_date)
VALUES (1, 1, DEFAULT);
I want to exclude the strings in the address column which start with the letters C, G or S.
I have tried this expression,
SELECT * FROM guest_list
WHERE address NOT LIKE ("C%" OR "F%" OR "G%");
and it gives me the warnings like this:
'Warning: truncated incorrect DOUBLE value: 'C%'
'Warning: truncated incorrect DOUBLE value: 'G%'
here is the script to create my table.
CREATE TABLE `guest_list` (
`id` int(11) NOT NULL,
`first_name` varchar(50) DEFAULT NULL,
`last_name` varchar(50) DEFAULT NULL,
`address` varchar(50) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
INSERT INTO `guest_list` VALUES (1001,'Jim','Dowd','Lewisham West and Penge'),
(1002,'Lyn','Brown','West Ham'),(1003,'Ann','Clwyd','Cynon Valley'),
(1004,'Nic','Dakin','Scunthorpe'),(1005,'Pat','Glass','North West Durham'),
(1006,'Kate','Hoey','Vauxhall'),(1007,'Mike','Kane','Wythenshawe and Sale East'),
(1008,'John','Mann','Bassetlaw'),(1009,'Joan','Ryan','Enfield North'),
(1010,'Cat','Smith','Lancaster and Fleetwood'),(1011,'Mark','Tami','Alyn and Deeside'),
(1012,'Keith','Vaz','Leicester East'),(1013,'Ian','Austin','Dudley North'),
(1014,'Liam','Byrne','Birmingham, Hodge Hill'),(1015,'Ann','Coffey','Stockport');
What is the correct way to do this with LIKE please (if one does not use regex)?
LIKE is standard operator for comparing a column value to another column value, or to a quoted string.
"C%" OR "F%" OR "G%" is not a quoted string, but a mathematical operation with the result 0, since all values are string (but not numerical) values and converted to zero.
Instead of using one LIKE operator, you have to use n LIKE operators, where n is the number of your conditions:
SELECT columns FROM guest_list
WHERE address NOT LIKE ("C%") AND address NOT LIKE("F%") AND address NOT LIKE("G%")
By default, LIKE conditions are case-insensitive, so if you want to filter only uppercase, you have to specify a binary collation, e.g. LIKE ("C%" collate latin1_bin)..
I have an empty table defined in snowflake as;
CREATE OR REPLACE TABLE db1.schema1.table(
ACCOUNT_ID NUMBER NOT NULL PRIMARY KEY,
PREDICTED_PROBABILITY FLOAT,
TIME_PREDICTED TIMESTAMP
);
And it creates the correct table, which has been checked using desc command in sql. Then using a snowflake python connector we are trying to execute following query;
insert_query = f'INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) VALUES ({accountId}, {risk_score},{ct});'
ctx.cursor().execute(insert_query)
Just before this query the variables are defined, The main challenge is getting the current time stamp written into snowflake. Here the value of ct is defined as;
import datetime
ct = datetime.datetime.now()
print(ct)
2021-04-30 21:54:41.676406
But when we try to execute this INSERT query we get the following errr message;
ProgrammingError: 001003 (42000): SQL compilation error:
syntax error line 1 at position 157 unexpected '21'.
Can I kindly get some help on ow to format the date time value here? Help is appreciated.
In addition to the answer #Lukasz provided you could also think about defining the current_timestamp() as default for the TIME_PREDICTED column:
CREATE OR REPLACE TABLE db1.schema1.table(
ACCOUNT_ID NUMBER NOT NULL PRIMARY KEY,
PREDICTED_PROBABILITY FLOAT,
TIME_PREDICTED TIMESTAMP DEFAULT current_timestamp
);
And then just insert ACCOUNT_ID and PREDICTED_PROBABILITY:
insert_query = f'INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY) VALUES ({accountId}, {risk_score});'
ctx.cursor().execute(insert_query)
It will automatically assign the insert time to TIME_PREDICTED
Educated guess. When performing insert with:
insert_query = f'INSERT INTO ...(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED)
VALUES ({accountId}, {risk_score},{ct});'
It is a string interpolation. The ct is provided as string representation of datetime, which does not match a timestamp data type, thus error.
I would suggest using proper variable binding instead:
ctx.cursor().execute("INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES "
"(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) "
"VALUES(:1, :2, :3)",
(accountId,
risk_score,
("TIMESTAMP_LTZ", ct)
)
);
Avoid SQL Injection Attacks
Avoid binding data using Python’s formatting function because you risk SQL injection. For example:
# Binding data (UNSAFE EXAMPLE)
con.cursor().execute(
"INSERT INTO testtable(col1, col2) "
"VALUES({col1}, '{col2}')".format(
col1=789,
col2='test string3')
)
Instead, store the values in variables, check those values (for example, by looking for suspicious semicolons inside strings), and then bind the parameters using qmark or numeric binding style.
You forgot to place the quotes before and after the {ct}. The code should be :
insert_query = "INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) VALUES ({accountId}, {risk_score},'{ct}');".format(accountId=accountId,risk_score=risk_score,ct=ct)
ctx.cursor().execute(insert_query)
I have the following function defined to insert several rows with iteration in Python using Psycopg2 and PostgreSQL 11.
When I receive the same obj (with same id), I want to update its date.
def insert_execute_values_iterator(
connection,
objs: Iterator[Dict[str, Any]],
page_size: int = 1000,
) -> None:
with connection.cursor() as cursor:
try:
psycopg2.extras.execute_values(cursor, """
INSERT INTO objs(\
id,\
date,\
) VALUES %s \
ON CONFLICT (id) \
DO UPDATE SET (date) = (EXCLUDED.date) \
""", ((
obj['id'],
obj['date'],
) for obj in objs), page_size=page_size)
except (Exception, Error) as error:
print("Error while inserting as in database", error)
When a conflict happens on the unique primary key of the table while inserting an element, I get the error:
Error while inserting as in database ON CONFLICT DO UPDATE command
cannot affect row a second time
HINT: Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
FYI, the clause works on PostgreSQL directly but not from the Python code.
Use unique VALUE-combinations in your INSERT statement:
create table foo(id int primary key, date date);
This should work:
INSERT INTO foo(id, date)
VALUES(1,'2021-02-17')
ON CONFLICT(id)
DO UPDATE SET date = excluded.date;
This one won't:
INSERT INTO foo(id, date)
VALUES(1,'2021-02-17') , (1, '2021-02-16') -- 2 conflicting rows
ON CONFLICT(id)
DO UPDATE SET date = excluded.date;
DEMO
You can fix this by using DISTINCT ON() in a SELECT statement:
INSERT INTO foo(id, date)
SELECT DISTINCT ON(id) id, date
FROM (VALUES(1,CAST('2021-02-17' AS date)) , (1, '2021-02-16')) s(id, date)
ORDER BY id, date ASC
ON CONFLICT(id)
DO UPDATE SET date = excluded.date;
I am attempting to convert my varchar column data, which is stringifed JSON, to MAP datatype so I can reference the data as elements.
WITH
data(c) AS (
SELECT message from mydb.mytable
),
parsed AS (
SELECT cast(json_parse(c) as map(varchar, varchar)) AS m
FROM data
)
SELECT m['action'], m['uuid']
FROM parsed
Sample data looks like:
{"action":"send","timestamp":1566432054,"uuid":"1234"}
I tried solution provided here: How to cast varchar to MAP(VARCHAR,VARCHAR) in presto, which is where I got the query from replacing values with a select statement but it did not work. I get error:
INVALID_CAST_ARGUMENT: Value cannot be cast to map(varchar,varchar)
json_parse + cast work on your example data:
SELECT CAST(json_parse(str) AS map(varchar, varchar))
FROM (VALUES '{"action":"send","timestamp":1566432054,"uuid":"1234"}') t(str);
I tested this on Presto 317:
presto> SELECT CAST(json_parse(str) AS map(varchar, varchar))
-> FROM (VALUES '{"action":"send","timestamp":1566432054,"uuid":"1234"}') t(str);
_col0
------------------------------------------------
{action=send, uuid=1234, timestamp=1566432054}
(1 row)
My guess is that some data row is different than your example and this data row cannot be cast. You can find it with try:
SELECT str
FROM your_table
WHERE str IS NOT NULL
AND try(CAST(json_parse(str) AS map(varchar, varchar))) IS NULL;