I wish to execute the statement to delete records from a postgresql table older than 45 days in my python script as below:
Consider only the code below:
import psycopg2
from datetime import datetime
cur = conn.cursor()
mpath = None
sql1 = cur.execute(
"Delete from table1 where mdatetime < datetime.today() - interval '45 days'")
This causes the following error:
psycopg2.errors.InvalidSchemaName: schema "datetime" does not exist
LINE 1: Delete from logsearch_maillogs2 where mdatetime <
datetime.t...
How do I exactly change the format or resolve this. Do I need to convert. Saw a few posts which say that postgresql DateTime doesn't exist in PostgreSQL etc, but didn't find exact code to resolve this issue. Please guide.
The query is running in Postgres not Python you need to use SQL timestamp function not Python ones if you are writing a hard coded string. So datetime.today() --> now() per Current Date/time.
sql1 = cur.execute(
"Delete from table1 where mdatetime < now() - interval '45 days'")
Or you need to use parameters per here Parameters to pass in a Python datetime if you want a dynamic query.
sql1 = cur.execute(
"Delete from table1 where mdatetime < %s - interval '45 days'", [datetime.today()])
I have a Postgres table with a jsonb column containing UTC timestamp data in ISO format like the following:
{
"time": "2021-04-13T20:14:56Z"
}
The Django model for this table looks like:
class DateModel(models.Model):
values = models.JSONField(default=dict)
I need to query the table for all records with a timestamp on a certain date (ignoring time)
I'm looking for a solution similar to the following:
DateModel.objects.filter(values__time__date='2021-04-13')
The other solution I have found is to query for records with date greater than the previous day and less than the next one. This works but I am looking for a way to do it with a single query so the code would be more concise.
Any suggestions?
There's a couple of annotations you need to perform on the queryset to extract the time field and convert it to a datetime.
First you need to extract the time string by using django.contrib.postgres.fields.jsonb.KeyTextTransform
from django.contrib.postgres.fields.jsonb import KeyTextTransform
query = DateModel.objects.annotate(time_str=KeyTextTransform('time', 'values'))
Then you need to convert that string to a datetime using Cast
from django.db.models.functions import Cast
from django.db.models import DateTimeField
query = query.annotate(time=Cast('time_str', output_field=DateTimeField()))
Then you can filter by that annotation
query = query.filter(time__date='2021-04-13')
I have an empty table defined in snowflake as;
CREATE OR REPLACE TABLE db1.schema1.table(
ACCOUNT_ID NUMBER NOT NULL PRIMARY KEY,
PREDICTED_PROBABILITY FLOAT,
TIME_PREDICTED TIMESTAMP
);
And it creates the correct table, which has been checked using desc command in sql. Then using a snowflake python connector we are trying to execute following query;
insert_query = f'INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) VALUES ({accountId}, {risk_score},{ct});'
ctx.cursor().execute(insert_query)
Just before this query the variables are defined, The main challenge is getting the current time stamp written into snowflake. Here the value of ct is defined as;
import datetime
ct = datetime.datetime.now()
print(ct)
2021-04-30 21:54:41.676406
But when we try to execute this INSERT query we get the following errr message;
ProgrammingError: 001003 (42000): SQL compilation error:
syntax error line 1 at position 157 unexpected '21'.
Can I kindly get some help on ow to format the date time value here? Help is appreciated.
In addition to the answer #Lukasz provided you could also think about defining the current_timestamp() as default for the TIME_PREDICTED column:
CREATE OR REPLACE TABLE db1.schema1.table(
ACCOUNT_ID NUMBER NOT NULL PRIMARY KEY,
PREDICTED_PROBABILITY FLOAT,
TIME_PREDICTED TIMESTAMP DEFAULT current_timestamp
);
And then just insert ACCOUNT_ID and PREDICTED_PROBABILITY:
insert_query = f'INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY) VALUES ({accountId}, {risk_score});'
ctx.cursor().execute(insert_query)
It will automatically assign the insert time to TIME_PREDICTED
Educated guess. When performing insert with:
insert_query = f'INSERT INTO ...(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED)
VALUES ({accountId}, {risk_score},{ct});'
It is a string interpolation. The ct is provided as string representation of datetime, which does not match a timestamp data type, thus error.
I would suggest using proper variable binding instead:
ctx.cursor().execute("INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES "
"(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) "
"VALUES(:1, :2, :3)",
(accountId,
risk_score,
("TIMESTAMP_LTZ", ct)
)
);
Avoid SQL Injection Attacks
Avoid binding data using Python’s formatting function because you risk SQL injection. For example:
# Binding data (UNSAFE EXAMPLE)
con.cursor().execute(
"INSERT INTO testtable(col1, col2) "
"VALUES({col1}, '{col2}')".format(
col1=789,
col2='test string3')
)
Instead, store the values in variables, check those values (for example, by looking for suspicious semicolons inside strings), and then bind the parameters using qmark or numeric binding style.
You forgot to place the quotes before and after the {ct}. The code should be :
insert_query = "INSERT INTO DATA_LAKE.CUSTOMER.ACT_PREDICTED_PROBABILITIES(ACCOUNT_ID, PREDICTED_PROBABILITY, TIME_PREDICTED) VALUES ({accountId}, {risk_score},'{ct}');".format(accountId=accountId,risk_score=risk_score,ct=ct)
ctx.cursor().execute(insert_query)
I am trying to compare a string variable in my Python script with a varchar variable in SQL. Following is a snippet from my code:
todaysDate = datetime.now().date().strftime('%d-%m-%y')
read(conn, f'select cou_id from NewCourier where date_created = {todaysDate}'
Where:
todaysDate => variable in python storing current date
cou_id & date_created => columns in a relational DB table NewCourier
I tried:
read(conn, f'select cou_id from NewCourier where CAST(date_created AS INT) = {todaysDate}
I've also gone through a couple of questions and solutions on StackOverflow and other sites and found only this solution similar to my problem but I didn't get a clear idea of how to solve it.
Thanks!
Python 3.8.5
Microsoft SQL Server Management Studio 18
It seems like the column data type is integer and you are trying to compare it to a string. If you can find code in your program that inserts into the table you should be able to figure out how to convert your date time properly
Instead of creating a variable to store the current date in Python, I used the GETDATE() method of SQL and converted it to varchar using the CONVERT() method in SQL which solved my problem
(i.e., it allowed me to get the courier IDs of all the records generated on that particular day).
Here's the updated line:
read(conn, f'select cou_id from NewCourier where date_created = CONVERT(varchar, getdate(), 23)
I am taking a dataframe and inserting it into a Postgresql table.
One column in the dataframe is a datetime64 dtype. The column type in PostgreSQL is 'timestamp without time zone.' To prepare the dataframe to insert, I am using to_records:
listdf = df.to_records(index=False).tolist()
When I run the to_records, it gives an error at the psycopg2's cur.executemany() that I am trying to insert Biginit into a Timestamp without timezone.
So I tried to add a dict of column_dtypes to the to_records. But that doesn't work. The below gives the error: "ValueError: Cannot convert from specific units to generic units in NumPy datetimes or timedeltas"
DictofDTypes = dict.fromkeys(SQLdfColHAedings,'float')
DictofDTypes['Date_Time'] = 'datetime64'
listdf = df.to_records(index=False,column_dtypes=DictofDTypes).tolist()
I have also tried type of str, int, and float. None worked in the above three lines.
How do I convert the column properly to be able to insert the column into a timestamp sql column?
I removed defining the dtypes from to_records.
And before to_recordes, I converted the datetime to str with:
df['Date_Time'] = df['Date_Time'].apply(lambda x: x.strftime('%Y-%m-%d %H:%M:%S'))
The sql insert command then worked.