I wish to execute the statement to delete records from a postgresql table older than 45 days in my python script as below:
Consider only the code below:
import psycopg2
from datetime import datetime
cur = conn.cursor()
mpath = None
sql1 = cur.execute(
"Delete from table1 where mdatetime < datetime.today() - interval '45 days'")
This causes the following error:
psycopg2.errors.InvalidSchemaName: schema "datetime" does not exist
LINE 1: Delete from logsearch_maillogs2 where mdatetime <
datetime.t...
How do I exactly change the format or resolve this. Do I need to convert. Saw a few posts which say that postgresql DateTime doesn't exist in PostgreSQL etc, but didn't find exact code to resolve this issue. Please guide.
The query is running in Postgres not Python you need to use SQL timestamp function not Python ones if you are writing a hard coded string. So datetime.today() --> now() per Current Date/time.
sql1 = cur.execute(
"Delete from table1 where mdatetime < now() - interval '45 days'")
Or you need to use parameters per here Parameters to pass in a Python datetime if you want a dynamic query.
sql1 = cur.execute(
"Delete from table1 where mdatetime < %s - interval '45 days'", [datetime.today()])
Related
Hi all I'm new to Dask.
I faced an error when I tried using read_sql_query to get data from Oracle database.
Here is my python script:
con_str = "oracle+cx_oracle://{UserID}:{Password}#{Domain}/?service_name={Servicename}"
sql= "
column_a, column_b
from
database.tablename
where
mydatetime >= to_date('1997-01-01 00:00:00','YYYY-MM-DD HH24:MI:SS')
"
from sqlalchemy.sql import select, text
from dask.dataframe import read_sql_query
sa_query= select(text(sql))
ddf = read_sql_query(sql=sa_query, con=con, index_col="index", head_rows=5)
I refered this post: Reading an SQL query into a Dask DataFrame
Remove "select" string from my query.
And I got an cx_Oracle.DatabaseError with missing expression [SQL: SELECT FROM DUAL WHERE ROWNUM <= 5]
But I don't get it where the query came from.
Seem like it didn't execute the sql code I provided.
I'm not sure which part I did not config right.
*Note: using pandas.read_sql is ok , only fail when using dask.dataframe.read_sql_query
I have the following SQL statement where i am reading the database to get the records for 1 day. Here is what i tried in pgAdmin console -
SELECT * FROM public.orders WHERE createdat >= now()::date AND type='t_order'
I want to convert this to the syntax of psycopg2but somehow it throws me errors -
Database connection failed due to invalid input syntax for type timestamp: "now()::date"
Here is what i am doing -
query = f"SELECT * FROM {table} WHERE (createdat>=%s AND type=%s)"
cur.execute(query, ("now()::date", "t_order"))
records = cur.fetchall()
Any help is deeply appreciated.
DO NOT use f strings. Use proper Parameter Passing
now()::date is better expressed as current_date. See Current Date/Time.
You want:
query = "SELECT * FROM public.orders WHERE (createdat>=current_date AND type=%s)"
cur.execute(query, ["t_order"])
If you want dynamic identifiers, table/column names then:
from psycopg2 import sql
query = sql.SQL("SELECT * FROM {} WHERE (createdat>=current_date AND type=%s)").format(sql.Identifier(table))
cur.execute(query, ["t_order"])
For more information see sql.
I want to create a for loop that queries data from my database one week at a time for three months.
For example:
import tempfile
import pandas as pd
DB_URL = 'db/url/here:1234'
engine = sqlalchemy.create_engine(DB_URL)
conn = engine.connect()
queries=
{'d_week1_start':2017-12-1,
'd_week1_end':2017-12-7,
'd_week2_start':2017-12-7,
'd_week2_end':2017-12-14,
'd_week3_start': 2017-12-14,
'd_week3_end':2017-12-21,
'd_week4_start': 2017-12-21,
'd_week4_end':2017-12-31}
for qr in queries:
one_week_df = pd.read_sql(qr, conn)
one_week_df.to_pickle('one_week.pkl')
How would I go about doing something like this?
Update: I realized I wasn't passing any queries. I changed the items in the dictionary to queries like:
'''SELECT * FROM table WHERE time > '20180201T120000'' AND time < '20180207T120000' ORDER BY time ASC'''
First question on here, so let me know if more information is needed. I am using the Python psycopg2-binary==2.7.7 package in an attempt to pull PostgreSQL 9.6.11 timestamptz fields out of a database.
With that said, the 'psycopg2' package seems to be coercing the timestamptz date-times to a different timezone than is present in the database.
For instance, the following query will return the correct offset if run in a PostgreSQL client:
SQL
SELECT row_to_json(t)
FROM (
SELECT '2019-01-24T08:24:00-05:00'::timestamptz AS tz
)t;
Result
{"tz":"2019-01-24 08:24:00-05"}
However, if I run the same query via the psycopg2.cursor.fetchall method, I get a different offset than expected/returned:
import time
import psycopg2
import logging
logger = logging.getLogger()
def getRows(query, printRows=False, **kwargs):
try:
cs = "dbname={dbname} user={dbuser} password={dbpass} host={server} port={port}".format(
**kwargs)
con = psycopg2.connect(cs)
con.set_session(readonly=True, autocommit=True)
except Exception:
logger.exception("-->>>>Something went wrong connecting to db")
return None
end = None
try:
start = time.time()
cur = con.cursor()
cur.execute(query)
rows = cur.fetchall()
if printRows:
for i in rows:
print(i)
cur.close()
con.commit()
con.close()
end = time.time()
logger.info(
"-->>>>Query took {} seconds...".format(round(end - start, 2)))
return rows
except Exception:
end = time.time()
cur.close()
con.commit()
con.close()
logger.exception("-->>>>Something went wrong with the query...")
logger.info(
"-->>>>Query took {} seconds...".format(round(end - start, 2)))
if __name__ == '__main__':
test = getRows("""SELECT row_to_json(t) AS "result"
FROM(
SELECT '2019-01-24T08:24:00-05:00'::timestamptz AS tz
)t;
""", printRows=True, **DBSECRETS)
print(test[0][0])
Result
{'tz': '2019-01-24T05:24:00-08:00'}
As seen above, the EST timezone (offset of -5)to PostgreSQL is being converted to a -08:00 offset via the psycopg2 package.
I've checked the psycopg2 documentation but could not find any conclusive examples to fix this issue. Specifically, I've checked here:
http://initd.org/psycopg/docs/cursor.html#cursor.tzinfo_factory
It turns out that the SQL Client, Dbeaver, coerces a timestamptz to the local OS timezone, which in this case is EST.
How to change DBeaver timezone / How to stop DBeaver from converting date and time
The PostgreSQL server, however, has a native timezone of Pacific time or PST. Thus, the psycopg2 package was interpreting the timestamptz correctly according to the server, i.e. PST.
I am trying to use cx_Oracle to query a table in oracle DB (version 11.2) and get rows with values in a column between a datetime range.
I have tried the following approaches:
Tried between clause as described here, but cursor gets 0 rows
parameters = (startDateTime, endDateTime)
query = "select * from employee where joining_date between :1 and :2"
cur = con.cursor()
cur.execute(query, parameters)
Tried the TO_DATE() function and Date'' qualifiers. Still no result for Between or >= operator. Noteworthy is that < operator works. I also got the same query and tried in a sql client, and the query returns results. Code:
#returns no rows:
query = "select * from employee where joining_date >= TO_DATE('" + startDateTime.strftime("%Y-%m-%d") + "','yyyy-mm-dd')"
cur = con.cursor()
cur.execute(query)
#tried following just to ensure that some query runs fine, it returns results:
query = query.replace(">=", "<")
cur.execute(query)
Any pointers about why the between and >= operators are failing for me? (my second approach was in line with the answer in Oracle date comparison in where clause but still doesn't work for me)
I am using python 3.4.3 and used cx_Oracle 5.3 and 5.2 with oracle client 11g on windows 7 machine
Assume that your employee table contains the field emp_id and the row with emp_id=1234567 should be retrieved by your query.
Make two copies of your a program that execute the following queries
query = "select to_char(:1,'YYYY-MM-DD HH24:MI:SS')||' >= '||to_char(joining_date,'YYYY-MM-DD HH24:MI:SS')||' >= '||to_char(:2,'YYYY-MM-DD HH24:MI:SS') resultstring from employee where emp_id=1234567"
and
query="select to_char(joining_date,'YYYY-MM-DD HH24:MI:SS')||' >= '||to_char(TO_DATE('" + startDateTime.strftime("%Y-%m-%d") + "','yyyy-mm-dd'),'YYYY-MM-DD HH24:MI:SS') resultstring from employee where emp_id=1234567"
Show us the code and the value of the column resultstring
You are constructing SQL queries as strings when you should be using parameterized queries. You can't use parameterization to substitute the comparison operators, but you should use it for the dates.
Also, note that the referenced answer uses the PostgreSQL parameterisation format, whereas Oracle requires you to use the ":name" format.