Using parameters in query to search between dates - python-3.x

Silly newbe here.
So I'm banging my head on this:
Can't quite figure out the parameterized query and if it's properly formatted.
import sqlite3
def readSqliteTable():
try:
sqliteConnection = sqlite3.connect('testDB.sqlite')
cursor = sqliteConnection.cursor()
print("Connected to SQLite")
startdate = "2022-11-05"
enddate = "2022-11-25"
print("startdate =", startdate, "enddate =", enddate)
cursor.execute("SELECT * FROM tz WHERE UL_Time BETWEEN '%s' AND '%s'" % (startdate, enddate))
print(cursor.fetchall())
records = cursor.fetchall()
print("Total rows are: ", len(records))
print("Printing each row")
for row in records:
print("Id: ", row[0])
print("Updated: ", row[1])
print("Title: ", row[2])
print("UL_Time: ", row[3])
print("Size: ", row[4])
print("\n")
cursor.close()
except sqlite3.Error as error:
print("Failed to read data from sqlite table", error)
finally:
if sqliteConnection:
sqliteConnection.close()
print("The SQLite connection is closed")
It works fine if I substitute arbitrary dates as:
cursor.execute("SELECT * FROM tz WHERE UL_Time BETWEEN 2022-11-01 AND 2022-11-25")
but won't work in this form

First of all, you don't understand what a parameterized query is. Read the official Python documentation and do tutorials.
Your expression
"SELECT * FROM tz WHERE UL_Time BETWEEN '%s' AND '%s'" % (startdate, enddate))
shows string interpolation and has nothing to do with parameterized queries. This query also implements textual filter. It should work properly, so long all your dates are formatted as YYYY-MM-DD.
The second query is meaningless, as your WHERE clause defines an integer filter:
WHERE UL_Time BETWEEN 2010 AND 1986

Related

Getting Syntax error in parametrize insert query in python postgresql

I am trying to send data to PostgreSQL, data is a tuple of strings i.e (time, price).
The problem is when I send data using a simple query (not parametrized) it works fine!
Following is the simple query working perfectly.
cur.execute("INSERT INTO paxos (date,price) VALUES ('2020-04-09 14:39:58.145804', '$1,664.08');");
But as those values aren't fixed, so I want to store them in variable and use a parameterized query for sending data, but the parameterized query isn't working for me. Here is the parameterized query.
cur.execute("INSERT INTO paxos (date, price) values (?, ?)",(time, price))
Here is the complete function I am trying to implement:
def insert_data(time, price):
con = psycopg2.connect(database="", user="", password="", host="", port="5432")
print("Database opened successfully")
cur = con.cursor()
data_tuple = (time, price)
cur.execute("insert into paxos (date, price) values (?, ?)",(time, price))
con.commit()
print("Record inserted successfully")
con.close()
insert_data("2020-04-09 14:39:58.145804", "$1,664.08")
Here is the error message:
It seems like this is a python syntax error. I think you should use the format method of the string object (see codesnippet below).
I can't test it right now, but according to some old code of mine, I always "built" a query string first and then passed the string object to the cursor. Try something like that:
artist = "Aphex Twin"
title = "Windowlicker"
query = '''SELECT EXISTS(SELECT * FROM tracks
WHERE artist ILIKE \'{}\'AND
title ILIKE \'{}\')'''.format(artist, title)
cursor.execute(query)

Postgres query with '%%' parameter not returning results via psycopg2

When I execute the below query in a query editor like DBeaver - it returns a result but if I execute the same query via Python & psycopg2 it does not return a result. '%%' should match any title/location so there will always return something. I'm just testing this for a category without keywords but it will also take an array of keywords if they exist depending on the category. So the array could be ['%%'] or ['%boston%', '%cambridge%'] and both should work.
select title, link
from internal.jobs
where (title ilike any(array['%%'])
or location ilike any(array['%%']))
order by "publishDate" desc
limit 1;
I've tried adding the E flag at the beginning of the string. E.g. E'%%'
Python:
import psycopg2
FILTERS = {
'AllJobs': [],
'BostonJobs': ['boston', 'cambridge'],
'MachineLearningJobs': ['ml', 'machine learning']
}
conn = psycopg2.connect("dbname=test user=postgres")
cur = conn.cursor()
sql = """
select title, link
from internal.jobs
where (title ilike any(array[%s])
or location ilike any(array[%s]))
order by "publishDate" desc
limit 1;
"""
for title, tags in FILTERS.items():
if not tags:
formatted_filters = "'%%'" # Will match any record
else:
formatted_filters = ','.join([f"'%{keyword}%'" for keyword in tags])
cur.execute(sql, (formatted_filters))
results = cur.fetchone()
print(results)
You can use the cur.mogrify() query to look at the SQL finally generated, check in psql if it works, and how you need to tweak it.
Most likely you have to double every %.
Thanks to Piro for the very useful cur.mogrify() clue. That helped me further debug the query to figure out what was going wrong.
I ended up removing the extra set of quotes, I used a named parameter and now it works as expected.
Updated code:
import psycopg2
FILTERS = {
'AllJobs': [],
'BostonJobs': ['boston', 'cambridge'],
'MachineLearningJobs': ['ml', 'machine learning']
}
conn = psycopg2.connect("dbname=test user=postgres")
cur = conn.cursor()
sql = """
select title, link
from internal.jobs
where (title ilike any(array[%(filter)s])
or location ilike any(array[%(filter)s]))
order by "publishDate" desc
limit 1;
"""
for title, tags in FILTERS.items():
if not tags:
formatted_filters = '%%' # Will match any record
else:
formatted_filters = [f'%{keyword}%' for keyword in tags]
print(cur.mogrify(sql, {'filter': formatted_filters}))
cur.execute(sql, {'filter': formatted_filters})
results = cur.fetchone()
print(results)

psycopg2 insert statement with LIKE clause containing % sign

Can't execute a psycopg2 insert query (Postgres db), which uses the best practice %s sign for inserting and contains a LIKE statement which includes a % sign.
% sign in LIKE statement is interpreted as an insert placeholder.
'IndexError: tuple index out of range' is thrown.
Tried escaping % with backslash, didn't work out.
with psycopg2.connect(some_url) as conn:
with conn.cursor() as cur:
query = """
SELECT id
FROM users
WHERE surname IN %s AND named LIKE '%john'
"""
cur.execute(query, (tuple(["smith", "mcnamara"]),))
data = cur.fetchall()
Try using a placeholder also for the LIKE expression, and then bind a literal with a wildcard to it:
query = """
SELECT id
FROM users
WHERE surname IN %s AND named LIKE %s"""
cur.execute(query, (tuple(["smith", "mcnamara"]), "%John",))
data = cur.fetchall()
try this one:
with psycopg2.connect(some_url) as conn:
with conn.cursor() as cur:
query = """
SELECT id
FROM users
WHERE surname IN %s AND named LIKE '%sjohn'
"""
cur.execute(query, (tuple(["smith", "mcnamara"]), '%'))
data = cur.fetchall()

fetchall method converting Postgresql timestamptz field to different timezone

First question on here, so let me know if more information is needed. I am using the Python psycopg2-binary==2.7.7 package in an attempt to pull PostgreSQL 9.6.11 timestamptz fields out of a database.
With that said, the 'psycopg2' package seems to be coercing the timestamptz date-times to a different timezone than is present in the database.
For instance, the following query will return the correct offset if run in a PostgreSQL client:
SQL
SELECT row_to_json(t)
FROM (
SELECT '2019-01-24T08:24:00-05:00'::timestamptz AS tz
)t;
Result
{"tz":"2019-01-24 08:24:00-05"}
However, if I run the same query via the psycopg2.cursor.fetchall method, I get a different offset than expected/returned:
import time
import psycopg2
import logging
logger = logging.getLogger()
def getRows(query, printRows=False, **kwargs):
try:
cs = "dbname={dbname} user={dbuser} password={dbpass} host={server} port={port}".format(
**kwargs)
con = psycopg2.connect(cs)
con.set_session(readonly=True, autocommit=True)
except Exception:
logger.exception("-->>>>Something went wrong connecting to db")
return None
end = None
try:
start = time.time()
cur = con.cursor()
cur.execute(query)
rows = cur.fetchall()
if printRows:
for i in rows:
print(i)
cur.close()
con.commit()
con.close()
end = time.time()
logger.info(
"-->>>>Query took {} seconds...".format(round(end - start, 2)))
return rows
except Exception:
end = time.time()
cur.close()
con.commit()
con.close()
logger.exception("-->>>>Something went wrong with the query...")
logger.info(
"-->>>>Query took {} seconds...".format(round(end - start, 2)))
if __name__ == '__main__':
test = getRows("""SELECT row_to_json(t) AS "result"
FROM(
SELECT '2019-01-24T08:24:00-05:00'::timestamptz AS tz
)t;
""", printRows=True, **DBSECRETS)
print(test[0][0])
Result
{'tz': '2019-01-24T05:24:00-08:00'}
As seen above, the EST timezone (offset of -5)to PostgreSQL is being converted to a -08:00 offset via the psycopg2 package.
I've checked the psycopg2 documentation but could not find any conclusive examples to fix this issue. Specifically, I've checked here:
http://initd.org/psycopg/docs/cursor.html#cursor.tzinfo_factory
It turns out that the SQL Client, Dbeaver, coerces a timestamptz to the local OS timezone, which in this case is EST.
How to change DBeaver timezone / How to stop DBeaver from converting date and time
The PostgreSQL server, however, has a native timezone of Pacific time or PST. Thus, the psycopg2 package was interpreting the timestamptz correctly according to the server, i.e. PST.

Value Error:unsupported format character 'd' (0x64) at index %Id

I tried to insert integer values in my table, But i was subjected to "Value Error"
import psycopg2
def connect():
con=psycopg2.connect("dbname='book_store' user='postgres' password='5283' host='localhost' port='5432' ")
cur=con.cursor()
cur.execute("CREATE TABLE if not exists books(id SERIAL PRIMARY KEY,title TEXT NOT NULL,author TEXT NOT NULL,year integer NOT NULL,isbn integer NOT NULL)")
con.commit()
con.close()
def insert(title,author,year,isbn):
con=psycopg2.connect("dbname='book_store' user='postgres' password='5283' host='localhost' port='5432'")
cur=con.cursor()
cur.execute("INSERT INTO books(title,author,year,isbn) VALUES(%s,%s,%d,%d)",(title,author,year,isbn))
con.commit()
con.close()
connect()
insert("the sun","helen",1997,23456777)
From the Psycopg FAQ:
Q: I can’t pass an integer or a float parameter to my query: it says a
number is required, but it is a number!
A: In your query string, you always have to use %s placeholders, even when passing a number. All Python objects are converted by Psycopg in
their SQL representation, so they get passed to the query as strings.
See Passing parameters to SQL queries.
So I guess you just have to replace the %d with %s.

Resources