Is it possible to pass more than one query in a double with cursor opening statement with psycopg2 (first to open connection, then cursor)?
E.g. to replace:
import psycopg2
def connector():
return psycopg2.connect(**DB_DICT_PARAMS)
########
sql_update1 = ("UPDATE table SET array = %s::varchar[], "
"array_created = true, timestamp = now() AT TIME ZONE 'UTC' "
"WHERE id = %s")
sql_update2 = ("UPDATE table SET json_field = %s "
"WHERE id = %s")
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_update1, [stringArray, ID])
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_update2, [jsonString, ID])
by:
#(...)
sql_update1 = ("UPDATE table SET array = %s::varchar[], "
"array_created = true, timestamp = now() AT TIME ZONE 'UTC' "
"WHERE id = %s")
sql_update2 = ("UPDATE table SET json_field = %s "
"WHERE id = %s")
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_update1, [stringArray, ID])
curs.execute(sql_update2, [jsonString, ID])
What if the second query needs the first one to be completed before, and what if not?
In the shown case, they will definitely update the same record (i.e. row) in the database but not the same fields (i.e. attributes or columns).
Is this precisely authorized because the two SQL statement are committed sequentially, i.e. the first finishes first. Then, after and only after, the second is executed.?
Or is it actually forbidden because they can be executed in parallel, each query without knowing the state of the other at any instant t?
There are no fancy triggers or procedures in the DB. Let's make it first simple.
(Please note that I have purposefully written two queries here, where a single one would have perfectly fit, but it's not always the case, as some computations are still in the way before saving some other results to the same record in the DB).
If you want them to execute at the same time, simply put them in the same string seperated by a semicolon. I'm a little rusty but I think the following should work:
sql_updates = ("UPDATE table SET array = %s::varchar[], "
"array_created = true, timestamp = now() AT TIME ZONE 'UTC' "
"WHERE id = %s;"
"UPDATE table SET json_field = %s "
"WHERE id = %s;")
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_updates, [stringArray, ID, jsonString, ID])
Better avoid this:
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_update1, [stringArray, ID])
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_update2, [jsonString, ID])
Opening a database connection is pretty slow compared to doing a query, so it is much better to reuse it rather than opening a new one for each query. If your program is a script, typically you'd just open the connection at startup and close it at exit.
However, if your program spends a long time waiting between queries, and there will be many instances running, then it would be better to close the connection to not consume valuable RAM on the postgres server for doing nothing. This is common in client/server applications where the client mostly waits for user input. If there are many clients you can also use connection pooling, which offers the best of both worlds at the cost of a bit extra complexity. But if it's just a script, no need to bother with that.
with connector() as conn:
with conn.cursor() as curs:
curs.execute(sql_update1, [stringArray, ID])
curs.execute(sql_update2, [jsonString, ID])
This would be faster. You don't need to build a new cursor, you can reuse the same one. note if you don't fetch the results of the first query before reusing the cursor, then you won't be able to do so after executing the second query, because a cursor only stores the results of the last query. Since these are updates, there are no results, unless you want to check the rowcount to see if it did update a row.
What if the second query needs the first one to be completed before, and what if not?
Don't care. execute() processes the whole query before returning, so by the time python gets to the next bit of code, the query is done.
Is this precisely authorized because the two SQL statement are committed sequentially, i.e. the first finishes first. Then, after and only after, the second is executed.?
Yes
Or is it actually forbidden because they can be executed in parallel, each query without knowing the state of the other at any instant t?
If you want to execute several queries in parallel, for example because a query takes a while and you want to execute it while still running other queries, then you need several DB connections and of course one python thread for each because execute() is blocking. It's not used often.
Related
const result = await connection.execute(
`SELECT * from no_example `, [], { maxRows: 1000 } // bind value for :id
);
but in result i always get empty rows
If you are inserting rows in another tool, or another program. Make sure that you COMMIT the data:
SQL> create table t (c number);
Table created.
SQL> insert into t (c) values (1);
1 row created.
SQL> commit;
Commit complete.
If you are insert using Node.js, look at the autoCommit attribute and connection.commit() function. Also see the node-oracledb documentation on Transaction Management.
Unrelated to your problem, but you almost certainly shouldn't be using maxRows. By default node-oracledb will return all rows. If you don't want all, then add some kind of WHERE clause or row-limiting clause to your query. If you expect a big number of rows, then use a result set so you can access consecutive batches of rows.
We are using psycopg2 jsonb cursor to fetch the data and processing but when ever new thread or processing coming it should not fetch and process the same records which first process or thread.
For that we have try to use the FOR UPDATE but we just want to know whether we are using correct syntax or not.
con = self.dbPool.getconn()
cur = conn.cursor()
sql="""SELECT jsondoc FROM %s WHERE jsondoc #> %s"”"
if 'sql' in queryFilter:
sql += queryFilter 'sql’]
When we print this query, it will be shown as below:
Query: "SELECT jsondoc FROM %s WHERE jsondoc #> %s AND (jsondoc ->> ‘claimDate')::float <= 1536613219.0 AND ( jsondoc ->> ‘claimstatus' = ‘done' OR jsondoc ->> 'claimstatus' = 'failed' ) limit 2 FOR UPDATE"
cur.execute(sql, (AsIs(self.tablename), Json(queryFilter),))
cur.execute()
dbResult = cur.fetchall()
Please help us to clarify the syntax and explain if that syntax is correct then how this query lock the fetched records of first thread.
Thanks,
Sanjay.
If this exemplary query is executed
select *
from my_table
order by id
limit 2
for update; -- wrong
then two resulting rows are locked until the end of the transaction (i.e. next connection.rollback() or connection.commit() or the connection is closed). If another transaction tries to run the same query during this time, it will be stopped until the two rows are unlocked. So it is not the behaviour you are expected. You should add skip locked clause:
select *
from my_table
order by id
limit 2
for update skip locked; -- correct
With this clause the second transaction will skip the locked rows and return next two onces without waiting.
Read about it in the documentation.
How to avoid creating table again and again in python using Oracle database?
Every time I call the function CREATE table query is executed and data is not inserted because the table already exists.
import cx_Oracle
import time
def Database(name,idd,contact):
try:
con = cx_Oracle.connect('arslanhaider/12345#AHS:1521/XE')
cur = con.cursor()
cur.execute("CREATE TABLE Mazdoor(Name varchar(255),EmpID INT,ContactNo INT)")
cur.execute("INSERT INTO Mazdoor VALUES(:1, :2, :3)",( name,idd,contact))
con.commit()
cur.execute("SELECT * FROM Mazdoor")
data = cur.fetchall()
for row in data:
print(row)
except cx_Oracle.Error:
if con:
con.rollback()
finally:
if con:
con.close()
if__name__="__main__"
while True:
n=input("Enter Name::")
i=input("Enter Idd::")
c=input("Enter Contact No::")
Database(n,i,c)
time.sleep(3)
print("Record Successfully Stored......\n\n")
"Obviously, (koff, koff ...) you must know what you are doing!"
If you ask Oracle to CREATE TABLE, knowing in advance that the table might already exist, then your logic should at least be prepared ... through the use of multiple try..except..finally blocks as appropriate, to handle this situation.
If the CREATE TABLE statement fails because the table already exists, then you can be quite sure that an exception will be thrown, and that you, in the relevant except clause, can determine that "this, indeed, is the reason." You might reasonably then choose to ignore this possibility, and to "soldier on."
I have to code on python sqlite3 a function to count rows of a table.
The thing is that the user should input the name of that table once the function is executed.
So far I have the following. However, I don't know how to "connect" the variable (table) with the function, once it's executed.
Any help would be great.
Thanks
def RT():
import sqlite3
conn= sqlite3.connect ("MyDB.db")
table=input("enter table name: ")
cur = conn.cursor()
cur.execute("Select count(*) from ?", [table])
for row in cur:
print str(row[0])
conn.close()
Columns and Tables Can't be Parameterized
As explained in this SO answer, Columns and tables can't be parameterized. A fact that might not be documented by any authoritative source (I couldn't find one, so if you you know of one please edit this answer and/or the one linked above), but instead has been learned through people trying exactly what was attempted in the question.
The only way to dynamically insert a column or table name is through standard python string formatting:
cur.execute("Select count(*) from {0}".format(table))
Unfortunately This opens you up to the possibility of SQL injection
Whitelist Acceptable Column/Table Names
This SO answer explains that you should use a whitelist to check against acceptable table names. This is what it would look like for you:
import sqlite3
def RT():
conn = sqlite3.connect ("MyDB.db")
table = input("enter table name: ")
cur = conn.cursor()
if table not in ['user', 'blog', 'comment', ...]:
raise ... #Include your own error here
execute("Select count(*) from {0}".format(table))
for row in cur:
print str(row[0])
conn.close()
The same SO answer cautions accepting submitted names directly "because the validation and the actual table could go out of sync, or you could forget the check." Meaning, you should only derive the name of the table yourself. You could do this by making a clear distinction between accepting user input and the actual query. Here is an example of what you might do.
import sqlite3
acceptable_table_names = ['user', 'blog', 'comment', ...]
def RT():
"""
Client side logic: Prompt the user to enter table name.
You could also give a list of names that you associate with ids
"""
table = input("enter table name: ")
if table in acceptable_table_names:
table_index = table_names.index(table)
RT_index(table_index)
def RT_index(table_index):
"""
Backend logic: Accept table index instead of querying user for
table name.
"""
conn = sqlite3.connect ("MyDB.db")
cur = conn.cursor()
table = acceptable_table_names[table_index]
execute("Select count(*) from {0}".format(table))
for row in cur:
print str(row[0])
conn.close()
This may seem frivolous, but this keeps the original interface while addressing the potential problem of forgetting to check against a whitelist. The validation and the actual table could still go out of sync; you'll need to write tests to fight against that.
How do I sort this one out?
code:
c.execute("INSERT INTO INPUT33 (NAME) VALUES (?);", (name3,))
c.execute("select MAX(rowid) from [input33];")
conn.commit()
for rowid in cursor:break
for elem in rowid:
m = elem
print(m)
c.execute("select MAX(rowid) from [input];")
for rowid in c:break
for elem in rowid:
m = elem
c.execute("DELETE FROM input WHERE rowid = ?", (m,))
conn.commit()
After running this, i get this:
sqlite3.OperationalError: database is locked
Taken from Python Docs
When a database is accessed by multiple connections, and one of the processes modifies the database, the SQLite database is locked until that transaction is committed. The timeout parameter specifies how long the connection should wait for the lock to go away until raising an exception. The default for the timeout parameter is 5.0 (five seconds).