http://initd.org/psycopg/docs/extras.html
psycopg2.extras.execute_values has a parameters page_size.
I'm doing an INSERT INTO... ON CONFLICT... with RETURNING ID.
The problem is that the cursor.fetchall() give me back only the last "page", that is, 100 ids (default of page_size).
Without modifying page_size parameters, is it possible to iterate over the results, to get the total number of rows updated ?
The best and shortest answer would be using fetch = True in parameter as stated in here
all_ids = psycopg2.extras.execute_values(cur, query, data, template=None, page_size=10000, fetch=True)
# all_ids will return all affected rows with array like this [ [1], [2], [3] .... ]
I ran into the same issue. I work around this issue by batching my calls to execute_values(). I'll set my_page_size=1000, then iterate over my values, filling argslist until i have my_page_size items. Then I'll call execute_values(cur, sql, argslist, page_size=my_page_size). And iterate over cur to get those ids.
Without modifying page_size parameters, is it possible to iterate over
the results, to get the total number of rows updated ?
Yes.
try:
conn = psycopg2.connect(...)
cur = conn.cursor()
query = """
WITH
items (eggs) AS (VALUES %s),
inserted AS (
INSERT INTO spam (eggs)
SELECT eggs FROM items
ON CONFLICT (eggs) DO NOTHING
RETURNING id
)
SELECT id FROM spam
WHERE eggs IN (SELECT eggs FROM items)
UNION
SELECT id FROM inserted
"""
eggs = (('egg_{}'.format(i % 666),) for i in range(10_000))
ids = psycopg2.extras.execute_values(cur, query, argslist=eggs, fetch=True)
# Do whatever with `ids`. `len(ids)` I suppose?
finally:
if connection:
cur.close()
conn.close()
I overkilled query on purpose to address some gotchas:
WITH items (eggs) AS (VALUES %s) is done to be able to use argslist in two places at once;
RETURNING with ON CONFLICT will return only ids which were actually inserted, conflicting ones are omitted from INSERT' direct results. To solve that all this SELECT ... WHERE ... UNION SELECT mumbo jumbo is done;
to get all values which you asked for: ids = psycopg2.extras.execute_values(..., fetch=True).
A horrible interface oddity considering that all other cases are done like
cur.execute(...) # or other kind of `execute`
rows = cur.fetchall() # or other kind of `fetch`
So if you want only the number of inserted rows then do
try:
conn = psycopg2.connect(...)
cur = conn.cursor()
query = """
INSERT INTO spam (eggs)
VALUES %s
ON CONFLICT (eggs) DO NOTHING
RETURNING id
"""
eggs = (('egg_{}'.format(i % 666),) for i in range(10_000))
ids = psycopg2.extras.execute_values(cur, query, argslist=eggs, fetch=True)
print(len(ids)
finally:
if connection:
cur.close()
conn.close()
Related
This connection works, but the result is just the text of the query itself:
Connection = cx_Oracle.connect(user=username, password=password, dsn=dsn, encoding=enc)
query = 'simple select statement'
cursor = Connection.cursor()
cursor.execute(query)
Connection.commit()
cursor.close()
print(query)
The result in the dataframe prints 'SELECT RECV_MBR_ID...' instead of the ID's. What am I missing?
This is not unexpected! You are simply printing the value to which you set that variable! You need to fetch the results. You can do this in one of several ways. I'll show a couple of the more common ones here:
for row in cursor.execute(query):
print(row)
OR
cursor.execute(query)
print(cursor.fetchall())
I am trying to understand how to capture the number of records deleted from a delete command in Python code using psycopg2. Typically after a delete command is issued in PostgreSql the number of rows deleted appears (i.e., DELETE 8, where 8 represents the number of rows deleted). I would like to record this count after each delete in order to report back to the caller the number of rows deleted without needing to make a separate SELECT count(*) before the DELETE command.
Consider this function:
def qexe(conn, query, fetch_type):
cursor = conn.cursor()
cursor.execute(query)
if fetch_type != None:
if fetch_type == 'fetchall':
query_result = cursor.fetchall()
return query_result
elif fetch_type == 'fetchone':
query_result = cursor.fetchone()
return query_result
else:
raise Exception(f"Invalid arg fetch_type: {fetch_type}")
cursor.close()
After running each of the following, I keep getting the error: psycopg2.ProgrammingError: no results to fetch:
qq = """DELETE FROM INV WHERE ITEM_ID = '{}';"""
item='123abc'
resp0 = qexe(conn, qq.format(item), 'fetchone')
resp1 = qexe(conn, qq.format(item), 'fetchall')
You could make use of RETURNING, so you will be able to get 'something' back which can be fetched from the cursor:
qq = """DELETE FROM INV WHERE ITEM_ID = '{}' RETURNING ITEM_ID"""
resp0 = qexe(conn, qq.format(item), 'fetchone')
I'm trying to create a pop function getting a row of data from a sqlite database and deleting that same row. I would like to not have to create an ID column so I am using ROWID. I want to always get the first row and return it. This is the code I have:
import sqlite3
db = sqlite3.connect("Test.db")
c=db.cursor()
def sqlpop():
c.execute("SELECT * from DATA WHERE ROWID=1")
data = c.fetchall()
c.execute("DELETE from DATA WHERE ROWID=1")
db.commit()
return(data)
when I call the function it gets the first item correctly, but after the first call the function returns nothing. like this:
>>> sqlpop()
[(1603216325, 'placeholder IP line 124', 'placeholder Device line 124', '1,2,0', 1528, 1564)]
>>> sqlpop()
[]
>>> sqlpop()
[]
>>> sqlpop()
[]
what do I need to change for this function to work correctly?
update:
using what Schwern said I got the funtion to work:
def sqlpop():
c.execute("SELECT * from DATA ORDER BY ROWID LIMIT 1")
data = c.fetchone()
c.execute("DELETE from DATA ORDER BY ROWID LIMIT 1")
db.commit()
return data
rowid is not the row order, it is a unique identifier for the row created by SQLite unless you say otherwise.
SQL rows have no inherent order. You could grab just one row...
select * from table limit 1;
But you'll get them in no guaranteed order. And without a rowid you have no way to identify it again to delete it.
If you want to get the "first" row you must define what "first" means. To do that you need something to order by. For example, a timestamp. Or perhaps an auto-incrementing integer. You cannot use rowid, rowids are not guaranteed to be assigned in any particular order.
select *
from table
where created_at = max(created_at)
limit 1
So long as created_at is indexed, that should work fine. Then delete by its rowid.
You also don't need to use fetchall to fetch one row, use fetchone. In general, fetchall should be avoided as it risks consuming all your memory by slurping all the data in at once. Instead, use iterators.
for row in c.execute(...)
Hi I'm having an issue fetching all rows from my table and printing them all in one rather that fetching a row and returning the data one by one. Is there a way you can get all rows in one.
Here is what I'm working with:
async def r_list(self, ctx):
"""Gets a list of restricted users."""
guild = ctx.message.guild
for member in guild.members:
conn = psycopg2.connect(DATABASE_URL, sslmode='require')
cursor = conn.cursor()
cursor.execute("SELECT time, username FROM blacklist WHERE username=%s AND time IS NOT NULL", (member.id, ))
results = cursor.fetchall()
for row in results:
user_id=row[1]
username = self.bot.get_user(user_id)
timestamp=row[0].strftime("%#d %b %Y, at %I:%M%p")
await ctx.send(f"User: {username} Until: {timestamp}")
cursor.close()
conn.close()
Your WHERE clause will filter out most of the other rows, so in case you really want all rows, you can use
SELECT * FROM blacklist
This will return every column from every row, rather than only the rows where the id's match. Afterwards, fetchall() should do the trick.
I'm trying to delete a row from my pysimplegui table that will also delete the same row data from my sqlite3 database. Using events, I've tried to use the index eg. -TABLE- {'-TABLE-': [1]} to index the row position using values['-TABLE-'] like so:
if event == 'Delete':
row_index = 0
for num in values['-TABLE-']:
row_index = num + 1
c.execute('DELETE FROM goals WHERE item_id = ?', (row_index,))
conn.commit()
window.Element('-TABLE-').Update(values=get_table_data())
I realized that this wouldn't work since I'm using a ROW_ID in my database that Auto-increments with every new row of data and stays fixed like so (this is just to show how my database is set up):
conn = sqlite3.connect('goals.db')
c = conn.cursor()
c.execute('''CREATE TABLE goals (item_id INTEGER PRIMARY KEY, goal_name text, goal_type text)''')
conn.commit()
conn.close()
Is there a way to use the index ( values['-TABLE-'] ) to find the data inside the selected row in pysimplegui and then using the selected row's data to find the row in my sqlite3 database to delete it, or is there any other way of doing this that I'm not aware of?
////////////////////////////////////////
FIX:
Upon more reading into the docs I discovered a .get() method. This method returns a nested list of all Table Rows, the method is callable on the element of '-TABLE-'. Using values['-TABLE-'] I can also find the row index and use the .get() method to index the specific list where the Data lays which I want to delete.
Here is the edited code that made it work for me:
if event == 'Delete':
row_index = 0
for num in values['-TABLE-']:
row_index = num
# Returns nested list of all Table rows
all_table_vals = window.element('-TABLE-').get()
# Index the selected row
object_name_deletion = all_table_vals[row_index]
# [0] to Index the goal_name of my selected Row
selected_goal_name = object_name_deletion[0]
c.execute('DELETE FROM goals WHERE goal_name = ?', (selected_goal_name,))
conn.commit()
window.Element('-TABLE-').Update(values=get_table_data())
Here is a small example to delete a row from table
import sqlite3
def deleteRecord():
try:
sqliteConnection = sqlite3.connect('SQLite_Python.db')
cursor = sqliteConnection.cursor()
print("Connected to SQLite")
# Deleting single record now
sql_delete_query = """DELETE from SqliteDb_developers where id = 6"""
cursor.execute(sql_delete_query)
sqliteConnection.commit()
print("Record deleted successfully ")
cursor.close()
except sqlite3.Error as error:
print("Failed to delete record from sqlite table", error)
finally:
if (sqliteConnection):
sqliteConnection.close()
print("the sqlite connection is closed")
deleteRecord()
In your case id will me the name of any column name which has unique value for every row in thetable of the database