I'm trying to use Python to run a PostgreSQL function that inserts data from one table into another. If I run the function in pgAdmin it works, but if I run it through Python I get the return value, which says it's executed, but it doesn't insert any records.
Here is my Python code
import psycopg2
from parse_config import fn_parse_config
def run_function():
conn = None
try:
# read database configuration
login_params = fn_parse_config("etl_process.ini", "psycopg_login")
# connect to the PostgreSQL database
conn = psycopg2.connect(**login_params)
# create a cursor object for execution
cur = conn.cursor()
# another way to call a stored procedure
#cur.execute("SELECT * FROM usp_test();")
cur.callproc('usp_test')
# process the result set
row = cur.fetchone()
while row is not None:
print(row)
row = cur.fetchone()
# close the communication with the PostgreSQL database server
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
if __name__ == '__main__':
run_function()
Here is my function
CREATE OR REPLACE FUNCTION usp_test()
RETURNS integer AS $$
BEGIN
INSERT INTO staging_raw_file (Site,
StyleCode)
select CAST(a.Site as VARCHAR(10)),
CAST(a.StyleCode as VARCHAR(30))
from import_raw_file a;
return 5;
END; $$
LANGUAGE plpgsql;
Found the answer, it needs autocommit setting
conn = psycopg2.connect(**login_params)
conn.autocommit = True
without that it doesn't actually run the insert, but it will return results from just a select without that
Related
I am working on a process to automate the generation of a report in Word. I have a table in my template and I would like to insert images into some cells in the table. If I insert only one image into the table, it works perfectly. However, if I insert more than one image, the output file gets corrupted. While opening the file I am getting an error says "Word found unreadable content in XXXX. Do you want to recover?..."
My table in the template.docx looks like this:
import psycopg2
from docxtpl import DocxTemplate, InlineImage
from docx2pdf import convert
import os
from datetime import date
# Report template and output paths
report_name_docx = date.today().strftime("%Y%m%d") + 'xxx_report.docx'
report_template = "C:\\workspace\\xxx_template.docx"
report_output = os.path.join("C:\\workspace\\template\\",report_name_docx)
id = int(input('Enter please id:'))
# Connect and query tables in Postgresql
try:
connection = psycopg2.connect(user='xxx',
password='xxx',
host="xxx",
port="xxx",
database="xxx")
cursor = connection.cursor()
# Fetch records
postgreSQL_select_Query1 = f"select * from xxx where id = {id}"
cursor.execute(postgreSQL_select_Query1)
apsfr_records = cursor.fetchall()
# Get name of the apsfr
for row in apsfr_records:
name = row[3]
except (Exception, psycopg2.Error) as error:
print(f"Error while connecting to PostgreSQL {error}")
finally:
# closing database connection.
if (connection):
cursor.close()
connection.close()
print("PostgreSQL connection is closed")
# Don't want to overwrite your template with the loaded images version.
assert report_template!=report_output
doc = DocxTemplate(report_template)
context_to_load = {
'name': name
}
context_to_load['image1'] = InlineImage(doc, os.path.join(image_dirname, "umgesetztdouble.png"))
context_to_load['Image2'] = InlineImage(doc, os.path.join(image_dirname, "line.png"))
# load the context to the word template
doc.render(context_to_load)
doc.save(report_output)
print("-----")
print('----- FINITO')
def depositamt():
conn = sqlite3.connect("atm.db")
cursor = conn.cursor()
cursor.execute('''UPDATE data SET balance = balance + '%s' WHERE username = '%s' '''%(depositamount.get(),username.get()))
messagebox.showinfo("Successful", "deposit amount successfully")
conn.commit()
cursor.close()
conn.close()
This function gets the username from username Entry box and updates the record. But the database is not being updated.
Are you sure about your syntax?
It is not the syntax I am familiar with. Assuming both functions work as expected and username is already present into your base I would use the following statement:
cursor.execute("UPDATE data SET balance=? WHERE username=?", (depositamount.get(), username.get()))
Regards.
So I've queried data from oracle database using cursor.execute(). A relatively simple select query. It works.
But when I try to fetch data from it, python crashes.
The same occurs for fetchall(), fetchmany() and fetchone().
When the query first broke in fetchmany() I decided to loop through fetchone() and it worked for the first two rows then broke at the third.
I'm guessing it is because there's too much data in third row.
So, is there any way to bypass this issue and pull the data?
(Please ignore the wrong indents could not copy properly in my phone)
EDIT:
I removed four columns with type "ROWID". There was no issue after that. I was easily able to fetch 100 rows in one go.
So to confirm my suspicion I went ahead and created another copy with only those rowed columns and it crashes as expected.
So is there any issue with ROWID type?
Test table for the same.
Insert into TEST_FOR_CX_ORACLE (Z$OEX0_LINES,Z$OEX0_ORDER_INVOICES,Z$OEX0_ORDERS,Z$ITEM_ROWID) values ('ABoeqvAEyAAB0HOAAM','AAAL0DAEzAAClz7AAN','AAAVeuABHAAA4vdAAH','ABoeo+AIVAAE6dKAAQ');
Insert into TEST_FOR_CX_ORACLE (Z$OEX0_LINES,Z$OEX0_ORDER_INVOICES,Z$OEX0_ORDERS,Z$ITEM_ROWID) values ('ABoeqvABQAABKo6AAI','AAAL0DAEzAAClz7AAO','AAAVeuABHAAA4vdAAH','ABoeo+AIVAAE6dKAAQ');
Insert into TEST_FOR_CX_ORACLE (Z$OEX0_LINES,Z$OEX0_ORDER_INVOICES,Z$OEX0_ORDERS,Z$ITEM_ROWID) values ('ABoeqvABQAABKo6AAG','AAAL0DAEzAAClz7AAP','AAAVeuABHAAA4vdAAH','ABoeo+AHIAAN+OIAAM');
Insert into TEST_FOR_CX_ORACLE (Z$OEX0_LINES,Z$OEX0_ORDER_INVOICES,Z$OEX0_ORDERS,Z$ITEM_ROWID) values ('ABoeqvAEyAAB0HOAAK','AAAL0DAEzAACl0EAAC','AAAVeuABHAAA4vdAAH','ABoeo+AHIAAN+OIAAM');
Script:
from cx_Oracle import makedsn,connect,Cursor
from pandas import read_sql_table, DataFrame, Series
from time import time
def create_conn( host_link , port , service_name , user_name , password ):
dsn=makedsn(host_link,port,service_name=service_name)
return connect(user=user_name, password=password, dsn=dsn)
def initiate_connection(conn):
try:
dbconnection = create_conn(*conn)
print('Connected to '+conn[2]+' !')
except Exception as e:
print(e)
dbconnection = None
return dbconnection
def execute_query(query,conn):
dbconnection=initiate_connection(conn)
try:
cursor = dbconnection.cursor()
print ('Cursor Created!')
return cursor.execute(query)
except Exception as e:
print(e)
return None
start_time = time()
query='''SELECT * FROM test_for_cx_oracle'''
try:
cx_read_query = execute_query(query,ecspat_c)
time_after_execute_query = time()
print('Query Executed')
columns = [i[0] for i in cx_read_query.description]
time_after_getting_columns = time()
except Exception as e:
print(e)
print(time_after_execute_query-start_time,time_after_getting_columns-time_after_execute_query)
Unfortunately, this is a bug in the Oracle Client libraries. You will see it if you attempt to fetch the same rowid value multiple times in consecutive rows. If you avoid that situation all is well. You can also set the environment variable ORA_OCI_NO_OPTIMIZED_FETCH to the value 1 before you run the query to avoid the problem.
This has been reported earlier here: https://github.com/oracle/python-cx_Oracle/issues/120
I am using Python 3.6.2 and Tweepy 3.3.0.
So when I try to retrieve trends from Twitter via the Tweepy API and just print them in my console it works pretty fine like this:
auth = tweepy.OAuthHandler(cons_tok, cons_sec)
auth.set_access_token(app_tok, app_sec)
twitter_api = tweepy.API(auth)
trends = twitter_api.trends_place(1)
for trend in trends[0]["trends"]:
print(trend['name'])
It works fine and I have a list.
Now when I try to store these into a database (I choose SQLite just to try):
class TwitterMain():
def __init__(self, conn):
self.auth = tweepy.OAuthHandler(cons_tok, cons_sec)
self.auth.set_access_token(app_tok, app_sec)
self.api = tweepy.API(self.auth)
self.conn = conn
self.c = self.conn.cursor()
def get_trends(self):
trends = self.api.trends_place(1)
trend_data = []
for trend in trends[0]["trends"]:
trend_tweets = []
trend_tweets.append(trend['name'])
tt = tweepy.Cursor(self.api.search, q = trend['name']).items(3)
for t in tt:
trend_tweets.append(self.get_tweet_html(t.id))
trend_data.append(tuple(trend_tweets))
self.c.executemany("INSERT INTO trend_data VALUES (?,?,?,?, DATETIME('now'))", trend_data)
self.conn.commit()
def get_tweet_html(self, id):
oembed = self.api.get_oembed(id=id, hide_media = True, hide_thread = True)
tweet_html = oembed['html'].strip("\n")
return tweet_html
if __name__ == "__main__":
try:
conn = sqlite3.connect(db)
twit = TwitterMain(conn)
twit.get_trends()
except Exception as e:
print(e.__doc__)
finally:
conn.close()
This is just the part where I store the trends data. The database is already created as well as the table.
Running this creates a Tweepy Exception, and when searching online it says it's the time limit but if so why when I just print, it works fine and not when I try to store them?
There is no table to insert the value into, and the code that you have provided is throwing an exception:
self.c.executemany("INSERT INTO trend_data VALUES (?,?,?,?, DATETIME('now'))", trend_data)
sqlite3.OperationalError: no such table: trend_data
Just add a line to create the table as well in the init method:
self.c.execute("""CREATE TABLE mytable
(c1,c2,c3,c4,c5)""")
and use this to fill in the values:
self.c.executemany("INSERT INTO mytable VALUES (?,?,?,?, DATETIME('now'))", trend_data)
I am trying to write a function to do a simple insert.
Here is what I have tried so far
#! /usr/bin/env python3
#import
import sqlite3 as lite
#trying an insert version 1 (does nothing)
def createTableTask():
"""
Create a new table with the name Task
"""
#Connnection to the database and cursor creation
con = lite.connect('./exemple.sqlite')
con.row_factory = lite.Row
cur = con.cursor()
#that does nothing
try:
cur.execute('''CREATE TABLE Tasks (\
Name TEXT PRIMARY KEY, \
Description TEXT, \
Priority TEXT);''')
except lite.IntegrityError as error_SQLite:
print("error: "+ str(error_SQLite))
else:
print("No error has occured.")
con.close();
def insert1():
"""
insert a new task
"""
#Allocating variables data
taskName = 'finish code'
taskDescription = 'debug'
taskPriority = 'normal'
#Connnection to the database and cursor creation
con = lite.connect('./exemple.sqlite')
con.row_factory = lite.Row
cur = con.cursor()
#that does nothing
try:
with con:
cur.execute('''INSERT INTO Tasks (Name, Description, Priority) \
VALUES (?, ?, ?)''', (taskName, taskDescription, taskPriority))
except lite.IntegrityError as error_SQLite:
print("error: "+ str(error_SQLite))
else:
print("No error has occured. but no insert happend ?")
con.close();
def showResult():
"""
Show the result of the insert
"""
con = lite.connect('./exemple.sqlite')
con.row_factory = lite.Row
cur = con.cursor()
cur.execute\
('''SELECT * FROM Tasks ;''')
row = cur.fetchone()
while row:
print(row["Name"], ' | ', row["Description"], ' | ', \
row["Priority"])
row = cur.fetchone()
con.close();
#trying an insert version 2 (this one crash giving :Value error)
def insert2():
"""
insert a new task
"""
#Allocating variables data
taskName = 'finish code'
taskDescription = 'debug'
taskPriority = 'normal'
#Connnection to the database and cursor creation
con = lite.connect('./exemple.sqlite')
con.row_factory = lite.Row
cur = con.cursor()
queryInsert = ('''INSERT INTO Tasks (Name, Description, Priority) \
VALUES (?, ?, ?)''', (taskName, taskDescription, taskPriority))
try:
with con:
cur.execute(queryInsert)
except lite.IntegrityError as error_SQLite:
print("error: "+ str(error_SQLite))
else:
print("No error has occured.")
con.close();
def run():
createTableTask()
insert1()
showResult()
insert2()
showResult()
#calling section
run()
The problem is that none of the insert that I have made so far worked.
The first one does actualy nothing but has a correct syntax
The second one, well it crash.
Here is the output:
spark#spark-Razer-Blade-Pro:~/Documents/testing$ ./exemp.py
No error has occured.
No error has occured. but no insert happend ?
Traceback (most recent call last):
File "./exemp.py", line 98, in
run()
File "./exemp.py", line 94, in run
insert2()
File "./exemp.py", line 83, in insert2
cur.execute(queryInsert)
ValueError: operation parameter must be str
spark#spark-Razer-Blade-Pro:~/Documents/testing$ sqlite3 exemple.sqlite
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> SELECT * FROM Tasks;
sqlite>
I am looking for the most simple fix and maybe know what is wrong with my code. Because Right now I do not know what is going on with the no insert one. Normally it should, or am I missing something ?
queryInsert = ('''INSERT ...''', (taskName, taskDescription, taskPriority))
This makes queryInsert a tuple with two elements.
But to call the execute method, you need two separate parameters.
You could just unpack the tuple:
cur.execute(*queryInsert)
but it might be clearer to use two separate variables:
queryString = '''INSERT ...'''
queryParams = (taskName, taskDescription, taskPriority)
cur.execute(queryString, queryParams)
ok I got around my error. Just posting it because it might help others.
cur.execute() is a fonction that seek a query as it's first argument, than the other argument are the variables needed for the query.
step one: make the query into a variable without it's parameters
queryString = ''' INSERT INTO someTables rowName, rowName2 ... VALUES (?, ?);'''
there should be as much as ? as there are variables needed. in this exemple I need 2
queryValue1 = 'something'
queryValue2 = '123'
Step 2 to call and execute the query :
cur.execute(queryString, queryValue1, queryValue2)
this should be working without problem