my sql commands are working for instance to truncate table - but my load data command isn't actually inserting any records - how can I confirm py is reading my input file correctly or at all?
cursor.execute("LOAD DATA LOCAL INFILE '/home/user/mongo_exported_users.csv' INTO TABLE tbl.users IGNORE 1 LINES")
row_count = cursor.rowcount
for i in range(row_count):
line = cursor.fetchone()
print(line)
mydb.close()
print("Done")
The output of the fetchone is just None for all rows so IDK whats going on!
MySQL transactions have to be managed by commit. add the below line before closing the connection.
mydb.commit()
or
you can make autocommit just after the connection as shown below
mydb.autocommit = true
This should do the job
Related
I have a Python program that executes an Oracle stored procedure. The SP creates a temp table and then the Python program queries that table and writes the data to an XML file with formatting.
Forgive the noob question, but for some reason the for loop that I'm using to export to the XML file does not export all records. If I limit the query that creates the XML to 15 rows, it works and creates the file. For any value above 15, the program completes, but the file isn't created.
However, this isn't always consistent. If I do multiple runs for 15, 16, or 17 rows, I get a file. But if I try 20, no file is created. No errors, just no file.
This was the initial code. The 'sql' runs against an Oracle private temp table and formats the XML:
cursor.execute(sql)
rows = cursor.fetchall()
with open(filename, 'a') as f:
f.write('<ROWSET>')
for row in rows:
f.write(" ".join(row))
f.write('</ROWSET>')
cursor.close()
Then I changed it to this, but again, no file is created:
cursor.execute(sql)
with open(filename, 'a') as f:
f.write('<ROWSET>')
while True:
rows = cursor.fechmany(15)
for row in rows:
f.write(" ".join(row))
f.write('</ROWSET>')
cursor.close()
I've run the 'free' command and reviewed it with my DBA, and it doesn't appear to be a memory issue. The typical size of the output table is about 600 rows. The table itself has 36 columns.
The indentation may not look right the way I've pasted it here, but the program does work. I just need a way to export all rows. Any insight would be greatly appreciated.
I'm on a Linux box using Python 3.8.5.
Here is the query (minus proprietary information) that is executed against the temp table in the cursor.execute(sql):
SELECT XMLELEMENT("ROW",
XMLFOREST(
carrier_cd,
prscrbr_full_name,
prscrbr_first_name,
prscrbr_last_name,
d_phys_mstr_id,
prscrbr_id,
prscrbr_addr_line_1,
prscrbr_addr_line_2,
prscrbr_city,
prscrbr_state_cd,
prscrbr_zip,
specialty_cd_1,
specialty,
unique_patient_reviewed,
patient_count_db_oral,
patient_count_cv_aa,
patient_count_cv_lipo,
PDC_DIABETES,
PDC_HTN,
PDC_STATINS,
Rating_Diabetes,
Rating_HTN,
Rating_Statins,
PDC_DIABETES,
PDC_HTN,
PDC_STATINS,
M_PC_DB_ORAL,
M_PC_CV_AA,
M_PC_CV_LIPO,
M_PDC_DIABETES,
M_PDC_HTN,
M_PDC_STATINS
),
XMLAGG
(
XMLFOREST(
case when carrier_hq_cd is not null
then XMLConcat(
XMLELEMENT("PATIENT_ID", patient_id),
XMLELEMENT("PATIENT_NAME", patient_name),
XMLELEMENT("DOB", dob),
XMLELEMENT("PHONE_NO", phone_no),
XMLELEMENT("MEMBER_PDC_DIABETES", MEMBER_PDC_DIABETES),
XMLELEMENT("MEMBER_PDC_HTN", MEMBER_PDC_HTN),
XMLELEMENT("MEMBER_PDC_STATINS", MEMBER_PDC_STATINS)
)
end "PATIENT_INFO"
)
ORDER BY patient_id
)
)XMLOUT
FROM ORA$PTT_QCARD_TEMP
GROUP BY
carrier_cd,
prscrbr_full_name,
prscrbr_first_name,
prscrbr_last_name,
d_phys_mstr_id,
prscrbr_id,
prscrbr_addr_line_1,
prscrbr_addr_line_2,
prscrbr_city,
prscrbr_state_cd,
prscrbr_zip,
specialty_cd_1,
specialty,
unique_patient_reviewed,
patient_count_db_oral,
patient_count_cv_aa,
patient_count_cv_lipo,
PDC_Diabetes,
PDC_HTN,
PDC_Statins,
Rating_Diabetes,
Rating_HTN,
Rating_Statins,
M_PC_DB_ORAL,
M_PC_CV_AA,
M_PC_CV_LIPO,
M_PDC_DIABETES,
M_PDC_HTN,
M_PDC_STATINS
If I could, I'd give #Axe319 credit as his idea that it was a database problem was correct. For some reason, Python didn't like that long XML query, so I incorporated it into the stored procedure. Then, the Python was like this:
# SQL query for XML data.
sql_out = """select * from DATA_OUT"""
cursor.execute(sql_out)
columns = [i[0] for i in cursor.description]
allRows = cursor.fetchall()
# Open the file for writing and write the first row.
xmlFile = open(filename, 'w')
xmlFile.write('<ROWSET>')
# Loop through the allRows data set and write it to the file.
for rows in allRows:
columnNumber = 0
for column in columns:
data = rows[columnNumber]
if data == None:
data = ''
xmlFile.write('%s' % (data))
columnNumber += 1
# Write the final row and close the file.
xmlFile.write('</ROWSET>')
xmlFile.close()
I am new Python3 so for give me for asking such question but I couldn't find answer on Google, I have Python scanning a file directory and I need to add in an open and close bracket and a new var into the Python result, so that I can insert it into the database.
Mysql requires inserts to be wrapped in brackets in the val =[('test.mp4',newip) ] This works as I get 1 was inserted when I run the hard coded script.
So what I am trying to archive is to modify the result of the scan and add the following
open/close brackets and the new newip into the result of the scan like the following example
Scan result
['test.mp4', 'test_2.mp4', 'test_3.mp4', test_4.mp4']
Insert new result (modified)
[('test',newip), ('test_2.mp4',newip), ('test_3.mp4',newip), ('test_4.mp4',newip)]
When hard coded its works
root#ubuntu:~# python3 testscan.py
['test.mp4', 'test_2.mp4', 'test_3.mp4', test_4.mp4']
1 was inserted.
Please can anyone advise how to achieve this, below is the full code
import os, mysql.connector, re, uuid
files = [f.name for f in (os.scandir('/var/www/html/media/usb1')) if f.name.endswith('.mp4')]
print(files)
newip = (':'.join(re.findall('..', '%012x' % uuid.getnode())))
mydb = mysql.connector.connect(
host="127.0.0.1",
user="user",
password="password",
database="database"
)
mycursor = mydb.cursor()
sql = "INSERT IGNORE INTO files (file,ip) VALUES (%s,%s)"
val =[('test.mp4',newip)]
mycursor.executemany(sql, val)
mydb.commit()
print(mycursor.rowcount, "was inserted.")
So if you want to add the newip to the scan you can use a list comprehension:
files = ['test.mp4', 'test_2.mp4', 'test_3.mp4', 'test_4.mp4']
sql_values = [(file, newip) for file in files]
the result looks like this:
[('test.mp4', newip), ('test2.mp4', newip), ('test3.mp4', newip), ('test4.mp4', newip)]
I have been reading a number of threads on how to accomplish this but for some reason it is not working.
I need to delete a Row from a database using a string variable from an entry widget as the "WHERE (variable name)=" is used in the DB query.
The entry widget data is stored as Snippet_Name and the same name is being used as a column name in the DB.
The database has 7 columns but I am only using the 1 column for the query and I want to delete the complete row which contains the entry variable. I have tried variations of DELETE with no success.
The code being used is:
def delete_code():
try:
snippetname = Snippet_Name.get()
sql_delete_query = ('DELETE FROM Code WHERE Snippet_Name = "?"', (snippetname))
c.execute(sql_delete_query)
conn.commit()
except:
messagebox.showerror('PYSnippet', 'Failed to delete record')
A little help and hint would be appreciated.
I went over the query and found 2 errors that needed to be address to correct the problem. The first problem was I had the ? enclosed in quotations which should not be there. Second problem was forgetting to a comma to the variable.
def delete_code():
try:
snippetname = Snippet_Name.get()
sql_delete_query = ('DELETE FROM Code WHERE Snippet_Name = ?' (snippetname,))
c.execute(sql_delete_query)
conn.commit()
except:
messagebox.showerror('PYSnippet', 'Failed to delete record')
I'm trying to extract some legacy data from a Teradata server, but some of the records contain weird characters that don't register in python, such as "U+ffffffc2".
Currently,
I'm using pyodbc to extract the data from Teradata
Placing the results into a numpy array (because when I put it directly into pandas, It interprets all of the columns as a single column of type string)
Then I turn the numpy array into a pandas dataframe to change things like Decimal("09809") and Date("2015,11,14") into [09809,"11,14,2015"]
Then I try to write it to a file, where this error occurs
ValueError: character U+ffffffc2 is not in range [U+0000; U+10ffff]
I don't have access to edit this data, so from a client perspective what can I do to skip or, preferably, remove the character before writing it trying to write it to a file and getting the error?
Currently, I have a "try and except" block to skip queries with erroneous data, but I have to query the data in row chunks of at least 100. So if I just skip it, I lose 100 or more lines at a time. As I mentioned before, however, I would prefer to keep the line, but remove the character.
Here's my code. (Feel free to point out any bad practices as well!)
#Python 3.4
#Python Teradata Extraction
#Created 01/28/16 by Maz Baig
#dependencies
import pyodbc
import numpy as np
import pandas as pd
import sys
import os
import psutil
from datetime import datetime
#create a global variable for start time
start_time=datetime.now()
#create global process variable to keep track of memory usage
process=psutil.Process(os.getpid())
def ResultIter(curs, arraysize):
#Get the specified number of rows at a time
while True:
results = curs.fetchmany(arraysize)
if not results:
break
#for result in results:
yield results
def WriteResult(curs,file_path,full_count):
rate=100
rows_extracted=0
for result in ResultIter(curs,rate):
table_matrix=np.array(result)
#Get shape to make sure its not a 1d matrix
rows, length = table_matrix.shape
#if it is a 1D matrix, add a row of nothing to make sure pandas doesn't throw an error
if rows < 2:
dummyrow=np.zeros((1,length))
dummyrow[:]=None
df = pd.DataFrame(table_matrix)
#give the user a status update
rows_extracted=rows+rows_extracted
StatusUpdate(rows_extracted,full_count)
with open(file_path,'a') as f:
try:
df.to_csv(file_path,sep='\u0001',encoding='latin-1',header=False,index=False)
except ValueError:
#pass afterwards
print("This record was giving you issues")
print(table_matrix)
pass
print('\n')
if (rows_extracted < full_count):
print("All of the records were not extracted")
#print the run durration
print("Duration: "+str(datetime.now() - start_time))
sys.exit(3)
f.close()
def StatusUpdate(rows_ex,full_count):
print(" ::Rows Extracted:"+str(rows_ex)+" of "+str(full_count)+" | Memory Usage: "+str(process.memory_info().rss/78
def main(args):
#get Username and Password
usr = args[1]
pwd = args[2]
#Define Table
view_name=args[3]
table_name=args[4]
run_date=args[5]
#get the select statement as an input
select_statement=args[6]
if select_statement=='':
select_statement='*'
#create the output filename from tablename and run date
file_name=run_date + "_" + table_name +"_hist.dat"
file_path="/prod/data/cohl/rfnry/cohl_mort_loan_perfnc/temp/"+file_name
if ( not os.path.exists(file_path)):
#create connection
print("Logging In")
con_str = 'DRIVER={Teradata};DBCNAME=oneview;UID='+usr+';PWD='+pwd+';QUIETMODE=YES;'
conn = pyodbc.connect(con_str)
print("Logged In")
#Get number of records in the file
count_query = 'select count (*) from '+view_name+'.'+table_name
count_curs = conn.cursor()
count_curs.execute(count_query)
full_count = count_curs.fetchone()[0]
#Generate query to retrieve all of the table data
query = 'select '+select_statement+' from '+view_name+'.'+table_name
#create cursor
curs = conn.cursor()
#execute query
curs.execute(query)
#save contents of the query into a matrix
print("Writting Result Into File Now")
WriteResult(curs,file_path,full_count)
print("Table: "+table_name+" was successfully extracted")
#print the scripts run duration
print("Duration: "+str(datetime.now() - start_time))
sys.exit(0)
else:
print("AlreadyThere Exception\nThe file already exists at "+file_path+". Please remove it before continuing\n")
#print the scripts run duration
print("Duration: "+str(datetime.now() - start_time))
sys.exit(2)
main(sys.argv)
Thanks,
Maz
If you have only 4-byte unicode points giving an error, this probably may help.
One solution is to register a custom error handler using codecs.register_error, which would filter out error points and then just try to decode:
import codecs
def error_handler(error):
return '', error.end+6
codecs.register_error('nonunicode', error_handler)
b'abc\xffffffc2def'.decode(errors='nonunicode')
# gives you 'abcdef' which's exactly what you want
You may futher impove your handler to catch more complicated errors, see https://docs.python.org/3/library/exceptions.html#UnicodeError and https://docs.python.org/3/library/codecs.html#codecs.register_error for details
I'm trying to fetch some data from a column whose DATA_TYPE=NUMBER(1,0) with this piece of code:
import cx_Oracle
conn = cx_Oracle.connect(usr, pwd, url)
cursor = conn.cursor()
cursor.execute("SELECT DELETED FROM SERVICEORDER WHERE ORDERID='TEST'")
print(cursor.fetchone()[0])
which complains thus:
Traceback (most recent call last):
File "main.py", line 247, in <module>
check = completed()
File "main.py", line 57, in completed
deleted = cursor.fetchone()[0]
cx_Oracle.DatabaseError: OCI-22061: invalid format text [T
Replacing 'DELETED' column with one whose DATA_TYPE=VARCHAR2 does not throw such a complaint.
I am running in to this problem now using cx_Oracle 5.0.4 with Unicode support. The above accepted solution did not work for me. The DELETED column in the question is a Numeric column, which is what causes this bug.
According to the mailing list ( http://comments.gmane.org/gmane.comp.python.db.cx-oracle/2390 ) it may be a bug in Oracle that shows only in cx_Oracle with Unicode support.
from the link:
"When I build cx_Oracle without Unicode support, it all works as expected.
When I build cx_Oracle with Unicode support, attempting to use a query
that returns a numeric value (such as):
con = Connection( ... )
cur = con.cursor()
cur.execute( 'SELECT 1 FROM DUAL' )
rows = cur.fetchall()
results in this exception:
cx_Oracle.DatabaseError: OCI-22061: invalid format text [T
"
What I did to work around it, is on the select statement, do:
cur.execute( 'SELECT to_char(1) FROM DUAL' )
rows = cur.fetchall()
for row in rows:
val = int(row[0])
It's pretty ugly, but it works.
These types of errors went away when I upgraded to cx_Oracle 5.1. If the RPM doesn't install (like it happened for me on Red Hat 5.5) then you can usually rpm2cpio the file, take the cx_Oracle.so and put it into your python site-packages directory.
A work-around is putting time.sleep(1) before cursor.fetchone():
...
cursor.execute("SELECT DELETED FROM SERVICEORDER WHERE ORDERID='TEST'")
time.sleep(1)
print(cursor.fetchone()[0])
I had the same error.
Commit helped me:
conn = cx_Oracle.connect(...)
...
cursor.execute()
conn.commit()