I am trying to import an employee list with null values; however there are date/time nulls as well as varchar/text. It looks like it's trying to insert a string in any of the date/time fields if its null and this is throwing an error. How do I get around this? Any feedback is greatly appreciated.
The error:
psycopg2.errors.InvalidDatetimeFormat: invalid input syntax for type date: ""
CONTEXT: COPY aides_hcs, line 5, column first_work_date: ""
My code:
import psycopg2
conn = psycopg2.connect(host="192.168.0.250",database="People", user="hcsadmin", password="Temp#2018")
cur = conn.cursor()
with open('C:\Reports\Data_Files\HCS\Active_Employees.csv', 'r') as f:
cur.copy_from(f,'aides_hcs',sep=',', null='None')
conn.commit()
cur.close
conn.close
Related
I've lots of records to be inserted into sql server. I'm using pyodbc and cursor.fast_executemany() to achieve this. The regular cursor.execute() is inserting records too slowly.
I'm following this article as a reference: https://towardsdatascience.com/how-i-made-inserts-into-sql-server-100x-faster-with-pyodbc-5a0b5afdba5
sample code I'm using:
query = "SELECT * FROM dbo.my_table"
df = pd.read_sql(query, conn)
my_insert_statement = f"INSERT INTO myschema.new_table(colA, colB, colC, colD, colE) values(?,?,?,?,?)"
cursor = conn.cursor()
cursor.fast_executemany = True
cursor.fast_executemany(my_insert_statement,df.values.tolist())
conn.commit()
cursor.close()
conn.close()
But I keep getting the below error although I don't have any boolean columns.
'bool' object is not callable
I don't know how to surpass this error and I really need to insert records in bulk and quick into my database table.
Any ideas on why this error occurs and how to solve this?
The second line in the following snippet has to be wrong. You set fast_executemany to True and then try to call it with fast_executemany().
cursor.fast_executemany = True
cursor.fast_executemany(my_insert_statement,df.values.tolist())
I looked at your guide and you have to replace the second line in the snippet with:
cursor.executemany(my_insert_statement,df.values.tolist())
I've got my data put into an SQLite3 database, and now I'm trying to work on a little script to access data I want for given dates. I got the SELECT statement to work with the date ranges, but I can't seem to add another condition to fine tune the search.
db columns id, date, driverid, drivername, pickupStop, pickupPkg, delStop, delPkg
What I've got so far:
import pandas as pd
import sqlite3
sql_data = 'driverperformance.sqlite'
conn = sqlite3.connect(sql_data)
cur = conn.cursor()
date_start = "2021-12-04"
date_end = "2021-12-10"
df = pd.read_sql_query("SELECT DISTINCT drivername FROM DriverPerf WHERE date BETWEEN :dstart and :dend", params={"dstart": date_start, "dend": date_end}, con=conn)
drivers = df.values.tolist()
for d in drivers:
driverDF = pd.read_sql_query("SELECT * FROM DriverPerf WHERE drivername = :driver AND date BETWEEN :dstart and :dend", params={"driver": d, "dstart": date_start, "dend": date_end}, con=conn)
I've tried a few different versions of the "WHERE drivername" part but it always seems to fail.
Thanks!
If I'm not mistaken, drivers will be a list of lists. Have you tried
.... params={"driver": d[0] ....
While loading the data from csv file into oracle db using python code. After loading data into oracle db True value is displaying as 1 and false value is displaying as 0
Iam using following python code:
import cx_Oracle as orcCon
from cx_Oracle import DatabaseError
import pandas as pd
import csv
csv_input=pd.read_csv(r"E:\Python_projects\test\python1.csv",index_col=False,na_values=" ").fillna('')
#csv_input.head()
#print(csv_input)
try:
conn = orcCon.connect('scott/tiger#localhost:1522/orcl',encoding="UTF-8")
if conn:
print("cx_Oracle version:", orcCon.version)
print("Database version:", conn.version)
print("Client version:", orcCon.clientversion())
# Now execute the sqlquery
cursor = conn.cursor()
print("You're connected.................")
print("TRUNCATING THE TARGET TABLE")
cursor.execute("TRUNCATE TABLE PYTHON_TESTING")
print("Inserting data into table")
for i,row in csv_input.iterrows():
sql = "INSERT INTO PYTHON_TESTING(C1,C2,C3)VALUES(:1,:2,:3)"
cursor.execute(sql, tuple(row))
# the connection is not autocommitted by default, so we must commit to save our changes
conn.commit()
#print("Record inserted successfullly")
except DatabaseError as e:
err, = e.args
print("Oracle-Error-Code:", err.code)
print("Oracle-Error-Message:", err.message)
finally:
cursor.close()
conn.close()
I have created testing csv file as:
When data loaded into Oracle Table Data is displaying as:
Iam using python 3.9.2
You are converting a Python bool data type to an (Oracle dialect) SQL VARCHAR2 data type. In SQL, Oracle does not have a concept of a boolean so there must be some mapping from one data type to another and the typical mapping use is True => 1 and False => 0 (if, in Python you try True == 1 then the output is True but if you try True == 'True' then the output is False).
So your Python code is mapping bool to an integer and passing it to Oracle and then Oracle is making an implicit cast from the number to a string.
You need to break the chain of implicit casts and make an explicit cast to the string values you are expecting.
One way would be to map the numeric truthy values to string literals and can use:
INSERT INTO PYTHON_TESTING(C1,C2,C3)
VALUES(
CASE :1 WHEN 1 THEN 'TRUE' ELSE 'FALSE' END,
CASE :2 WHEN 1 THEN 'TRUE' ELSE 'FALSE' END,
CASE :3 WHEN 1 THEN 'TRUE' ELSE 'FALSE' END
);
Another way would be, in Python, to covert your tuple values from bool to strings; something like:
cursor.execute(sql, tuple('TRUE' if col else 'FALSE' for col in row))
I am getting the error while trying to insert values.
This is for storing userid and password of users in sqlite3 database.
import sqlite3, hashlib
conn = sqlite3.connect('user.db')
a = input("Enter id:")
b = input("Enter password:")
b = str.encode(b)
b = hashlib.sha224(b).hexdigest()
conn.execute("insert into user(id,pass) values(%s, %s);"%(a,b))
print("Done")
Create statement was:
CREATE TABLE user(id varchar[50], pass varchar[50])
When I try to enter values A(id) and a (password), I get the following error:
conn.execute("insert into user(id,pass) values(%s, %s);"%(a,b))
sqlite3.OperationalError: no such column: A
You must use placeholders for the parameters.
So your code will become:
conn.execute("insert into user(id,pass) values(? , ?);" , (a,b))
Another solution that is bad practice, because of the high risk of SQL injection is:
conn.execute("insert into user(id,pass) values('%s', '%s');"%(a,b))
SQLite3 Python module documentation:
https://docs.python.org/3/library/sqlite3.html#sqlite3.Cursor.execute
I'm trying to populate the 1st row(i.e. the column names) from an excel sheet to the ms-access database, but it's giving me 'Data type conversion Error(3421)'. Any idea why this is happening ?
from comtypes.client import CreateObject
from xlrd import open_workbook,cellname
import os
from comtypes.gen import Access
access = CreateObject('Access.Application')
DBEngine = access.DBEngine
db = DBEngine.CreateDatabase('test.mdb', Access.DB_LANG_GENERAL)
excel_file = open_workbook('test_excel_file.xlsx')
work_sheet = excel_file.sheet_by_index(0)
db.BeginTrans()
db.Execute("CREATE TABLE MY_TABLE (ID Text)")
for row_index in range(0, 1):
for col_index in range(0, work_sheet.ncols):
cell_value = work_sheet.cell(row_index,col_index).value
db.Execute("ALTER TABLE MY_TABLE ADD COLUMN %s", cell_value)
db.CommitTrans()
db.Close()
Error:
Traceback (most recent call last):
File "My_DB_Code.py", line 21, in <module>
db.Execute("ALTER TABLE MY_TABLE ADD COLUMN %s", cell_value)
_ctypes.COMError: (-2146824867, None, (u'Data type conversion error.', u'DAO.Dat
abase', u'jeterr40.chm', 5003421, None))
A parameterized query allows parameters to be substituted for the values of a column, but not for table and column names themselves. You will need to use string formatting for that, as in
db.Execute("ALTER TABLE MY_TABLE ADD COLUMN [{0}] TEXT(50)".format(cell_value))