I'm running the linux terminal command 'strings' on a file and storing the result, which is a series of readable strings, in a variable. The results are also written to a text file. I then need to upload the results to a database. The problem is that the result often contains ' and " characters in an unpredictable order. These cause an SQL error. I've tried to string.replace with an empty string and a \ escape. I've also tried """ or ''' round the string but neither work as I don't know which type of quotation mark will be first. Any suggestions would be appreciated.
fileName = filePath.rsplit('/', 1)[1]
stream = os.popen('strings ' + filePath)
output = stream.readlines()
file = open(filePath + "/" + fileName + "_StringsResults.txt", "w+")
for o in output:
file.write(str(o))
results += str(o)
file.close()
dsn = 'postgresql://############localhost:########/test?sslmode=disable'
conn = psycopg2.connect(dsn)
with conn.cursor() as cur:
cur.execute(
"UPSERT INTO test (testID) VALUES ('%s')" % (results))
conn.commit()
yes that worked, thanks a million. For anyone else who's interested the solution was roughly:
query = """UPSERT INTO test (testID) VALUES (%s)"""
#Connection code etc.
with conn.cursor() as cur:
cur.execute(query, [results])
conn.commit()
The [] round the parameter was necessary to avoid a type error.
Related
I want to generate a string for updating values in SQL, from a python dict.
So far, I have the following:
def sqlquote(value):
"""Naive SQL quoting
All values except NULL are returned as SQL strings in single quotes,
with any embedded quotes doubled.
"""
if value is None:
return 'NULL'
return '"{}"'.format(str(value).replace("'", "''"))
part1="UPDATE products_list SET"
#This parts prints with the desired format, but I don't know how to create a string with this:
part2:
for key, value in article_dict.items():
print(key, '=', sqlquote(value))
part3="WHERE product_id=article_dict['product_id']
How can I "concat" the part2 of the string to make a unique string that can be used later to update a sqlite database?
The SQL you're producing isn't valid, since you're missing spaces between the different parts. Also, there's no need for the sqlquote function, it's better to use bound parameters:
values = []
sql = "UPDATE products_list SET "
# For each item in the dictionary, add to the SQL string
temp = []
for key, value in article_dict.items():
if key != 'product_id':
temp.append(f"{key}=?")
# And store the value in a seperate list
values.append(value)
# Concat the different set instructions
sql += ",".join(temp) + " "
# And add a WHERE clause to the end
sql += "WHERE product_id=?;"
values.append(article_dict['product_id'])
# TODO:
# db.execute(sql, values)
Update:
Here is the code that got me what I needed after using Achampion's resolution:
# make a list generated from a previous operation (dynamic list), eg. listA - listB = dynLst
dynLst
['R10YW', 'R13YW', 'R32YWBY','NOSVCYW']
# construct item request for a SQL statement using dynLst, this is Achampion's code
reqSQL = '(' + ', '.join([r"'{}'".format(x) for x in dynLst]) + ')'
# specify query field from table:
f1 = '\"Product_Code\"'
WC = f1 + ' IN ' + reqSQL
WC
'\"Product_Code\" IN (\'R10YW\', \'R13YW\', \'R32YWBY\',\'NOSVCYW\')'
I have a list composed of derived values from a prior python operation. The items in this list will change every time the previous python command runs. Therefor I need a dynamic way to format the list. I'm using it with the 'IN' operand as part of a SQL statement/where clause being used in a subsequent python function.
In order to build the where clause the right way I need it formatted like this:
field = '\"Product_Code\"'
qValues = '(\'R10YW\', \'R13YW\', \'R32YWBY\',\'NOSVCYW\')'
whereClause = field + ' IN ' + qValues
I've tried this:
qValues = '(\\'
for val in dynLst: # dynLst = a dynamic List
val = val + '\\'
qValues = qValues + val + "," + '\\'
qValues = qValues[:-1]
qValues = qValues + ')
But the format I get is this:
'(\\R10YW\\, \\R13YW\\, \\R32YWBY\\,\\NOSVCYW\\)'
I've also tried raw_string('(\') which I think is the python3 way of doing r"string\string", but no luck. I just get a syntax error.
Are you sure you are not just confusing the escaped output.
It looks like you just need a string with ' in them, e.g.:
In []:
dynlist = ['R10YW', 'R13YW', 'R32YWBY', 'NOSVCYW']
'(' + ', '.join(["'{}'".format(x) for x in dynlist]) + ')'
Out[]:
"('R10YW', 'R13YW', 'R32YWBY', 'NOSVCYW')"
And if you really need a \ in the output then you will get a double \\ because that is how Python shows a single \:
In []:
'(' + ', '.join([r"\'{}\'".format(x) for x in dynlist]) + ')'
Out[]:
"(\\'R10YW\\', \\'R13YW\\', \\'R32YWBY\\', \\'NOSVCYW\\')"
Your code works (sort of,) in that it actually produces the output you want. You should never make an SQL query this way however as it opens your code up to SQL injection attacks (I assumed SQLite other DBs have the same vulnerability and some method to avoid it.)
https://www.tutlane.com/tutorial/sqlite/sqlite-injection-attacks
https://sebastianraschka.com/Articles/2014_sqlite_in_python_tutorial.html#security-and-injection-attacks
That being said if you use the python3 interpreter as an interactive shell (or ipython or Jupyter,) when you have a string that is the result of an operation it prints it formatted for a programmer:
In [10]: s = "\\hello"
In [11]: s
Out[11]: '\\hello'
but if you print that same string:
In [12]: print(s)
\hello
It's just as you wanted it to be. I tried your code and other than missing a ' at the end of the last line it works as you seem to want it to.
edit: Redid the slashes as the site was also escaping things. The example now looks just like it will come out of ipython.
I'm trying to insert the value of the variable test_text into a Postgres 9.6 database, each time the database_insert function is triggered.
I'm using Python 3.6 and psycopg2 v 2.7
If I use the below code without the placeholder: e.g replace %s with 'test' and remove , (test_text) - it works as I would expect...
def database_insert(update):
test_text = 'This is some test text'
with psycopg2.connect("DB CONNECTION DETAILS ARE HERE'") as conn:
cur = conn.cursor()
cur.execute("INSERT INTO test VALUES(%s);", (test_text))
conn.commit()
cur.close()
conn.close()
However when the function trys to insert the value of the test_text variable using the %s placeholder, I get the error below...
cur.execute("INSERT INTO test VALUES(%s);", (test_text))
TypeError: not all arguments converted during string formatting
Any help on where I am going wrong with this will be much appreciated!
There's a subtle issue here.
You need a comma to make a tuple not just the parens/brackets.
So simply change to:
cur.execute("INSERT INTO test VALUES(%s);", (test_text,))
And you should be good!
I have this code that simply creates a list from user input. I want to load this into sqlite Db instead of list shown but am not conversant with Sqlite. please help
HERE IS THE CODE
listQ = []
while True:
read = input("Type in a line. ").lower().split()
for item in read:
listQ.append( input("Type in a line. ") )
for line in listQ:
import sqlite3
conn = sqlite3.connect('/C/project/new/sqlite_file.db')
c = conn.cursor()
for item in listQ:
c.execute('insert into tablename values (?,?,?)', item)
#print(line)
I have small code like below :
#!/usr/bin/python
import psycopg2, sys
try:
conn = psycopg2.connect("dbname='smdr' user='bino'");
except:
print "I am unable to connect to the database"
cur = conn.cursor()
v_num = '1'
cur.execute("SELECT * from genctr WHERE code = %(num)s", dict(num=v_num))
rows = cur.fetchall()
print "\nShow me the databases:\n"
ctrnum =0
for row in rows:
print row[0]+","+row[1]
when i run it, i got
bino#erp:~/mydoc/openerp/smdr$ ./genctr.py
Show me the databases:
1,Bahamas
1,Barbados
1,Canada
1,Cayman Islands
1,United States
1,Virgin Islands U.S.
I try to replace "v_num = '1' " with "v_num = sys.stdin.read()"
#!/usr/bin/python
import psycopg2, sys
try:
conn = psycopg2.connect("dbname='smdr' user='bino'");
except:
print "I am unable to connect to the database"
cur = conn.cursor()
#v_num = '1'
v_num = sys.stdin.read()
cur.execute("SELECT * from genctr WHERE code = %(num)s", dict(num=v_num))
rows = cur.fetchall()
print "\nShow me the databases:\n"
ctrnum =0
for row in rows:
print row[0]+","+row[1]
But when I run it , I only got this :
bino#erp:~/mydoc/openerp/smdr$ echo 1 |./genctr.py
Show me the databases:
Kindly please give me your enlightment on how to fix it
Sincerely
-bino-
echo 1 is going to give "1\n" to your program (that is, "1" with a newline character afterward). sys.stdin.read() is going to return that exact string, and then psycopg2 is going to prepare the SQL statement as SELECT * from genctr WHERE code = '1\n'. The result is going to be no matching results, so the code inside the for loop will never execute, which is why you don't see any extra output.
Try doing either echo -n 1 to suppress the newline, or sys.stdin.read().strip() to remove any leading and trailing whitespace from the string. If the code field is an integer, it might be a good idea to cast the result of sys.stdin.read() to an int, too, like so:
int(sys.stdin.read().strip())