I'm using the pg_trgm for similarity search on PostgreSQL DB and I need to return the results to the PostGIS table, but I'm getting this programmer error, I learned that this error is related to syntax of the sql query I tried same query in PostgreSQL and it worked, but couldn't get it to working with python. what I use (Windows 10, Python 3.8, PostgreSQL 12.6)
class OSMGeocoder(object):
""" A class to provide geocoding features using an OSM dataset in PostGIS."""
def __init__(self, db_connectionstring):
#db initialize db connection parameters
self.db_connectionstring = db_connectionstring
def geocode(self, placename):
""" Geocode a given place name."""
# here we crete the connection object
conn = psycopg2.connect(self.db_connectionstring)
cur = conn.cursor()
#this is the core sql query, using triagrams to detect streets similar to a given placename
#here we execute the sql and return all of the results
cur.execute("SELECT name, name <-> '%s' AS weight, ST_AsText(ST_Centroid(wkb_geometry)) AS point FROM public.osm_roads ORDER BY weight LIMIT 10;" % placename)
rows = cur.fetchall()
conn.commit()
cur.close()
conn.close()
return rows
if __name__ =='__main__':
# the user must provide at least two parameters, the place name and the connection string to PostGIS
if len(sys.argv) < 3 or len(sys.argv) > 3:
print("usage: <placename> <connection string>")
raise SystemExit
placename = sys.argv[1]
db_connectionstring = sys.argv[2]
#here we instantiate the geocoder, providing the needed PostGIS connection parameters
geocoder = OSMGeocoder(db_connectionstring)
#here we query the geocode methiod, for getting the geocoded points for the given placenames
results = geocoder.geocode(placename)
print(results)
ProgrammingError: invalid dsn: missing "=" after "C:\Users\Lenovo\AppData\Roaming\jupyter\runtime\kernel-ee3068bc-0b95-4bba-a373-752c8196980f.json" in connection info string
I have python code in which I use print statements to print "PASS: " and "FAIL: " pattern depending on certain conditions inside the python code. When I run my test case using Robot Framework it always declares the test case to be a PASS as the python code runs successfully. The output of the test case is saved automatically to robot framework log.html file. I see all my print statements from my python code in this log file.
What I want to achieve is, declare the robot framework test case to be a PASS or FAIL based on the pattern in log.html file. Is there a way I can search the log.html file and declare my test case to be a pass or fail accordingly? or some other solution?
Robot Framework Calling Python Function:
*** Test Cases ***
Establish PDX Connection
[documentation] Check PDX database connection
Connection Request PDX Database ${Var 1} ${Var 2} ${Var 3} ${Var 4}
Python Function: (this defines the steps for test case)
def connection_request_pdx_database(server_name, database_name, trusted_connection, query):
pdx_server = TH_SQLConn()
""" Send connection request to PDX Server """
connection_request = pdx_server.connect(server_name, database_name, trusted_connection)
""" Run SQL queries """
query_pdx_server = pdx_server.query(connection_request, query)
""" Close Connection """
pdx_server.close(query_pdx_server, connection_request)
Class Definition:
import sys
import pyodbc
class TH_SQLConn:
def __init__(self):
self.conn_established = True
def connect(self, server_name, database_name, trusted_connection='yes'):
try:
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server}; \
SERVER= ' + server_name + '; \
DATABASE=' + database_name + '; \
Trusted_Connection=' + trusted_connection + ';')
if conn is not None:
print("PASS: Connection to SQL Database is established successfully")
return conn
else:
print("FAIL: Connection to SQL Database is NOT successful. Check login credentials.")
self.conn_established = False
# sys.exit()
except TypeError:
print(sys.exc_info()[0])
print("FAIL: Connection to SQL Database is NOT successful. Check datatype of parameters passed.")
self.conn_established = False
# sys.exit()
except pyodbc.OperationalError:
print(sys.exc_info()[0])
print("FAIL: Connection to SQL Database is NOT successful. Unable to complete login process due to delay in opening server connection.")
self.conn_established = False
# sys.exit()
except:
print(sys.exc_info()[0])
print("FAIL: Connection to SQL Database is NOT successful.")
self.conn_established = False
# sys.exit()
def query(self, conn, sql_stmt):
"""
Create the connection cursor
Read data from the SQL database table
"""
if self.conn_established is True:
try:
cursor = conn.cursor()
cursor.execute(sql_stmt)
# Loop through the result of SQL query
for row in cursor:
print(row)
return cursor
except AttributeError:
print(sys.exc_info()[0])
print("FAIL: Required parameters to run SQL query method are not available")
self.conn_established = False
# sys.exit()
except pyodbc.ProgrammingError:
print(sys.exc_info()[0])
print("FAIL: Either the user tried to run a SQL query without establishing connection to SQL server database first.\n OR Check for SQL statement syntax error, an error is found in SQL query.")
self.conn_established = False
# sys.exit()
else:
pass
def close(self, cursor, conn):
"""
Remove the cursor and close the connection
"""
if self.conn_established is True:
try:
cursor.close()
conn.close()
# Query the database for a list of all databases on SQL server to check if connection is successfully closed
cursor.execute("SELECT * FROM master.sys.databases")
except AttributeError:
print(sys.exc_info()[0])
print("FAIL: Required parameters to run SQL connection close method are not available")
self.conn_established = False
# sys.exit()
except TypeError:
print(sys.exc_info()[0])
print("FAIL: Required parameters to run SQL connection close method are not available")
self.conn_established = False
# sys.exit()
except pyodbc.ProgrammingError:
print("PASS: The connection to SQL Database is closed successfully")
except:
print(sys.exc_info()[0])
print("FAIL: The connection to SQL Database is NOT closed successfully")
self.conn_established = False
# sys.exit()
else:
pass
I am creating a dictionary attacking tool on PostgreSQL. The tool is inspired by the work of m8r0wn - enumdb tool. Mikes tool is aimed at MySQL and MSSQL. I aim to use the same approach he used but modify the actions and output file. The script should
1) Read a CSV file that contains targets and ports, one per line 127.0.0.1,3380.
2) when provided a list of usernames and/or passwords, it will cycle through each targeted host looking for valid credentials. By default, it will use newly discovered credentials to search for sensitive information in the host's databases via keyword searches on the table or column names.
3) This information can then be extracted and reported to a JSON, .csv or .xlsx output file.
I have a semi functional code, but I suspect the PostgreSQL connection function is not working due to the logic behind passing parameters. I am interested in suggestions on how best I could present the tools results as a JSON file.
I understand that in Python, we have several modules available to connect and work with PostgreSQL which include:
Psycopg2
pg8000
py-postgresql
PyGreSQL
ocpgdb
bpgsql
SQLAlchemy
see also https://www.a2hosting.co.za/kb/developer-corner/postgresql/connecting-to-postgresql-using-python
The connection methods I have tried include:
import psycopg2
from psycopg2 import Error
conn = psycopg2.connect(host=host, dbname=db_name, user=_user, password=_pass, port=port)
import pg
conn = pg.DB(host=args.hostname, user= _user, passwd= _pass)
sudo pip install pgdb
import pgdb
conn = pgdb.connect(host=args.hostname, user= _user, passwd= _pass)
I am not sure how to pass the different _user and _pass guesses into the pyscopg2 for instance, without breaking the code.
I have imported the following libraries
import re
import psycopg2
from psycopg2 import Error
import pgdb
#import MySQLdb
import pymssql
import argparse
from time import sleep
from sys import exit, argv
from getpass import getpass
from os import path, remove
from openpyxl import Workbook
from threading import Thread, activeCount
The PgSQL block is as follows:
##########################################
# PgSQL DB Class
##########################################
class pgsql():
def connect(self, host, port, user, passwd, verbose):
try:
con = pgdb.connect(host=host, port=port, user=user, password=passwd, connect_timeout=3)
con.query_timeout = 15
print_success("[*] Connection established {}:{}#{}".format(user,passwd,host))
return con
except Exception as e:
if verbose:
print_failure("[!] Login failed {}:{}#{}\t({})".format(user,passwd,host,e))
else:
print_failure("[!] Login failed {}:{}#{}".format(user, passwd, host))
return False
def db_query(self, con, cmd):
try:
cur = con.cursor()
cur.execute(cmd)
data = cur.fetchall()
cur.close()
except:
data = ''
return data
def get_databases(self, con):
databases = []
for x in self.db_query(con, 'SHOW DATABASES;'):
databases.append(x[0])
return databases
def get_tables(self, con, database):
tables = []
self.db_query(con, "USE {}".format(database))
for x in self.db_query(con, 'SHOW TABLES;'):
tables.append(x[0])
return tables
def get_columns(self, con, database, table):
# database var not used but kept to support mssql
columns = []
for x in self.db_query(con, 'SHOW COLUMNS FROM {}'.format(table)):
columns.append(x[0])
return columns
def get_data(self, con, database, table):
# database var not used but kept to support mssql
return self.db_query(con, 'SELECT * FROM {} LIMIT {}'.format(table, SELECT_LIMIT))
The MSSQL is as follows:
# MSSQL DB Class
class mssql():
def connect(self, host, port, user, passwd, verbose):
try:
con = pymssql.connect(server=host, port=port, user=user, password=passwd, login_timeout=3, timeout=15)
print_success("[*] Connection established {}:{}#{}".format(user,passwd,host))
return con
except Exception as e:
if verbose:
print_failure("[!] Login failed {}:{}#{}\t({})".format(user,passwd,host,e))
else:
print_failure("[!] Login failed {}:{}#{}".format(user, passwd, host))
return False
def db_query(self, con, cmd):
try:
cur = con.cursor()
cur.execute(cmd)
data = cur.fetchall()
cur.close()
except:
data = ''
return data
def get_databases(self, con):
databases = []
for x in self.db_query(con, 'SELECT NAME FROM sys.Databases;'):
databases.append(x[0])
return databases
def get_tables(self, con, database):
tables = []
for x in self.db_query(con, 'SELECT NAME FROM {}.sys.tables;'.format(database)):
tables.append(x[0])
return tables
def get_columns(self, con, database, table):
columns = []
for x in self.db_query(con, 'USE {};SELECT column_name FROM information_schema.columns WHERE table_name = \'{}\';'.format(database, table)):
columns.append(x[0])
return columns
def get_data(self, con, database, table):
return self.db_query(con, 'SELECT TOP({}) * FROM {}.dbo.{};'.format(SELECT_LIMIT, database, table))
The main function block:
def main(args):
try:
for t in args.target:
x = Thread(target=enum_db().db_main, args=(args, t,))
x.daemon = True
x.start()
# Do not exceed max threads
while activeCount() > args.max_threads:
sleep(0.001)
# Exit all threads before closing
while activeCount() > 1:
sleep(0.001)
except KeyboardInterrupt:
print("\n[!] Key Event Detected...\n\n")
exit(0)
if __name__ == '__main__':
version = '1.0.7'
try:
args = argparse.ArgumentParser(description=("""
{0} (v{1})
--------------------------------------------------
Brute force Juggernaut is a PgSQL brute forcing tool.**""").format(argv[0], version), formatter_class=argparse.RawTextHelpFormatter, usage=argparse.SUPPRESS)
user = args.add_mutually_exclusive_group(required=True)
user.add_argument('-u', dest='users', type=str, action='append', help='Single username')
user.add_argument('-U', dest='users', default=False, type=lambda x: file_exists(args, x), help='Users.txt file')
passwd = args.add_mutually_exclusive_group()
passwd.add_argument('-p', dest='passwords', action='append', default=[], help='Single password')
passwd.add_argument('-P', dest='passwords', default=False, type=lambda x: file_exists(args, x), help='Password.txt file')
args.add_argument('-threads', dest='max_threads', type=int, default=3, help='Max threads (Default: 3)')
args.add_argument('-port', dest='port', type=int, default=0, help='Specify non-standard port')
args.add_argument('-r', '-report', dest='report', type=str, default=False, help='Output Report: csv, xlsx (Default: None)')
args.add_argument('-t', dest='dbtype', type=str, required=True, help='Database types currently supported: mssql, pgsql')
args.add_argument('-c', '-columns', dest="column_search", action='store_true', help="Search for key words in column names (Default: table names)")
args.add_argument('-v', dest="verbose", action='store_true', help="Show failed login notices & keyword matches with Empty data sets")
args.add_argument('-brute', dest="brute", action='store_true', help='Brute force only, do not enumerate')
args.add_argument(dest='target', nargs='+', help='Target database server(s)')
args = args.parse_args()
# Put target input into an array
args.target = list_targets(args.target[0])
# Get Password if not provided
if not args.passwords:
args.passwords = [getpass("Enter password, or continue with null-value: ")]
# Define default port based on dbtype
if args.port == 0: args.port = default_port(args.dbtype)
# Launch Main
print("\nStarting enumdb v{}\n".format(version) + "-" * 25)
main(args)
except KeyboardInterrupt:
print("\n[!] Key Event Detected...\n\n")
exit(0)
I am aware that documentation states here http://initd.org/psycopg/docs/module.html states about how connection parameters can be specified. I would like to pass password guesses into the brute class and recursively try different combinations.
PEP-8 asks that you please give classes a name
starting with a capital letter, e.g. Pgsql.
You mentioned that the pgsql connect() method is not working properly,
but didn't offer any diagnostics such as a stack trace.
You seem to be working too hard, given that the sqlalchemy layer
has already addressed the DB porting issue quite nicely.
Just assemble a connect string starting with
the name of the appropriate DB package,
and let sqlalchemy take care of the rest.
All your methods accept con as an argument.
You really want to factor that out as the object attribute self.con.
The db_query() method apparently assumes that
arguments for WHERE clauses already appear, properly quoted, in cmd.
According to Little Bobby's mother,
it makes sense to accept query args according to the API,
rather than worrying about potential for SQL injection.
This program is a banking system. It connects to an online database which contains customer details and transaction details. However, when I execute the code, I get a blank output in python 3.4.0 shell:
import pyodbc
cnxn = pyodbc.connect('Driver={SQL Server};'
'Server=***;'
'Database=***;'
'uid=***;pwd=***')
cursor = cnxn.cursor()
def MainMenu():
print('##############################\n\tWelcome to the XYZ Banking System\n##############################')
print()
print('PLEASE ENTER THE NUMBER CORRESPONDING TO YOUR DESIRED COMMAND IN THE PROMPT BELOW : \n\t1.ACCESS CUSTOMER DETAILS\n\t2.ACCESS TRANSACTION PORTAL\n##############################')
print()
var_UserInput=input('>>>')
if var_UserInput=='1':
return CustomerPortal()
def CustomerPortal():
cursor.tables()
rows = cursor.fetchall()
for row in rows:
print (row.customer)
MainMenu()
Try this. I've made a few changes:
Moved the connection string into the function
Modified the code to be closer to PEP-8 https://www.python.org/dev/peps/pep-0008/
Fixed indentation
Here's the code.
import pyodbc
def main_menu():
print('##############################\n\tWelcome to the XYZ Banking System\n##############################')
print()
print('PLEASE ENTER THE NUMBER CORRESPONDING TO YOUR DESIRED COMMAND IN THE PROMPT BELOW : \n\t1.ACCESS CUSTOMER DETAILS\n\t2.ACCESS TRANSACTION PORTAL\n##############################')
print()
var_user_input=input('>>>')
if var_user_input=='1':
return customer_portal()
def customer_portal():
cnxn = pyodbc.connect('Driver={SQL Server};'
'Server=***;'
'Database=***;'
'uid=***;pwd=***')
cursor = cnxn.cursor()
cursor.tables()
rows = cursor.fetchall()
for row in rows:
print (row.customer)
cursor.close()
if __name__ == "__main__":
main_menu()
Good luck!
I want to put a simple csv into a MySQL table in a database. I know there are probably other ways to do this but for the system I have in place, I would like to stick with this method if possible.
I have read every other possible post about this and tried many of the suggestions but I can't get it to work. I can get it to work with python 2 but not in 3.
csv is
test1 test2
43.49 654.32
78.89 294.95
I can't seem to get the syntax correct and get various errors when I try different approaches.
import csv
import pymysql
db = pymysql.connect( host = '###.###.##.#',
user = 'test',passwd = 'test')
cursor = db.cursor()
csv_data = csv.reader('c:/tmp/sample.csv')
next(csv_data)
for row in csv_data:
cursor.execute('INSERT INTO test.table(test1, test2) VALUES(%s, %s)',row)
db.commit()
cursor.close()
print (Imported)
Any ideas???
Thanks in advance!!
For me it worked
import pymysql
import csv
db = pymysql.connect("localhost","root","12345678","data" )
cursor = db.cursor()
csv_data = csv.reader(open('test.csv'))
next(csv_data)
for row in csv_data:
cursor.execute('INSERT INTO PM(col1,col2) VALUES(%s, %s)',row)
db.commit()
cursor.close()
To answer my own question, this worked for me:
import csv
import pymysql
db = pymysql.connect( host = '###.###.##.#',
user = 'test',passwd = 'test')
cursor = db.cursor()
f = csv.reader(open('c:/tmp/sample.csv'))
sql = """INSERT INTO test.table(test1,test2) VALUES(%s,%s)"""
next(f)
for line in f:
line=[None if cell == '' else cell for cell in line]
cursor.execute(sql, line)
db.commit()
cursor.close()
print ("Imported")