how to use pg_trgm operators e.g (<-> ) in python - python-3.x

I'm using the pg_trgm for similarity search on PostgreSQL DB and I need to return the results to the PostGIS table, but I'm getting this programmer error, I learned that this error is related to syntax of the sql query I tried same query in PostgreSQL and it worked, but couldn't get it to working with python. what I use (Windows 10, Python 3.8, PostgreSQL 12.6)
class OSMGeocoder(object):
""" A class to provide geocoding features using an OSM dataset in PostGIS."""
def __init__(self, db_connectionstring):
#db initialize db connection parameters
self.db_connectionstring = db_connectionstring
def geocode(self, placename):
""" Geocode a given place name."""
# here we crete the connection object
conn = psycopg2.connect(self.db_connectionstring)
cur = conn.cursor()
#this is the core sql query, using triagrams to detect streets similar to a given placename
#here we execute the sql and return all of the results
cur.execute("SELECT name, name <-> '%s' AS weight, ST_AsText(ST_Centroid(wkb_geometry)) AS point FROM public.osm_roads ORDER BY weight LIMIT 10;" % placename)
rows = cur.fetchall()
conn.commit()
cur.close()
conn.close()
return rows
if __name__ =='__main__':
# the user must provide at least two parameters, the place name and the connection string to PostGIS
if len(sys.argv) < 3 or len(sys.argv) > 3:
print("usage: <placename> <connection string>")
raise SystemExit
placename = sys.argv[1]
db_connectionstring = sys.argv[2]
#here we instantiate the geocoder, providing the needed PostGIS connection parameters
geocoder = OSMGeocoder(db_connectionstring)
#here we query the geocode methiod, for getting the geocoded points for the given placenames
results = geocoder.geocode(placename)
print(results)
ProgrammingError: invalid dsn: missing "=" after "C:\Users\Lenovo\AppData\Roaming\jupyter\runtime\kernel-ee3068bc-0b95-4bba-a373-752c8196980f.json" in connection info string

Related

How to do validation in python?

I am making a gui for employee management system using python tkinter and sqlite3.
In this gui user can add, view, delete amd update employee info.
def save():
con = None
try:
con = connect("pro.db")
cursor = con.cursor()
sql = "insert into Employee values('%d', '%s', '%f')"
id = int(aw_ent_id.get())
name = aw_ent_name.get()
lenstring = False
while not lenstring:
if len(name) >= 2:
lenstring = True
else:
showerror("error","Enter atleast 2 letters")
break
salary = float(aw_ent_salary.get())
cursor.execute(sql%(id, name, salary))
con.commit()
showinfo("success", "record added")
aw_ent_id.delete(0, END)
aw_ent_name.delete(0, END)
aw_ent_salary.delete(0, END)
aw_ent_id.focus()
except Exception as e:
con.rollback()
showerror("issue", e)
finally:
if con is not None:
con.close()
the code is running but i am getting some errors in validating name and salary.
for name i have done validating but its not working. I am getting an error
the data is getting saved even after getting error.
What should i do to make it right?
It is better to:
validate the inputs before saving to database
raise exception if len(name) is less than 2 instead of using while loop checking (actually the while loop is meaningless)
use placeholders to avoid SQL injection
Below is updated save():
# avoid using wildcard import
import tkinter as tk
from tkinter.messagebox import showinfo, showerror
import sqlite3
...
def save():
con = None
try:
# validate inputs
# use emp_id instead of id because id is built-in function
emp_id = int(aw_ent_id.get().strip()) # raise ValueError if not a valid integer
name = aw_ent_name.get().strip()
if len(name) < 2:
raise Exception('Name too short, at least 2 letters')
salary = float(aw_ent_salary.get().strip()) # raise ValueError if not a valid float number
# validations on inputs are passed, try saving to database
sql = 'insert into Employee values (?, ?, ?)' # use placeholders to avoid SQL injection
con = sqlite3.connect('pro.db')
cursor = con.cursor()
cursor.execute(sql, (emp_id, name, salary))
con.commit()
showinfo('Success', 'Employee added')
aw_ent_id.delete(0, tk.END)
aw_ent_name.delete(0, tk.END)
aw_ent_salary.delete(0, tk.END)
aw_ent_id.focus_set()
except Exception as e:
if con:
con.rollback()
showerror("Error", e)
finally:
if con:
con.close()
...

Can I search log.html file for specific keywords using Robot Framework test case?

I have python code in which I use print statements to print "PASS: " and "FAIL: " pattern depending on certain conditions inside the python code. When I run my test case using Robot Framework it always declares the test case to be a PASS as the python code runs successfully. The output of the test case is saved automatically to robot framework log.html file. I see all my print statements from my python code in this log file.
What I want to achieve is, declare the robot framework test case to be a PASS or FAIL based on the pattern in log.html file. Is there a way I can search the log.html file and declare my test case to be a pass or fail accordingly? or some other solution?
Robot Framework Calling Python Function:
*** Test Cases ***
Establish PDX Connection
[documentation] Check PDX database connection
Connection Request PDX Database ${Var 1} ${Var 2} ${Var 3} ${Var 4}
Python Function: (this defines the steps for test case)
def connection_request_pdx_database(server_name, database_name, trusted_connection, query):
pdx_server = TH_SQLConn()
""" Send connection request to PDX Server """
connection_request = pdx_server.connect(server_name, database_name, trusted_connection)
""" Run SQL queries """
query_pdx_server = pdx_server.query(connection_request, query)
""" Close Connection """
pdx_server.close(query_pdx_server, connection_request)
Class Definition:
import sys
import pyodbc
class TH_SQLConn:
def __init__(self):
self.conn_established = True
def connect(self, server_name, database_name, trusted_connection='yes'):
try:
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server}; \
SERVER= ' + server_name + '; \
DATABASE=' + database_name + '; \
Trusted_Connection=' + trusted_connection + ';')
if conn is not None:
print("PASS: Connection to SQL Database is established successfully")
return conn
else:
print("FAIL: Connection to SQL Database is NOT successful. Check login credentials.")
self.conn_established = False
# sys.exit()
except TypeError:
print(sys.exc_info()[0])
print("FAIL: Connection to SQL Database is NOT successful. Check datatype of parameters passed.")
self.conn_established = False
# sys.exit()
except pyodbc.OperationalError:
print(sys.exc_info()[0])
print("FAIL: Connection to SQL Database is NOT successful. Unable to complete login process due to delay in opening server connection.")
self.conn_established = False
# sys.exit()
except:
print(sys.exc_info()[0])
print("FAIL: Connection to SQL Database is NOT successful.")
self.conn_established = False
# sys.exit()
def query(self, conn, sql_stmt):
"""
Create the connection cursor
Read data from the SQL database table
"""
if self.conn_established is True:
try:
cursor = conn.cursor()
cursor.execute(sql_stmt)
# Loop through the result of SQL query
for row in cursor:
print(row)
return cursor
except AttributeError:
print(sys.exc_info()[0])
print("FAIL: Required parameters to run SQL query method are not available")
self.conn_established = False
# sys.exit()
except pyodbc.ProgrammingError:
print(sys.exc_info()[0])
print("FAIL: Either the user tried to run a SQL query without establishing connection to SQL server database first.\n OR Check for SQL statement syntax error, an error is found in SQL query.")
self.conn_established = False
# sys.exit()
else:
pass
def close(self, cursor, conn):
"""
Remove the cursor and close the connection
"""
if self.conn_established is True:
try:
cursor.close()
conn.close()
# Query the database for a list of all databases on SQL server to check if connection is successfully closed
cursor.execute("SELECT * FROM master.sys.databases")
except AttributeError:
print(sys.exc_info()[0])
print("FAIL: Required parameters to run SQL connection close method are not available")
self.conn_established = False
# sys.exit()
except TypeError:
print(sys.exc_info()[0])
print("FAIL: Required parameters to run SQL connection close method are not available")
self.conn_established = False
# sys.exit()
except pyodbc.ProgrammingError:
print("PASS: The connection to SQL Database is closed successfully")
except:
print(sys.exc_info()[0])
print("FAIL: The connection to SQL Database is NOT closed successfully")
self.conn_established = False
# sys.exit()
else:
pass

Using same connection with multiple resolvers in graphene

I have a code like this,
# SWAMI KARUPPASWAMI THUNNAI
import jwt
import graphene
from flask import request
from auth.helper import medease_token
from database.get_connection import get_connection
from flask_graphql import GraphQLView
class CredentialInformation(graphene.ObjectType):
"""
graphene object type to get the personal information about the user
"""
country_code = graphene.String()
phone = graphene.String()
verified = graphene.Int()
#medease_token
def resolve_country_code(self, root):
customer_token = request.headers["x-access-token"]
decoded_token = jwt.decode(customer_token, verify=False)
customer_id = decoded_token["customer_id"]
try:
connection = get_connection()
cursor = connection.cursor()
cursor.execute("select country_code from customer_credential where id=%s limit 1", (customer_id, ))
result = cursor.fetchone()
return result["country_code"]
finally:
cursor.close()
connection.close()
#medease_token
def resolve_phone(self, root):
customer_token = request.headers["x-access-token"]
decoded_token = jwt.decode(customer_token, verify=False)
customer_id = decoded_token["customer_id"]
try:
connection = get_connection()
cursor = connection.cursor()
cursor.execute("select phone from customer_credential where id=%s limit 1", (customer_id, ))
result = cursor.fetchone()
return result["phone"]
finally:
cursor.close()
connection.close()
#medease_token
def resolve_verified(self, root):
customer_token = request.headers["x-access-token"]
decoded_token = jwt.decode(customer_token, verify=False)
customer_id = decoded_token["customer_id"]
try:
connection = get_connection()
cursor = connection.cursor()
cursor.execute("select verified from customer_credential where id=%s limit 1", (customer_id,))
result = cursor.fetchone()
return result["verified"]
finally:
cursor.close()
connection.close()
def credential_information_wrapper():
return GraphQLView.as_view("graphql", schema=graphene.Schema(query=CredentialInformation))
which uses flask-graphql and graphene for Python graphql. The code works absolutely fine but I think I am missing something here because I need to open new connections in every resolver, and I need to write same query again so there is a data duplication, so is this the right way of doing or am I missing something?
Any help would be highly appreciated! Thank you in advance.
Opening a new connection on every query (resolver in this case) is okay. You could also set a connection pool to minimize the cost of opening connections.
Just curious, what database adapter or ORM are you using? Would be great if you could refactor the connection and cursor steps into a single function to call on every resolver and only have to pass the SQL query string.

How to modify the PostgreSQL psycopg to accept variables instead of values

I am creating a dictionary attacking tool on PostgreSQL. The tool is inspired by the work of m8r0wn - enumdb tool. Mikes tool is aimed at MySQL and MSSQL. I aim to use the same approach he used but modify the actions and output file. The script should
1) Read a CSV file that contains targets and ports, one per line 127.0.0.1,3380.
2) when provided a list of usernames and/or passwords, it will cycle through each targeted host looking for valid credentials. By default, it will use newly discovered credentials to search for sensitive information in the host's databases via keyword searches on the table or column names.
3) This information can then be extracted and reported to a JSON, .csv or .xlsx output file.
I have a semi functional code, but I suspect the PostgreSQL connection function is not working due to the logic behind passing parameters. I am interested in suggestions on how best I could present the tools results as a JSON file.
I understand that in Python, we have several modules available to connect and work with PostgreSQL which include:
Psycopg2
pg8000
py-postgresql
PyGreSQL
ocpgdb
bpgsql
SQLAlchemy
see also https://www.a2hosting.co.za/kb/developer-corner/postgresql/connecting-to-postgresql-using-python
The connection methods I have tried include:
import psycopg2
from psycopg2 import Error
conn = psycopg2.connect(host=host, dbname=db_name, user=_user, password=_pass, port=port)
import pg
conn = pg.DB(host=args.hostname, user= _user, passwd= _pass)
sudo pip install pgdb
import pgdb
conn = pgdb.connect(host=args.hostname, user= _user, passwd= _pass)
I am not sure how to pass the different _user and _pass guesses into the pyscopg2 for instance, without breaking the code.
I have imported the following libraries
import re
import psycopg2
from psycopg2 import Error
import pgdb
#import MySQLdb
import pymssql
import argparse
from time import sleep
from sys import exit, argv
from getpass import getpass
from os import path, remove
from openpyxl import Workbook
from threading import Thread, activeCount
The PgSQL block is as follows:
##########################################
# PgSQL DB Class
##########################################
class pgsql():
def connect(self, host, port, user, passwd, verbose):
try:
con = pgdb.connect(host=host, port=port, user=user, password=passwd, connect_timeout=3)
con.query_timeout = 15
print_success("[*] Connection established {}:{}#{}".format(user,passwd,host))
return con
except Exception as e:
if verbose:
print_failure("[!] Login failed {}:{}#{}\t({})".format(user,passwd,host,e))
else:
print_failure("[!] Login failed {}:{}#{}".format(user, passwd, host))
return False
def db_query(self, con, cmd):
try:
cur = con.cursor()
cur.execute(cmd)
data = cur.fetchall()
cur.close()
except:
data = ''
return data
def get_databases(self, con):
databases = []
for x in self.db_query(con, 'SHOW DATABASES;'):
databases.append(x[0])
return databases
def get_tables(self, con, database):
tables = []
self.db_query(con, "USE {}".format(database))
for x in self.db_query(con, 'SHOW TABLES;'):
tables.append(x[0])
return tables
def get_columns(self, con, database, table):
# database var not used but kept to support mssql
columns = []
for x in self.db_query(con, 'SHOW COLUMNS FROM {}'.format(table)):
columns.append(x[0])
return columns
def get_data(self, con, database, table):
# database var not used but kept to support mssql
return self.db_query(con, 'SELECT * FROM {} LIMIT {}'.format(table, SELECT_LIMIT))
The MSSQL is as follows:
# MSSQL DB Class
class mssql():
def connect(self, host, port, user, passwd, verbose):
try:
con = pymssql.connect(server=host, port=port, user=user, password=passwd, login_timeout=3, timeout=15)
print_success("[*] Connection established {}:{}#{}".format(user,passwd,host))
return con
except Exception as e:
if verbose:
print_failure("[!] Login failed {}:{}#{}\t({})".format(user,passwd,host,e))
else:
print_failure("[!] Login failed {}:{}#{}".format(user, passwd, host))
return False
def db_query(self, con, cmd):
try:
cur = con.cursor()
cur.execute(cmd)
data = cur.fetchall()
cur.close()
except:
data = ''
return data
def get_databases(self, con):
databases = []
for x in self.db_query(con, 'SELECT NAME FROM sys.Databases;'):
databases.append(x[0])
return databases
def get_tables(self, con, database):
tables = []
for x in self.db_query(con, 'SELECT NAME FROM {}.sys.tables;'.format(database)):
tables.append(x[0])
return tables
def get_columns(self, con, database, table):
columns = []
for x in self.db_query(con, 'USE {};SELECT column_name FROM information_schema.columns WHERE table_name = \'{}\';'.format(database, table)):
columns.append(x[0])
return columns
def get_data(self, con, database, table):
return self.db_query(con, 'SELECT TOP({}) * FROM {}.dbo.{};'.format(SELECT_LIMIT, database, table))
The main function block:
def main(args):
try:
for t in args.target:
x = Thread(target=enum_db().db_main, args=(args, t,))
x.daemon = True
x.start()
# Do not exceed max threads
while activeCount() > args.max_threads:
sleep(0.001)
# Exit all threads before closing
while activeCount() > 1:
sleep(0.001)
except KeyboardInterrupt:
print("\n[!] Key Event Detected...\n\n")
exit(0)
if __name__ == '__main__':
version = '1.0.7'
try:
args = argparse.ArgumentParser(description=("""
{0} (v{1})
--------------------------------------------------
Brute force Juggernaut is a PgSQL brute forcing tool.**""").format(argv[0], version), formatter_class=argparse.RawTextHelpFormatter, usage=argparse.SUPPRESS)
user = args.add_mutually_exclusive_group(required=True)
user.add_argument('-u', dest='users', type=str, action='append', help='Single username')
user.add_argument('-U', dest='users', default=False, type=lambda x: file_exists(args, x), help='Users.txt file')
passwd = args.add_mutually_exclusive_group()
passwd.add_argument('-p', dest='passwords', action='append', default=[], help='Single password')
passwd.add_argument('-P', dest='passwords', default=False, type=lambda x: file_exists(args, x), help='Password.txt file')
args.add_argument('-threads', dest='max_threads', type=int, default=3, help='Max threads (Default: 3)')
args.add_argument('-port', dest='port', type=int, default=0, help='Specify non-standard port')
args.add_argument('-r', '-report', dest='report', type=str, default=False, help='Output Report: csv, xlsx (Default: None)')
args.add_argument('-t', dest='dbtype', type=str, required=True, help='Database types currently supported: mssql, pgsql')
args.add_argument('-c', '-columns', dest="column_search", action='store_true', help="Search for key words in column names (Default: table names)")
args.add_argument('-v', dest="verbose", action='store_true', help="Show failed login notices & keyword matches with Empty data sets")
args.add_argument('-brute', dest="brute", action='store_true', help='Brute force only, do not enumerate')
args.add_argument(dest='target', nargs='+', help='Target database server(s)')
args = args.parse_args()
# Put target input into an array
args.target = list_targets(args.target[0])
# Get Password if not provided
if not args.passwords:
args.passwords = [getpass("Enter password, or continue with null-value: ")]
# Define default port based on dbtype
if args.port == 0: args.port = default_port(args.dbtype)
# Launch Main
print("\nStarting enumdb v{}\n".format(version) + "-" * 25)
main(args)
except KeyboardInterrupt:
print("\n[!] Key Event Detected...\n\n")
exit(0)
I am aware that documentation states here http://initd.org/psycopg/docs/module.html states about how connection parameters can be specified. I would like to pass password guesses into the brute class and recursively try different combinations.
PEP-8 asks that you please give classes a name
starting with a capital letter, e.g. Pgsql.
You mentioned that the pgsql connect() method is not working properly,
but didn't offer any diagnostics such as a stack trace.
You seem to be working too hard, given that the sqlalchemy layer
has already addressed the DB porting issue quite nicely.
Just assemble a connect string starting with
the name of the appropriate DB package,
and let sqlalchemy take care of the rest.
All your methods accept con as an argument.
You really want to factor that out as the object attribute self.con.
The db_query() method apparently assumes that
arguments for WHERE clauses already appear, properly quoted, in cmd.
According to Little Bobby's mother,
it makes sense to accept query args according to the API,
rather than worrying about potential for SQL injection.

Tweepy to sqlite3

I'm trying to use Twitter's Streaming API to save tweets to a database using sqlite3. The problem is my database comes back empty. I'm using DB Browser for SQLIte to check the schema and the table and columns are there but no values for any column. Any thoughts?
#create sql table
conn = sqlite3.connect('twitter_one5.db')
c = conn.cursor()
c.execute('''CREATE TABLE tweets
(coordinates integer,
place text)''')
conn.commit()
conn.close()
# open connection
conn = sqlite3.connect('twitter_one5.db')
c = conn.cursor()
Create a class to handle inserting tweet info into database:
class Tweet():
# data from tweet
def __init__(self, coordinates, place):
self.coordinates = coordinates
self.place = place
# insert data from tweet into DB
def insertTweet(self):
c.execute("INSERT INTO tweets (coordinates, place) VALUES(?, ?)",
(self.coordinates, self.place))
conn.commit()
Create a tweet listener using the Streaming API:
class listener(tweepy.StreamListener):
def on_status(self, status):
tweet_data = Tweet(status.coordinates, status.place)
tweet_data.insertTweet()
print('inserted')
Create instance of listener and filter for keywords:
l = listener()
stream = tweepy.Stream(auth, l)
stream.filter(track=['twitter'])

Resources