I'm trying to log all SQLAlchemy queries to the console while parsing the query and filling in the parameters (e.g. translating :param_1 to 123). I managed to find this answer on SO that does just that. The issue I'm running into is that parameters don't always get translated.
Here is the event I'm latching onto -
#event.listens_for(Engine, 'after_execute', named=True)
def after_cursor_execute(**kw):
conn = kw['conn']
params = kw['params']
result = kw['result']
stmt = kw['clauseelement']
multiparams = kw['multiparams']
print(literalquery(stmt))
Running this query will fail to translate my parameters. Instead, I'll see :param_1 in the output -
Model.query.get(123)
It yields a CompileError exception with message Bind parameter '%(38287064 param)s' without a renderable value not allowed here..
However, this query will translate :param_1 to 123 like I would expect -
db.session.query(Model).filter(Model.id == 123).first()
Is there any way to translate any and all queries that are run using SQLAlchemy?
FWIW I'm targeting SQL Server using the pyodbc driver.
If you set up the logging framework, you can get the SQL statements logged by setting the sqlalchemy.engine logger at INFO level, e.g.:
import logging
logging.basicConfig()
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)
Related
I'm trying to set up some simple web service with Python and Flask and SQlite3. It doesn't work.
The DB connection without web service works; the web service without DB connections works. Together they don't.
if I run this, it works:
import sqlite3
conn = sqlite3.connect('scuola.db')
sql = "SELECT matricola,cognome,nome FROM studenti"
cur = conn.cursor()
cur.execute(sql)
risultato = cur.fetchall()
conn.close()
print(risultato)
(so query is correct)
and if I run this, it works
import flask
app = flask.Flask(__name__)
def funzione():
return 'Applicazione Flask'
app.add_url_rule('/', 'funzione', funzione)
but if I run this...
from flask import Flask
import sqlite3
app = Flask(__name__)
#app.route('/',methods=['GET'])
def getStudenti():
conn = sqlite3.connect('scuola.db')
sql = "SELECT matricola,cognome,nome FROM studenti"
cur = conn.cursor()
cur.execute(sql)
risultato = cur.fetchall()
conn.close()
return risultato
It returns Internal Server Error in the browser, and
sqlite3.OperationalError: no such table: studenti
on the DOS prompt.
Thank you for your help!
You haven't provided the internal server error output - but my first guess is that you're trying to return the raw list object returned from fetchall.
When returning from a view function you need to send the results either by returning a template, or by jsonifying the output to make it a proper HTTP response that the browser can receive.
You need to add
from flask import jsonify
in your imports, then when returning;
return jsonify(risultato)
If you get errors like something is not JSON serializable if means you're trying to send an instance of a class or similar. You'll need to make sure you're returning only plain python data structures (e.g. list/dict/str etc).
For the command line problem, you need to make sure you've ran a CREATE TABLE command to first generate the table in the database, before you select from it. Also check you're accessing the correct sqlite database file with the table in it.
I'm not sure, but from the look of things I don't think you've configured the flask app to support the db you created There should be some sort of app.config() that integrates the db.
The database is in Azure cloud and not being used in production currently. There are 80.000 rows and a uprn is a VARCHAR(100);
I'm already using JOI to validate each UPRN as well;
I'm using KNEX with a SQL Server database with the following whereIn query:
knex(LOCATIONS.table).whereIn(LOCATIONS.uprn, req.body.uprns)
but this takes 8-12s to complete and sometimes timesout. if I use .toQuery() on the same thing, SSMS will return the result within 1-2.
If I do a raw query, the resulting .toQuery() or toString() works in SSMS and returns results. But if I try to use the raw directly, it will return 0 results.
I'm looking to either fix what's making whereIn so slow or get the raw query working.
EDIT 1:
After much debugging and trying -- it seems that the bug is due to how knex deals with arrays, so I made a for-of loop to add ? ? ? for each array element and then inputed the array for all params.
This led me to realizing the performance issue is due to SQL server way of parameterising.
I ended up building a raw query string with all of the parameters and validating the input with Joi string/regex config:
Joi.string()
.min(1)
.max(35)
.regex(/^[a-z\d\-_\s]+$/i)
allowing only for alphanumeric, dashes and spaces which should prevent sql injection.
I'm going to look deeper into security issues with this and might make a separate login that can only SELECT data from that table and nothing more to run with these queries.
Needed to just handle it raw and validate separately.
i am trying to work with multiple database and schemas using simple vertical partitioning in sqlalchemy and python .
Have create two database engines and configured successfully to sessionmaker()
Session = sessionmaker()
Session.configure(binds={BaseA:engine1, BaseB:engine2})
Able to get the required sql query generated successfully
driverssql = session.query(drivers)
but when i execute the above query to fetch the requslts i get the follwing error :
resultset=session.execute(driversql)
sqlalchemy.exc.UnboundExecutionError: Could not locate a bind configured on SQL expression or this Session (how can i associate the correct engine with execute statement)
I'm seeing here two variants:
You can create 2 sessionmakers here and use them separately according to an engine.
You can choose necessary engine when executing a query:
engine1 = create_engine(first_db)
engine2 = create_engine(second_db)
session.execute(drivers, bind=engine1)
I am attempting to set the database/document context programmatically via the Python API. My steps are as follows:
session = BaseXClient.Session("localhost", 1984, "admin", "admin")
query = session.query("//node")
query.context("doc('dbname')") # **NOT SURE HOW TO SET THE DB TO USE**
query.execute()
I already know that I can simply use the session object as follows and it works fine:
session.execute("xquery doc('dbname')//node/child")
But I am looking for a way to OPEN a database within the scope of the program call separate from the query string. I am not able to find the documentation to explicitly set the database prior to executing the query using the context object. I have looked at the source code for the python BaseXClient and there is context method for the Query() instance that is not well documented. I am attempting to use this to set the Database and not having much luck.
The context you have supplied is just a string. It is not evaluated. In a client server context it is difficult to see how one could pass in a database here.
I think your alternatives are to use the execute command to open a database before running the query. This will set the context. e.g.
var q = session.execute("open mydatabase",log.print)
var q = session.query("count(*)")
or use the query command bind command to pass parameters
var q = session.query("declare variable $db external; count(collection($db))")
q.bind("db", "mydatabase","",log.print);
q.execute(log.print);
Sorry these examples use Javascript and my BaseX Node client as I am not familiar with the Python API but I am sure the same applies in the Python API
I am connecting to snowflake datawarehouse from Python and I encounter a weird behavior. The Python program exits successfully if I retrieve fewer number of rows from SnowFlake but hangs in there in-definitely if I try to retrieve more than 200K rows. I am 100% sure that there are no issues with my machine because I am able to retrieve 5 to 10 million rows from other type of database systems such as Postgres.
My Python environment is Python 3.6 and I use the following version of the libraries -> SQLAlchemy 1.1.13, snowflake-connector-python 1.4.13, snowflake-sqlalchemy 1.0.7,
The following code prints the total number of rows and closes the connection.
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(URL(
account=xxxx,
user=xxxxx,
password=xxxxx,
database=xxxxx,
schema=xxxxxx,
warehouse=xxxxx))
query = """SELECT * FROM db_name.schema_name.table_name LIMIT 1000"""
results = engine.execute(query)
print (results.rowcount)
engine.dispose()
The following code prints the total number of rows but the connection doesn't close, it just hangs in there until I manually kill the Python process.
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(URL(
account=xxxx,
user=xxxxx,
password=xxxxx,
database=xxxxx,
schema=xxxxxx,
warehouse=xxxxx))
query = """SELECT * FROM db_name.schema_name.table_name LIMIT 500000"""
results = engine.execute(query)
print (results.rowcount)
engine.dispose()
I tried multiple different tables and I encounter the same issue with SnowFlake. Did anyone encounter similar issues?
Can you check the query status from UI? "History" page should include the query. If the warehouse is not ready, it may take a couple of minutes to start the query. (I guess that's very unlikely, though).
Try changing the connection to this:
connection = engine.connect()
results = connection.execute(query)
print (results.rowcount)
connection.close()
engine.dispose()
SQLAlchemy's dispose doesn't close the connection if the connection is not explicitly closed. I inquired before, but so far the workaround is just close the connection.
https://groups.google.com/forum/#!searchin/sqlalchemy/shige%7Csort:date/sqlalchemy/M7IIJkrlv0Q/HGaQLBFGAQAJ
Lastly, if the issue still persist, add the logger to top:
import logging
for logger_name in ['snowflake','botocore']:
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
ch = logging.FileHandler('log')
ch.setLevel(logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
logger.addHandler(ch)
and collect log.
If the output is too long to fit into here, I can take it at the issue page at https://github.com/snowflakedb/snowflake-sqlalchemy.
Note I tried it myself but cannot reproduce the issue so far.
Have you tried using a with statement instead to make your connection
instead of this:
engine = create_engine(URL(account=xxxx,user=xxxxx,password=xxxxx,database=xxxxx,schema=xxxxxx,warehouse=xxxxx))
results = engine.execute(query)
do the following:
with create_engine(URL(account=xxxx,user=xxxxx,password=xxxxx,database=xxxxx,schema=xxxxxx,warehouse=xxxxx)) as engine:
# do work
results = engine.execute(query)
...
After the with .. the engine object should be automatically closed out.