I'm using zodb with cherrypy. I'm passing data into a jinja2 template. I would like to close my zodb connection before I return the template render.
I can't because the template needs the connection when it's rendering because the data is zodb data.
This errors out...
zconn.close() #zconn.close closes the conn,db, and storage
return template.render(args....)
How do I close down the zodb conn after I'm already finished returning? Is there some "before request" and "after request" function I can define to always have a db connection? The cherrypy documentation doesn't have much explicit stuff on database connections save for "do it explicitly at the beginning of the exposed function - and close it before you return".
Or is it standard practice to say...
page = template.render(args...)
zconn.close()
return page
It is standard practice to only close the connection when the request is complete, yes.
Postpone the close to the latest possible moment. Use could use a context manager to handle the closing for you:
from contextlib import contextmanager
#contextmanager
def zodbconn(db):
conn = db.open()
yield conn.root()
conn.close()
then use:
with zodbconn(db) as zconn:
return template.render(args....)
and the connection will be closed automatically after the return statement has been processed.
Related
I'm coding a script that connects to the Binance websocket and uses the .run_forever() method to constantly get live data from the site. I want to be able to debug my code and watch the values of variables as the change but I'm not sure how to do this as the script basically hangs on the line with the .run_forever() method, because it is an infinite event loop. This is by design as I want to continuously get live data (it receives a message approximately every second), but I can't think of a way a good way to debug it.
I'm using VSCode and here are some snippets of my code to help understand my issue. The message function for the websocket is just a bunch of technical analysis and trade logic, but it is also the function that contains all the changing variables that I want to watch.
socket = f"wss://stream.binance.com:9443/ws/{Symbol}#kline_{interval}"
def on_open(ws):
print("open connection")
def on_message(ws, message):
global trade_list
global in_position
json_message = json.loads(message)
candle = json_message['k'] # Accesses candle data
...[trade logic code here]...
def on_close(ws):
print("Websocket connection close")
# ------------------------- Define a websocket object ------------------------ #
ws = websocket.WebSocketApp(socket, on_open=on_open, on_message=on_message, on_close=on_close)
ws.run_forever()
If more code is required to answer the question, then I can edit this question to include it (I'm thinking if you would like to have an idea of what variables I want to look at, I just thought it would be easier and simpler to show these parts).
Also, I know using global isn't great, once I've finished (or am close to finishing) the script, I want to go and tidy it up, I'll deal with it then.
I'm a little late to the party but the statement
websocket.enableTrace(True)
worked for me. Place it just before you define your websocket object and it will print all traffic in and out of the websocket including any exceptions that you might get as you process the messages.
I am using Flask and SqlAlchemy, the case is that I have a main database and then I have several databases where I get other information. But the credentials are not defined as fixed, so I have to obtain them from a table in the main database, depending on the plant where the user is. For this I use SQLALCHEMY_BINDS. the problem is that when I try to pass the connection string to the Bind I told myself that the function that returns it is out of context. Here a portion of the code
def scadaConnect():
idplanta_session=str(session['idPlanta'])
usernamequery = db.session.query(Scada.usernameScada).filter(Scada.idPlanta=='5')
hostquery = db.session.query(Scada.hostScada).filter(Scada.idPlanta=='5')
passwordquery = db.session.query(Scada.passScada).filter(Scada.idPlanta=='5')
nombredbquery = db.session.query(Scada.nombrebdScada).filter(Scada.idPlanta=='5')
nombredb = str(nombredbquery[0])[2:len(nombredbquery[0])-4]
host = str(hostquery[0])[2:len(hostquery[0])-4]
password = str(passwordquery[0])[2:len(passwordquery[0])-4]
username = str(usernamequery[0])[2:len(usernamequery[0])-4]
connexion = 'mysql+pymysql://'+username+":"+password+"#"+host+"/"+nombredb+"'"
def retorno():
return str(connexion)
from config import SQLALCHEMY_BINDS
SQLALCHEMY_BINDS['scada']= scadaConnect()
The error is as follows
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed an active HTTP request. Consult the documentation on testing for information about how to avoid this problem.
session (first line in scadaConnect()) is only available in a request context a.k.a. view. It lives in the session cookie on the client and is sent to the server (and available to your view) only when the browser makes a request.
You will have to move the call to scadaConnect() to a view if you want to use session.
I've got a simple FastAPI webapp going and I'd like to be able to check the database connection on startup (and retry connection if it fails)
I've got the following code, but it doesn't feel right
# main.py
import uvicorn
from backend.app import app
if __name__ == "__main__":
uvicorn.run(app, port=8001)
# app.py
# ... omitted for brevity
from backend.database import notes, tags
# ... omitted for brevity
# database.py
from motor.motor_asyncio import AsyncIOMotorClient
from asyncio import get_event_loop
client = AsyncIOMotorClient("localhost", 27027)
loop = get_event_loop()
data = loop.run_until_complete(client.server_info())
db = client.notes_db
notes = db.notes
tags = db.tags
Without get_event_loop() and the subsequent loop.run_until_complete() call it won't test the database connection until you actually try to access / write to it.
My goal is to be able to halt the startup process until it can successfully connect to a database, is there any clean way to do this with Python and motor.io (https://motor.readthedocs.io/, sorry there's no tag for it) ?
the startup event in FastAPI is the deal here I guess. I addition this repository is a nice example and this thread could even provide you with more information. You could execute your tests within the startup event. This means the application won't start until the startup event has been successfully executed.
I know for a PreparedRequest, we specifiy the timeout in session.send call, where session can be an instance of requests.Session class, as I've already seen here: http://docs.python-requests.org/en/latest/user/advanced/
But I need to put the timeout before I do the send. Somewhat inherently, to the PreparedRequest object. Because I'm using the session.send method as map function, to map to PreparedRequest instances.
def async_req2resp(reqs, session):
responses = []
try:
with futures.ThreadPoolExecutor(max_workers=workers) as executor:
for response in executor.map(session.send, requests):
responses.append(response)
except Exception as e:
print('async_requests2responses exception: ' + str(e))
return responses
Now I need to specify the timeout somehow. How can I do it?
The above code is wrapped inside a method. I am getting the session object as an argument. Is there a way I can set the timeout for all the requests that will be sent with the session object, before the parallel execution of session.send happens?
Thank you.
I'm waiting for your answer about sendit but until then, let me give you some information that should help you get the rest of the way there.
If you have a PreparedRequest object as request then you can do session.send(request, timeout=timeout_val) where timeout_val is whatever the value of the timeout is that you want.
With that in mind, this question becomes, "How do I pass that to sendit with each request?" and that question I don't have the answer for.
Side note:
You've tagged this with concurrent.futures which makes me suspect you're trying to optimize your application that uses this method. With that in mind, you should avoid appending to a list at all costs. You should, instead, try to do the following:
responses = []
try:
responses = list(sendit.map(session.send, requests))
# etc.
If you figure out how to pass the timeout parameter to sendit, feel free to suggest an edit to this answer to update it to be complete.
With the updated information in the question, I can more accurately answer your question.
Before you call executor.map you should do the following:
import functools
timedout_send = functools.partial(session.send, timeout=my_timeout)
try:
list(executor.map(timedout_send, requests))
# ...
I have a web application working with sqlite database.
My version of sqlite is the latest from official windows binary distribution - 3.7.13.
The problem is that under heavy load on database, sqlite API functions (such as sqlite3_step) are returning SQLITE_BUSY.
I pass the following pragmas when initializing a connection:
journal_mode = WAL
page_size = 4096
synchronous = FULL
foreign_keys = on
The databas is one-file database. And I'm using Mono 2.10.8 and Mono.Data.Sqlite assembly provided with it to access database.
I'm testing it with 50 parallel threads which are sending 50 subsequent http-requests each to my application. On every request some reading and writing are done to the database. Every set of IO operations is executed inside the transaction.
Everything goes well until near 400th - 700th request. In this (random) moment API functions are starting to return SQLITE_BUSY permanently (To be more exact - until the limit of retries is reached).
As far as i know WAL mode transparently supports parallel reads and writes. I've guessed that it could be because of attempt to read database while checkpoint operation is executed. But even after turning autocheckpoint off the situation remains the same.
What could be wrong in this situation?
How to serve large amount of parallel database IO correctly?
P.S.
Only one connection per request is supposed.
I use nhibernate configured with WebSessionContext.
I initialize my NHibernate session like this:
ISession session = null;
//factory variable is session factory
if (CurrentSessionContext.HasBind(factory))
{
session = factory.GetCurrentSession();
if (session == null)
CurrentSessionContext.Unbind(factory);
}
if (session == null)
{
session = factory.OpenSession();
CurrentSessionContext.Bind(session);
}
return session;
And on HttpApplication.EndRequest i release it like this:
//factory variable is session factory
if (CurrentSessionContext.HasBind(factory))
{
try
{
CurrentSessionContext.Unbind(factory)
.Dispose();
}
catch (Exception ee)
{
Logr.Error("Error uninitializing session", ee);
}
}
So as far as i know there should be only one connection per request life cycle. While proceessing the request, code is executed sequentially (ASP.NET MVC 3). So it doesn't look like any concurency is possible here. Can i conclude that no connections are shared in this case?
It's not clear to me if the request threads share the same connection or not. If they don't then you should not be having these issues.
Assuming that you are indeed sharing the connection object across multiple threads, you should use some locking mechanism as the the SqliteConnection isn't thread-safe (an old post, but the SQLite library maintained as part of Mono evolved from System.Data.SQLite found on http://sqlite.phxsoftware.com).
So assuming that you don't lock around using the SqliteConnection object, can you please try it? A simple way to accomplish this could look like this:
static readonly object _locker = new object();
public void ProcessRequest()
{
lock (_locker) {
using (IDbCommand dbcmd = conn.CreateCommand()) {
string sql = "INSERT INTO foo VALUES ('bar')";
dbcmd.CommandText = sql;
dbcmd.ExecuteNonQuery();
}
}
}
You may however choose to open a distinct connection with each thread to ensure you don't have any more threading issues with the SQLite library.
EDIT
Following-up on the code you posted, do you close the session after committing the transaction? If you don't use some ITransaction, do you flush and close the session? I'm asking since I don't see it in your code, and I see it mentioned in https://stackoverflow.com/a/43567/610650
I also see it mentioned on http://nhibernate.info/doc/nh/en/index.html#session-configuration:
Also note that you may call NHibernateHelper.GetCurrentSession(); as
many times as you like, you will always get the current ISession of
this HTTP request. You have to make sure the ISession is closed after
your unit-of-work completes, either in Application_EndRequest event
handler in your application class or in a HttpModule before the HTTP
response is sent.