I have a multithreading application that attach a session for each user of my CSV file.
def main():
executor = ThreadPoolExecutor()
futures = [executor.submit(bot.run,
user["login"],
user["pwd"],) for user in client_data_file]
The code above then call the run method to perform http operations after log in into the website:
def run(login, pwd):
session = login_service(user, pwd)
# Here I'm making http operations for each user
def login_service(user, pwd):
s = self.get_session()
def get_session()
"""Function so specify session to the particular thread on which it's running"""
if not hasattr(thread_local, "session"):
try:
thread_local.session = http.client.HTTPSConnection("url_for_my_site", 80)
return thread_local.session
except Exception as e:
print(e)
To do this http ops, I need to keep this user connection alive.
I was used requests library but is poor performance compared with http Python standard library.
Using requests, is simple:
session = requests.Session()
But, for http.client, I'm in doubt if that HTTPSConnection object keeps the session alive.
I'm trying keep a http session alive using low code libraries.
The need is not for make multiple requests at the same time, but for gain some seconds in high traffic website for each login in my CSV (that's why threading).
Feel free to suggest another low level library
Related
I have developed a Flask application connecting to Snowflake DB as the backend using SQLAlchemy. I ran into Snowflake authentication token expired issue if the website remains idle for more than 4 hrs. I looked over in Google and found an answer
https://github.com/snowflakedb/gosnowflake/issues/160
but it doesn't seem valid for Snowflake-SQLAlchemy.
I implemented a heartbeat code using APScheduler which fires "Select 1" every 50 mins but if the session is active in the browser (SSO) it throws "Authentication token has expired" error.
from sqlalchemy import create_engine
engine = create_engine('snowflake://<connection-string>' )
connection = engine.connect()
names = connection.execute_query("select names from employees")
def sensor():
""" Function for scheduling purposes. """
v = cursor.execute("""select 1""")
v = v.fetchall()
print(v)
sched = BackgroundScheduler(daemon=True)
sched.add_job(sensor,'interval',minutes=50)
sched.start()
I know this is years after the question has been asked. This is my first answer on SO - so probably a good place to practice :)
I suspect there are two separate things in play here - the first is keepalive of a session, which has been answered by others.
The second (and which seems to be your problem) is the time-to-live of your JWT - which is something separate from your session.
No matter what - that JWT token has a lifetime of one hour max. You will therefore need to implement a token refresh strategy.
The time-to-live of the JWT is documented here (halfway down the page - in a note)
https://docs.snowflake.com/en/developer-guide/sql-api/authenticating.html
You can use the internal implementation of snowflake python connector heartbeat like this:
from snowflake.sqlalchemy import URL
engine = create_engine(
URL(
account=DB_ACCOUNT,
user=DB_USER,
password=DB_PASSWORD
),
pool_size=50,
max_overflow=100,
connect_args={
'client_session_keep_alive': True,
})
Session = sessionmaker(engine)
If you don't use an engine, you can also pass the same client_session_keep_alive arg to a connection options (source):
ctx = snowflake.connector.connect(
user='<user_name>',
password='<password>',
account='myorganization-myaccount',
client_session_keep_alive=True
)
The connector will then start a background thread that will force sending a heartbeat to snowflake, so if the token is expiring it will refresh it. What you implemented with select 1 should also work, but it will cost you a little compute.
https://docs.snowflake.net/manuals/sql-reference/parameters.html#client-session-keep-alive
Try Client Session Keep Alive to keep the session active.
I have a pyramid application that send SSE messages. It works basically like these:
def message_generator():
for i in range(100):
print("Sending message:" + str(i))
yield "data: %s\n\n" % json.dumps({'message': str(i)})
time.sleep(random.randint(1, 10))
#view_config(route_name='events')
def events(request):
headers = [('Content-Type', 'text/event-stream'),
('Cache-Control', 'no-cache')]
response = Response(headerlist=headers)
response.app_iter = message_generator()
return response
When I browse to /events I get the events. When I move to another page the events stop, when I close the browser the events stop.
The problem happens for example if I am in /events and I switch off the computer. The server does not know that the client got lost and message_generator keeps sending messages to the void.
In this page: A Look at Server-Sent Events mention this:
...the server should detect this (when the client stops) and stop
sending further events as the client is no longer listening for them.
If the server does not do this, then it will essentially be sending
events out into a void.
Is there a way to detect this with Pyramid? I tried with
request.add_finished_callback()
but this callback seems to be called with
return response
I use Gunicorn with gevent to start the server.
Any idea is highly appreciated
From PEP 3333:
Applications returning a generator or other custom iterator should not assume the entire iterator will be consumed, as it may be closed early by the server.
Basically a WSGI server "should" invoke the close() method on the app_iter when a client disconnects (all generators, such as in your example, support this automatically). However, a server is not required to do it, and it seems many WSGI servers do not. For example, you mentioned gunicorn (which I haven't independently verified), but I did verify that waitress also does not. I opened [1] on waitress as a result, and have been working on a fix. Streaming responses in WSGI environments is shaky at best and usually depends on the server. For example, on waitress, you need to set send_bytes=0 to avoid it buffering the response data.
[1] https://github.com/Pylons/waitress/issues/236
I have a server-client application using xmlrpc.server and xmlrpc.client where the clients request data from the server. As the server only returns this data once certain conditions are met, the clients make the same call over and over again, and currently the tcp connection is re-initiated with each call. This creates a noticeable delay.
I have a fixed number of clients that connect to the server at the beginning of the application and shutdown when the whole application is finished.
I tried to google about keeping the TCP connection open, but all I could find either talked about xmlrpclib or did not apply to the python version.
Client-side code:
import xmlrpc.client as xc
server = xc.ServerProxy(host_IP,8000)
var = False
while type(var)==bool:
var = server.pull_update()
# this returns "False" while the server determines the conditions
# for the client to receive the update aren't met; and the update
# once the conditions are met
Server-side, I am extending xmlrpc.server.SimpleXMLRPCServer with the default xmlrpc.server.SimpleXMLRPCRequestHandler. The function in question is:
def export_pull_update(self):
if condition:
return self.var
else:
return False
Is there a way to get xmlrpc.server to keep the connection alive between calls for the server?
Or should I go the route of using ThreadingMixIn and not completing the client-request until the condition is met?
I have a Flask app which upon certain rest call is running several modules using a ProcessPoolExecutor.
UPDATED: Added redis as a message queue (using docker, redis as redis's host)
socketio = SocketIO(app, message_queue='redis://redis')
(...)
def emit_event(evt, message):
socketio.emit(evt, message, namespace='/test')
#app.route('/info', methods=['GET'])
def info():
emit_event('update_reports', '')
(...)
if __name__ == "__main__":
socketio.run(host='0.0.0.0', threaded=True)
Now that I added redis, it still works when emitting from the main app.
Here some from the code I'm running the sub-process:
def __init__(self):
self.executor = futures.ProcessPoolExecutor(max_workers=4)
self.socketio = SocketIO(async_mode='eventlet', message_queue='redis://redis')
(...)
future = self.executor.submit(process, params)
future.add_done_callback(functools.partial(self.finished_callback, pid))
Then in that callback I'm calling the emit_event method:
def finished_callback(self, pid, future):
pid.status = Status.DONE.value
pid.finished_at = datetime.datetime.utcnow
pid.save()
self.socketio.emit('update_reports', 'done', namespace='/test')
Getting and sending/ emitting messages from/to the client from my controller works just fine, also if I call /info from curl or postman my client gets the message -but- when trying to emit an event same way from within this subprocess callback, now it shows this error:
This is mostly for notifications, like notifying when a long process has finished and stuff like that.
INFO:socketio:emitting event "update_reports" to all [/test]
ERROR:socketio:Cannot publish to redis... retrying
ERROR:socketio:Cannot publish to redis... giving up
What I'm doing wrong?
Thanks!
There are specific rules that you need to follow in setting up the Flask-SocketIO extension so that external processes can emit, which include the use of a message queue that the main and external processes use to coordinate efforts. See the Emitting from an External Process section of the documentation for instructions.
Hopefully this should be a quick answer for somebody. I've looked through the docs a bit, but still haven't found a definitive answer. I have a number of 'idle' connections that stick around, even if I perform a session.close() in SQLAlchemy. Are these idle connections the way SQLAlchemy/Postgres handle connection pooling?
This is the query I used to check db connection activity
SELECT * FROM pg_stat_activity ;
Here is sample code:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
application = Flask(__name__)
application.config.from_object('config')
db = SQLAlchemy(application)
class Brand(db.Model):
id = db.Column(db.Integer, primary_key=True)
#application.route('/')
def documentation():
all = Brand.query.all()
db.session.remove() #don't need this since it's called on teardown
return str(len(all))
if __name__ == '__main__':
application.run(host='0.0.0.0', debug=True)
Yes. Closing a session does not immediately close the underlying DBAPI connection. The connection gets put back into the pool for subsequent reuse.
From the SQLAlchemy docs:
[...] For each Engine encountered, a Connection is associated with it, which is acquired via the Engine.contextual_connect() method. [...]
Then, Engine.contextual_connect() points you to Engine.connect(), which states the following:
The Connection object is a facade that uses a DBAPI connection internally in order to communicate with the database. This connection is procured from the connection-holding Pool referenced by this Engine. When the close() method of the Connection object is called, the underlying DBAPI connection is then returned to the connection pool, where it may be used again in a subsequent call to connect().