Why is this pattern discouraged in SQLAlchemy? - multithreading

When reading the official SQLAlchemy documentation, I found the example below:
### this is the **wrong way to do it** ###
class ThingOne(object):
def go(self):
session = Session()
try:
session.query(FooBar).update({"x": 5})
session.commit()
except:
session.rollback()
raise
class ThingTwo(object):
def go(self):
session = Session()
try:
session.query(Widget).update({"q": 18})
session.commit()
except:
session.rollback()
raise
def run_my_program():
ThingOne().go()
ThingTwo().go()
I don't really understand the drawbacks for this pattern. Actually I can think of one major ADVANTAGE to this: in a multi-threading context, this pattern can ensure that each session instance is a local variable to the function that actually uses it.
Could someone enlighten me by giving some potential drawbacks for the example above? Thanks.
Edit: As an example for advantages in a multiple threading context. If we have a web application server class here:
class WebApp:
def update(self, **kwargs):
session = Session()
try:...
Here, the page handler update has its own local variable session, so no matter how many threads it runs, it's always safe. In constrast, using another layer of function to contain session would introduce way more complexity in this case

In simple terms, sqlalchemy recommends that the handling of session is not mixed with the manipulation of data. As you can see from the next example.
### this is a **better** (but not the only) way to do it ###
class ThingOne(object):
def go(self, session):
session.query(FooBar).update({"x": 5})
class ThingTwo(object):
def go(self, session):
session.query(Widget).update({"q": 18})
def run_my_program():
session = Session()
try:
ThingOne().go(session)
ThingTwo().go(session)
session.commit()
except:
session.rollback()
raise
finally:
session.close()
ThingOne and ThingTwo is just doing it's thing in doing CRUD but the handling of session is done outside of those objects.
For multi threading, session scopes are thread local objects. Meaning, they can't be shared by different threads. You can declare them as what you have specified but it doesn't mean that handling session from an external entity is bad choice either.

Related

Keep a class running in a python subprocess or thread or process

I am using Squish to automate a Qt-based GUI application. I look up qt objects in the application recursively. Since it's time-intensive, I would like to cache objects once found for later reuse. I have the below class to maintain a cache of objects in a dictionary -
def __init__(self):
self.object_store = {}
#staticmethod
def instance():
if '_instance' not in ObjectCache.__dict__:
ObjectCache._instance = ObjectCache()
return ObjectCache._instance
def set(self, object_name, obj):
self.object_store[object_name] = obj
def remove(self, object_name):
del self.object_store[object_name]
def exists(self, object_name):
if object_name in self.object_store:
return True
return False
def get(self, object_name):
return self.object_store[object_name]
def get_all(self):
return self.object_store
I have below decorator for functions in my automation scripts to add/access/delete from this dictionary -
def object_caching_decorator(func):
def wrapper(*args, **kwargs):
object_cache = ObjectCache.instance()
if object_cache.exists(func.__name__):
try:
if waitForObject(object_cache.get(func.__name__)):
return object_cache.get(func.__name__)
except LookupError:
object_cache.remove(func.__name__)
obj = func(*args, **kwargs)
object_cache.set(func.__name__, obj)
return obj
return wrapper
One might ask why can't all scripts share this class object? because the Squish tool resets the global symbol table before starting every test script hence I need a way to persist this object.
How do I keep this class running so that the scripts running on another process (Squish runner) can access it seamlessly?
Each Squish test case gets executed in a new instance (process) of the squishrunner and the script interpreter hosted within.
The object references that Squish gives you in the test script are actually proxy objects that transparently (behind the scenes) access the actual object inside of the application process for you, without you having to do anything for this "magic" to happen (or becoming aware of it, most of the time). Caching/persisting these objects across test case will not work and is not possible.
Also, caching object references is a notorious problem as the life-time of the objects referenced by these proxy objects may change if the AUT (Application Under Test) gets changed, or used in a different manner.
Instead of this, you should revisit the way that you look up objects. It is very likely that there is a better way that allows ad-hoc object lookup (as intended) that is fast enough. (If in doubt, I recommend to contact the vendor of Squish, since your maintenance contract or subscription of their product entitles you to technical support.)
May be you transfer the object to it's "real name"-dictionary.
object_dictionary = objectMap.realName(object)
So you don't need the volatil object and have a persistant real-name-Object you can transfer.

Python class inheritance with Elasticsearch server connection

I'm trying to create a python class that inherits the Elasticsearch class and builds upon it with some custom methods. The issue i am facing is that i'd like the class constructor to connect to the server, so initialisation is simple. Usually to connect to the server it looks something like this:
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'XXXXXXX', 'port': XXXX}]
In my class, that i'm calling "Elastic", i'd like to connect to the server and return the Elasticsearch object upon initialisation of the class, i.e.:
es = Elastic()
which I can then use to perform existing Elasticsearch class methods, and my own custom operations, e.g.:
es.search() # existing class method
es.custom_method_example1() # new class method
I've been trying and failing to come up with a way to do this - my most recent attempt involved using the __new__ dunder method so that I could return the connected es object as the new class:
class Elastic(Elasticsearch):
def __new__(cls, timeout=10, max_retries=5, retry_on_timeout=True, *args, **kwargs):
"Connect to our ES server."
return Elasticsearch([{'host': 'XXXXX', 'port': XXXX}], timeout=10, max_retries=5, retry_on_timeout=True, *args, **kwargs)
def custom_method_example1(self, *args, **kwargs):
"""
Performs some custom method that wasn't possible with the standalone Elasticsearch class
"""
Firstly it doesn't work:
AttributeError: 'Elasticsearch' object has no attribute 'custom_method_example1', seems that it's no longer inheriting but replacing the class?
And secondly, I gather from reading about that __new__ generally doesn't have much use (particularly for amateur programmers like me) so I'm probably taking the wrong approach / overcomplicating it here. If anyone knows the "right" way to do this it would be much appreciated - I've been reading a bit about factory design and it seems like the right way to go in general but im still making sense of it all (i'm an analyst by trade). I figure decorators might come into use somewhere??
Thanks and sorry for the waffle
Indeed I was very much overcomplicating it. Didn't consider that class inheritance involves inheritance of the constructor itself - therefore I can call the subclass Elastic just as I was for the parent Elasticsearch:
from elasticsearch import Elasticsearch
class Elastic(Elasticsearch):
def custom_method_example1(self, *args, **kwargs):
"""
Performs some custom method that wasn't possible with the standalone Elasticsearch class
"""
To initialise the class and call it's methods:
es = Elastic([{'host': 'XXXXX', 'port': XXXX}], timeout=10, max_retries=5, retry_on_timeout=True)
es.custom_method_example1()
EDIT: I still had the issue of wanting to set new default parameters for my subclass constructor - I've now found out how to do this using super() which explicitly calls the parents constructor passing on arguments i set in the subclass constructor, leaving me with:
from elasticsearch import Elasticsearch
class Elastic(Elasticsearch):
def __init__(self, hosts=[{'host':'XXXXXX', 'port':XXX}], timeout=10, max_retries=5, retry_on_timeout=True):
super().__init__(hosts=hosts,timeout=timeout,max_retries=max_retries,retry_on_timeout=retry_on_timeout)
def custom_method_example1(self, *args, **kwargs):
"""
Performs some custom method that wasn't possible with the standalone Elasticsearch class
"""
allowing me to initialise the class like so:
es = Elastic()

Parent class to expose standard methods, child class to provide sub-methods to do the work

I'd like to set up a parent class that defines a standard interface and performs common things for all children instances. However, each child will have different specifics for how these methods get the job done. For example, the parent class would provide standard methods as follows:
class Camera():
camera_type = None
def __init__(self, save_to=None):
self.file_loc = save_to
def connect(self):
self.cam_connect()
with open(self.file_loc, 'w'):
# do something common to all cameras
def start_record(self):
self.cam_start_record()
# do something common to all cameras
Each of these methods refers to another method located only in the child. The child classes will have the actual details on how to perform the task required, which may include the combination of several methods. For example:
class AmazingCamera(Camera):
camera_type = 'Amazing Camera'
def __init__(self, host_ip='10.10.10.10', **kwargs):
super(AmazingCamera, self).__init__(**kwargs)
self.host_ip = host_ip
def cam_connect(self):
print('I are connectifying to {}'.format(self.host_ip))
# do a bunch of custom things including calling other
# local methods to get the job done.
def cam_start_record(self):
print('Recording from {}'.format(self.host_ip)
# do a bunch more things specific to this camera
### etc...
With the outcome of the above providing an interface such as:
mycamera = AmazingCamera(host_ip='1.2.3.4', save_to='/tmp/asdf')
mycamera.connect()
mycamera.start_record()
I understand fully that I can simply override the parent methods, but in cases where the parent methods do other things like handling files and such I'd prefer to not have to do that. What I have above seems to work just fine so far but before I continue creating this I'd like to know if there is there a better, more pythonic way to achieve what I'm after.
TIA!
I opted to keep the standard methods identical between the parent and child and minimize the use of child-specific helper methods. Just seemed cleaner.
As an example:
class Camera():
camera_type = None
def connect(self):
with open(self.file_loc, 'w'):
# do something common to all cameras
Then in the child I'm overriding the methods, but calling the method of the parent in the override as follows:
class AmazingCamera(Camera):
camera_type = 'Amazing Camera'
def cam_connect(self):
print('I are connectifying to {}'.format(self.host_ip))
# call the parent's method
super().connect()
# do a bunch of custom things specific to
# AmazingCamera

python tornado how to get id of a edge

I have a web application that runs on flask web framework. Flask couldn't answer application requirements anymore. So we have decided to migrate tornado.
I need to handle below two request.
/entry GET Method
/entry/id GET Method
When first request called it must return authenticated entry.
When second request called it must return the entry whose entry_id
is id
Is there a different nice solution to handle above request except my solution. The solution i have found it creates cyclomatic complexity.
def get(self):
id = self.request.path.split('/')[-1]
if id is None:
#return authenticated user
else:
#return the user whose entry_id is id
Note: I am looking a solution like this:
#rest_user.route('/user', methods=['GET'])
#some code
#rest_user.route('/user/<user_id>', methods=['GET'])
#some code
The analogous arrangement in Tornado uses two handler classes (perhaps with a common base class for shared methods):
class AuthedUserHandler(RequestHandler):
def get(self):
...
class UserHandler(RequestHandler):
def get(self, user_id):
...
app = Application([
('/user', AuthedUserHandler),
('/user/(.*)', UserHandler),
])

best practices for passing initialization arguments to superclasses?

I'm trying to figure out the best way to initialize sub/superclasses in Python3. Both the base and subclasses will take half a dozen parameters, all of which will be parsed from command line arguments.
The obvious way to implement this is to parse all the args at once, and pass them all in:
class Base:
def __init__(self, base_arg1, base_arg2, base_arg3):
class Sub(Base):
def __init__(self, sub_arg1, sub_arg2, sub_arg3,
base_arg1, base_arg2, base_arg3):
super().__init__(self, base_arg1, base_arg2, base_arg3)
main():
# parse args here
options = parser.parse_args()
obj = Sub(options.sub_arg1, options.sub_arg2, options.sub_arg3,
options.base_arg1, options.base_arg2, options.base_arg3)
If I have a Sub-subclass (which I will), things get really hairy in terms of the list of arguments passed up through successive super().init() calls.
But it occurs to me that argparse.parse_known_args() offers another path: I could have each subclass parse out the arguments it needs/recognizes and pass the rest of the arguments up the hierarchy:
class Base:
def __init__(self, args):
base_options = base_parser.parse_known_args(args)
class Sub(Base):
def __init__(self, args):
(sub_options, other_args) = sub_parser.parse_known_args(args)
super().__init__(self, other_args)
main():
obj = Sub(sys.argv)
This seems cleaner from an API point of view. But I can imagine that it violates some tenet of The Way Things Are Done In Python and is a bad idea for all sorts of reasons. My search of the web has not turned up any examples either way - could the mighty and all-knowing mind of Stack Overflow help me understand the Right Way to do this?
Look inside the argparse.py code. An ArgumentParser is a subclass of an _ActionsContainer. All the actions are subclasses of Action.
When you call
parser.add_argument('foo', action='store_action', ...)
the parameters are passed, mostly as *args and **kwargs to _StoreAction, which in turn passes them on to its supper (after a setting some defaults, etc).
As a module that is mean to be imported, and never run as a stand along script it does not have a if __name__.... block. But often I'll include such a block to invoke test code. That's the place to put the commandline parser, or at least to invoke it. If might be defined in a function in the body, but it normally shouldn't be called when module is imported.
In general argparse is a scripting tool, and shouldn't be part of a class definitions - unless you are a subclassing ArgumentParser to add some new functionality.
You might also want to look at https://pypi.python.org/pypi/plac. This package provides a different interface to argparse, and is a good example of subclassing this parser.
Thanks hpaulj! I think your response helped me figure out an even simpler way to go about it. I can parse all the options at the top level, then just pass the option namespace in, and let each subclass pull out the ones it needs. Kind of face-palm simple, compared to the other approaches:
class Base:
def __init__(self, options):
base_arg1 = options.base_arg1
base_arg2 = options.base_arg2
class Sub(Base):
def __init__(self, options):
super().__init__(self, options) # initialize base class
sub_arg1 = options.sub_arg1
sub_arg2 = options.sub_arg2
main():
options = parser.parse_args()
obj = Sub(options)

Resources