I'm a beginner. I've defined the following classes. They look the same. I'm a little dizzy. I don't know what the difference is?
My purpose is to define a base class. Why use object? I think B is what I want.
Do I have to use the super () function to return? What do CLS, * args, * kwargs stand for?
python code :
class A(object):
def __new__(cls, *args, **kwargs):
print("A.__new__called")
return super(A, cls).__new__(cls, *args, **kwargs)
class B:
def __new__(cls, *args, **kwargs):
print("B.__new__called")
return super(B, cls).__new__(cls, *args, **kwargs)
class C():
def __new__(cls, *args, **kwargs):
print("C.__new__called")
return super(C, cls).__new__(cls, *args, **kwargs)
class D():
def __new__(cls, *args, **kwargs):
print("D.__new__called")
return super(D, cls).__new__(cls)
class E():
def __new__(cls):
print("E.__new__called")
return super(E, cls).__new__(cls)
class F():
print("F.__new__called")
a = A()
b = B()
c = C()
d = D()
e = E()
f = F()
result :
F.__new__called
A.__new__called
B.__new__called
C.__new__called
D.__new__called
E.__new__called
Few comments:
object is inherited by default in python3.x onwards and thus no need for this explicit inheritance as required in python2.x
super() also doesn't need any arguments in py3.x.
class A:
def __init__(self):
print("Here in init")
def __new__(cls,*args,**kargs):
print("Here in new")
super().__new__(cls,**kargs)
new is always class method and called before an object is instantiated. For example, if you create the object above, following print statement will be in that order: first what is in new and then init. So new is used if you want to do something before an object is instantiated. super() calls the inherited base class (default object in this case) to create the object. *args and **kargs are just list of arguments and key arguments to pass when you create the object.
obj = A()
It will print following:
Here in new
Here in init
Related
The following are the decorators defined,
from functools import wraps
def decorator(func):
#wraps(func)
def wrapper(*args, **kwargs):
print(args, kwargs)
return func(*args, **kwargs)
return wrapper
and I know that these two ways of writing are equivalent.
#decorator
def test():
pass
def test():
pass
test = decorator(test)
The difference is that when decorating inside an instance method, the wrapper function will not get the self parameter.
class Test:
#decorator
def test(self):
pass
# class Test:
#
# def test(self):
# pass
#
# test = decorator(test)
Test().test()
# (<__main__.Test object at 0x10b6e2aa0>,) {}
class Test:
def test(self):
pass
def wrap(self):
test = decorator(self.test)
test()
Test().wrap()
# () {}
I'm guessing that it might be doing something during class initialization or class instantiation, like explicitly passing the self parameter?
I want to know about it, or how can I get the self parameter.
In the code below, I am using a metaclass along with a decorator to decorate all the user defined methods.
It works for all instance methods, but in cases of staticmethods it fails due to the self argument, to avoid that I am using a try and except block, which solves the problem. But in one of my projects, it's not working out.
Is there a better way of decorating the output of a staticmethod via a function decorator enclosed in a metaclass ?
from functools import wraps
import types
def decorator_function(input_function):
#wraps(input_function)
def wrapper(self, *args, **kwargs):
if kwargs.get("test_parameter"):
kwargs["test_parameter"] = 999
try:
result = input_function(self, *args, **kwargs)
except:
result = input_function(*args, **kwargs)
return result
return wrapper
class DecoratorMetaClass(type):
def __new__(meta, name, bases, class_dict):
klass = super().__new__(meta, name, bases, class_dict)
for key in dir(klass):
value = getattr(klass, key)
if isinstance(value, types.FunctionType) and "__" not in key:
wrapped = decorator_function(value)
setattr(klass, key, wrapped)
return klass
class InterfaceClass(metaclass=DecoratorMetaClass):
def function(self, test_parameter=1):
print(f"function - Test Parameter= {test_parameter}")
#staticmethod
def static_function(test_parameter=1):
print(f"static_function - Test Parameter= {test_parameter}")
class UserClass(InterfaceClass, metaclass=DecoratorMetaClass):
def __init__(self):
pass
def function_2(self, test_parameter=1):
print(f"function_2 - Test Parameter= {test_parameter}")
instance = UserClass()
instance.function(test_parameter=2)
instance.function_2(test_parameter=2)
instance.static_function(test_parameter=2)
print(isinstance(instance, InterfaceClass))
PS: I am not using a class decorator because it causes the isinstance checks to fail.
Explanation
The major problem goes down to the methods parameters. You were almost there.
You have to make the decorators arguments compatible to your methods parameters;
You can change the signature of the function wrapper from wrapper(self, *args, **kwargs) to wrapper(*args, **kwargs). Then just assign result = input_function(*args, **kwargs). You don't need the try/except block for this decorator;
def decorator_function(input_function):
#wraps(input_function)
def wrapper(*args, **kwargs):
if kwargs.get("test_parameter"):
kwargs["test_parameter"] = 999
return input_function(*args, **kwargs)
return wrapper
Ideally you should add to the methods *args (variable arguments) and **kwargs (variable named arguments) to make them compatible with your decorator;
In this case I added *args before the test_parameter=1 to the static_function in InterfaceClass.
class InterfaceClass(metaclass=DecoratorMetaClass):
#staticmethod
def static_function(*args, test_parameter=1):
print(f"static_function - Test Parameter= {test_parameter}")
Runnable Code
from functools import wraps
import types
def decorator_function(input_function):
#wraps(input_function)
def wrapper(*args, **kwargs):
if kwargs.get("test_parameter"):
kwargs["test_parameter"] = 999
return input_function(*args, **kwargs)
return wrapper
class DecoratorMetaClass(type):
def __new__(meta, name, bases, class_dict):
klass = super().__new__(meta, name, bases, class_dict)
for key in dir(klass):
value = getattr(klass, key)
if isinstance(value, types.FunctionType) and "__" not in key:
wrapped = decorator_function(value)
setattr(klass, key, wrapped)
return klass
class InterfaceClass(metaclass=DecoratorMetaClass):
def function(self, test_parameter=1):
print(f"function - Test Parameter= {test_parameter}")
#staticmethod
def static_function(*args, test_parameter=1):
print(f"static_function - Test Parameter= {test_parameter}")
class UserClass(InterfaceClass, metaclass=DecoratorMetaClass):
def __init__(self):
pass
def function_2(self, test_parameter=1):
print(f"function_2 - Test Parameter= {test_parameter}")
instance = UserClass()
instance.function(test_parameter=2)
instance.function_2(test_parameter=2)
instance.static_function(test_parameter=2)
UserClass.static_function(test_parameter=3)
print(isinstance(instance, InterfaceClass))
Output
function - Test Parameter= 999
function_2 - Test Parameter= 999
static_function - Test Parameter= 999
static_function - Test Parameter= 999
True
Addressing OP's comment
Considering test_parameter is always a named parameter, write the decorator_function as the following:
def decorator_function(input_function):
#wraps(input_function)
def wrapper(*args, **kwargs):
if kwargs.get("test_parameter"):
kwargs["test_parameter"] = 999
try:
result = input_function(*args, **kwargs)
except TypeError:
result = input_function(**kwargs)
return result
return wrapper
This way you don't need to change the methods signature.
If you call the functions also with positional arguments, you will need to check the type of the first argument inserted into args. Things get complicated and error prone.
The below is an attempt to implement Singleton in python 3 but it doesn't appear to work. When I instantiate, the _instance is always None and both instances (a and b) have different addresses in memory - why?
class Singleton(object):
_instance = None
def __call__(self, *args, **kwargs):
if self._instance is None:
self._instance = super().__call__(*args, **kwargs)
return self._instance
def __init__(self, *args, **kwargs):
print(self._instance, self)
a = Singleton()
b = Singleton()
The output is:
(None, <__main__.Singleton object at 0x7f382956c190>)
(None, <__main__.Singleton object at 0x7f382956c410>)
The __call__ method is not what you think it is. It is meant to make instances of classes callable like functions:
class A:
def __call__(self):
print("called")
a = A() # prints nothing
a() # prints "called"
What you are looking for is the __new__ method:
Called to create a new instance of class cls.
You can write the singleton like this (very similar to what you wrote):
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if cls._instance is None:
cls._instance = super(Singleton, cls).__new__(cls)
return cls._instance
def __init__(self, *args, **kwargs):
print(self._instance, self)
a = Singleton()
b = Singleton()
The output is now:
<__main__.Singleton object at 0x7f149bf3cc88> <__main__.Singleton object at 0x7f149bf3cc88>
<__main__.Singleton object at 0x7f149bf3cc88> <__main__.Singleton object at 0x7f149bf3cc88>
I got class Op:
class Pipeable(type):
def __get__(self, instance, owner):
def pipe_within(*args, **kwargs):
return self(*args, op=instance, **kwargs)
print('piping...')
return pipe_within
class Op(metaclass=Pipeable):
def __init__(self, op=None):
if op is not None:
print('piped!')
self.op = op
self.__dict__[type(self).__name__] = type(self)
I expect Op class itself to work as descriptor, because its metaclass has __get__ method, but the code
op = Op().Op()
doesn't invoke Op.__get__. Why?
It is hard to tell what you really want there. But a metaclass that would add a property to itself at every new class maybe works better for whatever you want.
As far as I can understand your code, older classes won't be populated with references to the newer ones, as you create new instances (that in turn, get the reference for others).
On a second though, dinamically creating properties inisde __new__ seems hacky - but you can just implement the metaclass __getattr__ and __dir__ methods for much less convoluted code:
The simple version works for classes, but not for their instances - because instances do not trigger the __getattr__ on the metaclass:
class Pipeable(type):
_classes = {}
def __new__(metacls, name, bases, namespace, **kwds):
cls = type.__new__(metacls, name, bases, namespace)
metacls._classes[name] = cls
return cls
def __getattr__(cls, attr):
classes = cls.__class__._classes
if attr not in classes:
raise AttributeError
def pipe_within(*args, **kwargs):
return cls(*args, op=classes[attr], **kwargs)
print('piping...')
return pipe_within
def __dir__(cls):
regular = super().__dir__()
return sorted(regular + list(cls.__class__._classes.keys()))
class Op(metaclass=Pipeable):
def __init__(self, op=None):
if op is not None:
print('piped!')
self.op = op
Op.Op()
(Note as well, that over time I picked this parameter naming convention to use on metaclasses - as most their methods take the class created with them in place of what is the "self" in ordinary classes, I find this naming easier to follow. It is not mandatory, not necessarily "correct", though)
But then, we can make it work for instances by creating the __dir__ and __getattr__ directly on the created classes as well. The catch with that is that the class you are creating already have a __getattr__ or custom __dir__, even in their super-classes, those have to be wrapped. And then, we don't want to re-wrap our own __dir__ and __getattr__, so some extra-care:
class Pipeable(type):
_classes = {}
def __new__(metacls, name, bases, namespace, **kwds):
cls = type.__new__(metacls, name, bases, namespace)
metacls._classes[name] = cls
original__getattr__ = getattr(cls, "__getattr__", None)
if hasattr(original__getattr__, "_metapipping"):
# Do not wrap our own (metaclass) implementation of __getattr__
original__getattr__ = None
original__dir__ = getattr(cls, "__dir__") # Exists in "object", so it is always found.
# these two functions have to be nested so they can get the
# values for the originals "__getattr__" and "__dir__" from
# the closure. These values could be set on the class created, alternatively.
def __getattr__(self, attr):
if original__getattr__:
# If it is desired that normal attribute lookup have
# less precedence than these injected operators
# move this "if" block down.
try:
value = original__getattr__(self, attr)
except AttributeError:
pass
else:
return value
classes = self.__class__.__class__._classes
if attr not in classes:
raise AttributeError
def pipe_within(*args, **kwargs):
return cls(*args, op=classes[attr], **kwargs)
print('piping...')
return pipe_within
__getattr__._pipping = True
def __dir__(self):
regular = original__dir__(self)
return sorted(regular + list(self.__class__.__class__._classes.keys()))
__dir__.pipping = True
if not original__getattr__ or not hasattr(original__getattr__, "_pipping"):
cls.__getattr__ = __getattr__
if not hasattr(original__dir__, "_pipping"):
cls.__dir__ = __dir__
return cls
def __getattr__(cls, attr):
classes = cls.__class__._classes
if attr not in classes:
raise AttributeError
def pipe_within(*args, **kwargs):
return cls(*args, op=classes[attr], **kwargs)
print('piping...')
return pipe_within
__getattr__._metapipping = True
def __dir__(cls):
regular = super().__dir__()
return sorted(regular + list(cls.__class__._classes.keys()))
class Op(metaclass=Pipeable):
def __init__(self, op=None):
if op is not None:
print('piped!')
Op().Op()
So, this ended up being lengthy - but it "does the right thing", by ensuring all classes and instances in the hierarchy can see each other, regardless of creation order.
Also, what make up for the complexity is correctly wrapping other possible customizations of __getattr__ and __dir__ in the class hierarchy - if you don't get any customization of those, this can be an order of magnitude simpler:
class Pipeable(type):
_classes = {}
def __new__(metacls, name, bases, namespace, **kwds):
cls = type.__new__(metacls, name, bases, namespace)
metacls._classes[name] = cls
def __getattr__(self, attr):
classes = self.__class__.__class__._classes
if attr not in classes:
raise AttributeError
def pipe_within(*args, **kwargs):
return cls(*args, op=classes[attr], **kwargs)
print('piping...')
return pipe_within
def __dir__(self):
regular = original__dir__(self)
return sorted(regular + list(self.__class__.__class__._classes.keys()))
cls.__getattr__ = __getattr__
cls.__dir__ = __dir__
return cls
def __getattr__(cls, attr):
classes = cls.__class__._classes
if attr not in classes:
raise AttributeError
def pipe_within(*args, **kwargs):
return cls(*args, op=classes[attr], **kwargs)
print('piping...')
return pipe_within
def __dir__(cls):
regular = super().__dir__()
return sorted(regular + list(cls.__class__._classes.keys()))
To get into work, descriptor must be class attribute, not that of instance.
This code does what was desired.
class Pipeable(type):
_instances = {}
def __new__(cls, name, bases, namespace, **kwds):
namespace.update(cls._instances)
instance = type.__new__(cls, name, bases, namespace)
cls._instances[name] = instance
for inst in cls._instances:
setattr(inst, name, instance)
return instance
def __get__(self, instance, owner):
def pipe_within(*args, **kwargs):
return self(*args, op=instance, **kwargs)
print('piping...')
return pipe_within
class Op(metaclass=Pipeable):
def __init__(self, op=None):
if op is not None:
print('piped!')
self.op = op
Op().Op()
I have a method decorator looking like
def debug_run(fn):
from functools import wraps
#wraps(fn)
def wrapper(self, *args, **kw):
# log some stuff
# timeit fn
res = fn(self, *args, **kw)
return wrapper
Right now I used to use it apply on each method that I want to debug. Now i'm trying to apply to all class method using a class decorator looking like.
Rather doing
class A():
#debug_run
def f(self):
pass
I do
#decallmethods(debug_run)
class A():
def f(self):
pass
def decallmethods(decorator):
def dectheclass(cls):
for name, m in inspect.getmembers(cls, inspect.ismethod):
if name in getattr(cls, 'METHODS_TO_INSPECT', []):
setattr(cls, name, decorator(m))
return cls
return dectheclass
Trying to apply to decorator to the base class, not working as expected. no log to the console. Now i wonder if this approach is the good or I should used something else (apply the debug decorator to selected method from base class to all sub classes).
[EDIT]
Finally found why no logs were printed
Why is there a difference between inspect.ismethod and inspect.isfunction from python 2 -> 3?
Here a complete example reflecting my code
import inspect
import time
import logging as logger
from functools import wraps
logger.basicConfig(format='LOGGER - %(asctime)s %(message)s', level=logger.DEBUG)
def debug_run(fn):
#wraps(fn)
def wrapper(self, *args, **kw):
logger.debug(
"call method %s of instance %s with %r and %s "
% (fn.__name__, self, args, kw))
time1 = time.time()
res = fn(self, *args, **kw)
time2 = time.time()
logger.debug(
"%s function %0.3f ms" % (fn, (time2-time1)*1000.0))
return res
return wrapper
def decallmethods(decorator):
def dectheclass(cls):
for name, m in inspect.getmembers(
cls, predicate=lambda x: inspect.isfunction(x) or inspect.ismethod(x)):
methods_to_inspect = getattr(cls, 'METHODS_TO_INSPECT', [])
if name in methods_to_inspect:
setattr(cls, name, decorator(m))
return cls
return dectheclass
class B(object):
METHODS_TO_INSPECT = ["bfoo1", "bfoo2", "foo"]
def __str__(self):
return "%s:%s" % (repr(self), id(self))
def bfoo1(self):
pass
def bfoo2(self):
pass
def foo(self):
pass
def run(self):
print("print - Base run doing nothing")
class C(object):
pass
#decallmethods(debug_run)
class A(B, C):
METHODS_TO_INSPECT = ["bfoo1", "bfoo2", "foo", "run"]
def foo(self):
print("print - A foo")
def run(self):
self.bfoo1()
self.bfoo2()
self.foo()
a = A()
b = B()
a.run()
b.run()
In this case applying decallmethods to B, will not affect the A so i must to apply to both A and B thus to all sub classes of B.
It is possible to have such mechanism that permit to apply decallmethods to all sub classes methods ?
look at this:
How can I decorate all functions of a class without typing it over and over for each method added? Python
delnan has a good answer,
only add this rule to his answer
if name in getattr(cls, 'METHODS_TO_INSPECT', []):