I'm trying to redifine the __getattr__ method, in a function.
I've tried this code:
def foo():
print("foo")
def addMethod(obj, func):
setattr(obj, func.__name__, types.MethodType(func, obj))
def __getattr__(obj, name):
print(name)
addMethod(foo, __getattr__)
foo.bar
but I get this error:
Traceback (most recent call last):
File blah blah, line 14, in <module>
foo.bar
AttributeError: 'function' object has no attribute 'bar'
I've inspected the "foo" function and it really has the method bounded to it, but it seems that if you set it dynamically, __getattr__ won't get called.
If I do the same thing to a class, if I set the __getattr__ using my "addMethod" function, the instance won't call the __getattr__ too!, so the problem must be the dynamic call!
BUT if I put the __getattr__ in the definition of the class, it will work, obviously.
The question is, how can I put the __getattr__ to the function to make it work? I can't put it from the beginning because it's a function!, I don't know how to do this
Thanks!
Well, you don't. If you want attributes, make a class. If you want instances to be callable, define __call__ for it.
class foo:
def __call__(self):
print("foo")
def __getattr__(self, name):
print(name)
f = foo()
f() # foo
f.bar # bar
Your problem is that Python will only look up __xxx__ magic methods on the class of an object, not the object itself. So even though you are setting __getattr__ on the instance of foo, it will never be automatically called.
That is also the problem you are having with class foo:
class foo():
pass
def addMethod(obj, func):
setattr(obj, func.__name__, func)
def __getattr__(obj, name):
print(name)
addMethod(foo, __getattr__)
foo.bar
Here Python is looking up bar on the class object foo which means . . .
the __getattr__ method you added to foo is not getting called, instead Python is looking at foo's metaclass, which is type.
If you change that last line to
foo().bar
and get an instance of the class of foo, it will work.
Related
I am trying to understand when to initialize a superclass when using inheritance in python. Initially I thought that just by declaring a class inheriting from a super class, ex. class my_class(superclass):, would make available all the superclass's attributes and methods to the subclass. Which makes sense for somebody coming from Java. Then I read that Python forces us to initialize superclasses before we can implement them in our subclass, either by using the superclass.init() or super().init(). Then I came across this piece of code where I am not initializing the parent's class, however Python gave me access to the self.queue attribute from superclass without having initialized the parent class. I read the Python documentation and sometimes I think I know what they mean and some other I dont. Can anyone please explain to me when do we have to initialize superclasses in our subclasses?
class QueueError(IndexError):
pass
class Queue:
def __init__(self):
self.queue = []
def put(self,elem):
self.queue.insert(0,elem)
def get(self):
if len(self.queue) > 0:
elem = self.queue[-1]
del self.queue[-1]
return elem
else:
raise QueueError
class SuperQueue(Queue):
def isempty(self):
if not self.queue:
return True
else:
return False
que = SuperQueue()
que.put(1)
que.put("dog")
que.put(False)
for i in range(4):
if not que.isempty():
print(que.get())
else:
print("Queue empty")
In general, if you override __init__ in your subclass, you should call the super __init__ method only. This is necessary if you extend the superclass. But if you want to overwrite the whole __init__, you can also omit the super call.
Example:
You have class A that has one attribute value1.
And now you need a second attribute, so you subclass A with B and overwrite the __init__ where you call the super class (A), so A can set value1 and in B you can not set value2.
But now you need some other attributes in C, but need the same methods as in A. So you can entirely overwrite __init__ and omit the super call to A.
class A:
def __init__(self, value1):
print("Initialize A")
self.value1 = value1
class B(A):
def __init__(self, value1, value2):
super().__init__(value1)
print("Initialize B")
self.value2 = value2
class C(A):
def __init__(self, value3):
print("Initialize C")
self.value3 = value3
a = A("foo")
b = B("foo", "bar")
c = C("baz")
print(a.value1)
print(b.value1, b.value2)
print(c.value3)
print(c.value1)
Output
$ python main.py
Initialize A
Initialize A
Initialize B
Initialize C
foo
foo bar
baz
Traceback (most recent call last):
File "main.py", line 27, in <module>
print(c.value1)
AttributeError: 'C' object has no attribute 'value1'
You can see C wasn't initialized with value1, because C didn't call A's __init__.
Working with python3 I am a bit stumped on why __getattribute__ isn't working with my class structure.
I googled around but could not understand why it won't work.
Here is the no-frills class structure, removing all that was not required:
class Platform:
def execute(self):
print("Calling execute in Platform")
class Host():
def __init__(self):
self.plat = Platform()
def __getattr__(self, name):
if name in dir(self):
return self.__getattribute__(name)
if name in dir(self.plat):
return self.plat.__getattribute__(name)
class Connector(Host):
pass
class LDAP():
def __init__(self):
self.conn = Connector()
def __getattr__(self, name):
return self.conn.__getattribute__(name)
I have a working object ldp = LDAP(). Calling ldp.execute() raises AttributeError: 'Connector' object has no attribute 'execute'
I would expect calling ldp.execute() would do the following:
Call __getattr__ from LDAP which calls __getattribute__ on Connector object
That would again call __getattr__ on Host which would call __getattribute__ on Platform object
That would eventually call execute method from Platform
And ofcourse that isn't working as expected :)
If however, I call self.conn.__getattr__(name) from __getattr__ in LDAP class, that works, maybe because I have __getattr__ defined in Host.
What am I missing here ?
Note: I can't modify the Platform and Host classes. Only Connector and LDAP are mine
It's generally a mistake to call __getattribute__ directly. Instead, use the getattr builtin:
class LDAP():
def __init__(self):
self.conn = Connector()
def __getattr__(self, name):
return getattr(self.conn, name)
The getattr function knows to look for __getattr__, while __getattribute__ does not.
Say I have a class:
class Example:
__slots__ = ("_attrs", "other_value")
def __init__(self):
self._attrs = OrderedDict()
self.other_value = 1
self.attribute = 0
def __setattr__(self, key, value):
if key in self.__slots__:
return super().__setattr__(key, value)
else:
self._attrs[key] = value
def __getattr__(self, key):
return self._attrs[key]
The goal is to have Example have two slots:
if those are set, then set them as usual. (works)
If additional attributes are set, assign them in _attrs. (works)
For getting attributes, the code should:
Act as usual if anything from slots is requested (works)
get the value from _attrs if it exists in _attrs.keys() (works)
error in any other case as usual (issue).
For the issue, I'd like the error to mimic what would normally happen if an attribute was not present for an object. Currently when running code I get a key error on self._attrs. Although this is fine, it would be nice for it to hide this nuance away. More annoyingly, if I debug in Pycharm, the autocomplete will chuck out a large error trying to look at dict before I've even hit enter:
Example().abc # hit tab in pycharm
# returns the error:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 1464, in do_it
def do_it(self, dbg):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/_pydev_completer.py", line 159, in generate_completions_as_xml
def generate_completions_as_xml(frame, act_tok):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/_pydev_completer.py", line 77, in complete
def complete(self, text):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/_pydev_completer.py", line 119, in attr_matches
def attr_matches(self, text):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/_pydev_imports_tipper.py", line 165, in generate_imports_tip_for_module
def generate_imports_tip_for_module(obj_to_complete, dir_comps=None, getattr=getattr, filter=lambda name:True):
File "/Users/xxxxxxxxx/", line 46, in __getattr__
def __getattr__(self, key: str) -> None:
KeyError: '__dict__'
Is there a way to suppress this by writing the code differently?
You might be able to make it work by implementing __dir__ on the class, so it has a canonical source of names that can be completed:
def __dir__(self):
return 'other_value', *self._attrs.keys()
I can't swear to how PyCharm implements their tab-completion, so there's no guarantee it works, but this is the way to define the set of enumerable attributes for a type, and hopefully PyCharm will use it when available, rather than going for __dict__.
The other approach (and this is probably a good idea regardless) it to make sure you raise the right error when __getattr__ fails so PyCharm knows the problem is a missing attribute, not some unrelated issue with a dict:
def __getattr__(self, key):
try:
return self._attrs[key]
except KeyError:
raise AttributeError(key)
I have a superclass that has a retrieve() method, and its subclasses each implement their own retrieve() method. I'd like every retrieve() method to be decorated to cache the return value when it receive the same args, without having to decorate the method in every subclass.
Decorators don't seem to be inherited. I could probably call the superclass's method which would in turn set the cache, but currently my superclass raises a NotImplemented exception, which I like.
import json
import operator
from cachetools import cachedmethod, TTLCache
def simple_decorator(func):
def wrapper(*args, **kwargs):
#check cache
print("simple decorator")
func(*args, **kwargs)
#set cache
return wrapper
class AbstractInput(object):
def __init__(self, cacheparams = {'maxsize': 10, 'ttl': 300}):
self.cache = TTLCache(**cacheparams)
super().__init__()
#simple_decorator
def retrieve(self, params):
print("AbstractInput retrieve")
raise NotImplementedError("AbstractInput inheritors must implement retrieve() method")
class JsonInput(AbstractInput):
def retrieve(self, params):
print("JsonInput retrieve")
return json.dumps(params)
class SillyJsonInput(JsonInput):
def retrieve(self, params):
print("SillyJsonInput retrieve")
params["silly"] = True
return json.dumps(params)
Actual results:
>>> ai.retrieve(params)
ai.retrieve(params)
simple decorator
AbstractInput retrieve
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 8, in wrapper
File "<string>", line 22, in retrieve
NotImplementedError: AbstractInput inheritors must implement retrieve() method
>>> ji.retrieve(params)
ji.retrieve(params)
JsonInput retrieve
'{"happy": "go lucky", "angry": "as a wasp"}'
Desired results:
>>> ai.retrieve(params)
ai.retrieve(params)
simple decorator
AbstractInput retrieve
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 8, in wrapper
File "<string>", line 22, in retrieve
NotImplementedError: AbstractInput inheritors must implement retrieve() method
>>> ji.retrieve(params)
simple decorator
ji.retrieve(params)
JsonInput retrieve
'{"happy": "go lucky", "angry": "as a wasp"}'
Yes, the use of a metaclass to force a decorator on an specific method, as you put in your own answer is correct. With a few changes, it can be made so that the method to be decorated is not fixed - for example, an attribute set in the decorated function can be used as a "mark" that such a decorator should be forced upon overriding methods.
Besides that, since Python 3.6, there is a new class level mechanism - the special method __init_subclass__, which has the specific objective of diminishing the need for metaclasses. Metaclasses can be complicated, and if your class hierarchy needs to combine more than one metaclass, you may be in for some headache.
The __init_subclass__ method is placed on the base class, and it is called once each time a child class is created. The wrapping logic can be put there.
Basically, you can just modify your decorator to put the mark I mentioned above, and add this class in your inheritance hierarchy - it can be put as mixin class in multiple inheritance, so it can be reused for various class-trees, if needed:
def simple_decorator(func):
def wrapper(*args, **kwargs):
print("check cache")
rt = func(*args, **kwargs)
print("set cache")
return rt
wrapper.inherit_decorator = simple_decorator
return wrapper
class InheritDecoratorsMixin:
def __init_subclass__(cls, *args, **kwargs):
super().__init_subclass__(*args, **kwargs)
decorator_registry = getattr(cls, "_decorator_registry", {}).copy()
cls._decorator_registry = decorator_registry
# Check for decorated objects in the mixin itself- optional:
for name, obj in __class__.__dict__.items():
if getattr(obj, "inherit_decorator", False) and not name in decorator_registry:
decorator_registry[name] = obj.inherit_decorator
# annotate newly decorated methods in the current subclass:
for name, obj in cls.__dict__.items():
if getattr(obj, "inherit_decorator", False) and not name in decorator_registry:
decorator_registry[name] = obj.inherit_decorator
# finally, decorate all methods anottated in the registry:
for name, decorator in decorator_registry.items():
if name in cls.__dict__ and getattr(getattr(cls, name), "inherit_decorator", None) != decorator:
setattr(cls, name, decorator(cls.__dict__[name]))
So, that is it - each new subclass will have its own _decorator_registry attribute, where the name of the decorated methods in all ancestors, along with which decorator to apply is annotated.
If the decorator should be used one single time for the method, and not be repeated when the overridden method performs the super() call for its ancestors (not the case when you are decorating for cache, since the super-methods won't be called) that gets trickier - but can be done.
However, it is tricky to do - as the decorator instances in the superclasses would be other instances than the decorator on the subclass - one way to pass information to then that the "decorator code for this method is already run in this chain call" is to use an instance-level marker - which should be a thread-local variable if the code is to support parallelism.
All this checking will result in quite some complicated boilerplate to put into what could be a simple decorator - so we can create a "decorator" for the "decorators" that we want to run a single time. In other wors, decoratos decorated with childmost bellow will run only on the "childmost" class, but not on the corresponding methods in the superclasses when they call super()
import threading
def childmost(decorator_func):
def inheritable_decorator_that_runs_once(func):
decorated_func = decorator_func(func)
name = func.__name__
def wrapper(self, *args, **kw):
if not hasattr(self, f"_running_{name}"):
setattr(self, f"_running_{name}", threading.local())
running_registry = getattr(self, f"_running_{name}")
try:
if not getattr(running_registry, "running", False):
running_registry.running = True
rt = decorated_func(self, *args, **kw)
else:
rt = func(self, *args, **kw)
finally:
running_registry.running = False
return rt
wrapper.inherit_decorator = inheritable_decorator_that_runs_once
return wrapper
return inheritable_decorator_that_runs_once
Example using the first listing:
class A: pass
class B(A, InheritDecoratorsMixin):
#simple_decorator
def method(self):
print(__class__, "method called")
class C(B):
def method(self):
print(__class__, "method called")
super().method()
And after pasting the listing-1 and these A=B-C class in the
interpreter, the result is this:
In [9]: C().method()
check cache
<class '__main__.C'> method called
check cache
<class '__main__.B'> method called
set cache
set cache
(the "A" class here is entirely optional and can be left out)
Example using the second listing:
# Decorating the same decorator above:
#childmost
def simple_decorator2(func):
def wrapper(*args, **kwargs):
print("check cache")
rt = func(*args, **kwargs)
print("set cache")
return rt
return wrapper
class D: pass
class E(D, InheritDecoratorsMixin):
#simple_decorator2
def method(self):
print(__class__, "method called")
class F(E):
def method(self):
print(__class__, "method called")
super().method()
And the result:
In [19]: F().method()
check cache
<class '__main__.F'> method called
<class '__main__.E'> method called
set cache
OK, it seems that I can "decorate" a method in a superclass and have the subclasses also inherit that decoration, even if the method is overwritten in the subclass, using metaclasses. In this case, I'm decorating all "retrieve" methods in AbstractInput and its subclasses with simple_decorator using a metaclass named CacheRetrieval.
def simple_decorator(func):
def wrapper(*args, **kwargs):
print("check cache")
rt = func(*args, **kwargs)
print("set cache")
return rt
return wrapper
class CacheRetrieval(type):
def __new__(cls, name, bases, attr):
# Replace each function with
# a print statement of the function name
# followed by running the computation with the provided args and returning the computation result
attr["retrieve"] = simple_decorator(attr["retrieve"])
return super(CacheRetrieval, cls).__new__(cls, name, bases, attr)
class AbstractInput(object, metaclass= CacheRetrieval):
def __init__(self, cacheparams = {'maxsize': 10, 'ttl': 300}):
self.cache = TTLCache(**cacheparams)
super().__init__()
def retrieve(self, params):
print("AbstractInput retrieve")
raise NotImplementedError("DataInput must implement retrieve() method")
class JsonInput(AbstractInput):
def retrieve(self, params):
print("JsonInput retrieve")
return json.dumps(params)
class SillyJsonInput(JsonInput):
def retrieve(self, params):
print("SillyJsonInput retrieve")
params["silly"] = True
return json.dumps(params)
I was helped by this page:
https://stackabuse.com/python-metaclasses-and-metaprogramming/
If I have a class ...
class MyClass:
def method(arg):
print(arg)
... which I use to create an object ...
my_object = MyClass()
... on which I call method("foo") like so ...
>>> my_object.method("foo")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method() takes exactly 1 positional argument (2 given)
... why does Python tell me I gave it two arguments, when I only gave one?
In Python, this:
my_object.method("foo")
... is syntactic sugar, which the interpreter translates behind the scenes into:
MyClass.method(my_object, "foo")
... which, as you can see, does indeed have two arguments - it's just that the first one is implicit, from the point of view of the caller.
This is because most methods do some work with the object they're called on, so there needs to be some way for that object to be referred to inside the method. By convention, this first argument is called self inside the method definition:
class MyNewClass:
def method(self, arg):
print(self)
print(arg)
If you call method("foo") on an instance of MyNewClass, it works as expected:
>>> my_new_object = MyNewClass()
>>> my_new_object.method("foo")
<__main__.MyNewClass object at 0x29045d0>
foo
Occasionally (but not often), you really don't care about the object that your method is bound to, and in that circumstance, you can decorate the method with the builtin staticmethod() function to say so:
class MyOtherClass:
#staticmethod
def method(arg):
print(arg)
... in which case you don't need to add a self argument to the method definition, and it still works:
>>> my_other_object = MyOtherClass()
>>> my_other_object.method("foo")
foo
In simple words
In Python you should add self as the first parameter to all defined methods in classes:
class MyClass:
def method(self, arg):
print(arg)
Then you can use your method according to your intuition:
>>> my_object = MyClass()
>>> my_object.method("foo")
foo
For a better understanding, you can also read the answers to this question: What is the purpose of self?
Something else to consider when this type of error is encountered:
I was running into this error message and found this post helpful. Turns out in my case I had overridden an __init__() where there was object inheritance.
The inherited example is rather long, so I'll skip to a more simple example that doesn't use inheritance:
class MyBadInitClass:
def ___init__(self, name):
self.name = name
def name_foo(self, arg):
print(self)
print(arg)
print("My name is", self.name)
class MyNewClass:
def new_foo(self, arg):
print(self)
print(arg)
my_new_object = MyNewClass()
my_new_object.new_foo("NewFoo")
my_bad_init_object = MyBadInitClass(name="Test Name")
my_bad_init_object.name_foo("name foo")
Result is:
<__main__.MyNewClass object at 0x033C48D0>
NewFoo
Traceback (most recent call last):
File "C:/Users/Orange/PycharmProjects/Chapter9/bad_init_example.py", line 41, in <module>
my_bad_init_object = MyBadInitClass(name="Test Name")
TypeError: object() takes no parameters
PyCharm didn't catch this typo. Nor did Notepad++ (other editors/IDE's might).
Granted, this is a "takes no parameters" TypeError, it isn't much different than "got two" when expecting one, in terms of object initialization in Python.
Addressing the topic: An overloading initializer will be used if syntactically correct, but if not it will be ignored and the built-in used instead. The object won't expect/handle this and the error is thrown.
In the case of the sytax error: The fix is simple, just edit the custom init statement:
def __init__(self, name):
self.name = name
Newcomer to Python, I had this issue when I was using the Python's ** feature in a wrong way. Trying to call this definition from somewhere:
def create_properties_frame(self, parent, **kwargs):
using a call without a double star was causing the problem:
self.create_properties_frame(frame, kw_gsp)
TypeError: create_properties_frame() takes 2 positional arguments but 3 were given
The solution is to add ** to the argument:
self.create_properties_frame(frame, **kw_gsp)
As mentioned in other answers - when you use an instance method you need to pass self as the first argument - this is the source of the error.
With addition to that,it is important to understand that only instance methods take self as the first argument in order to refer to the instance.
In case the method is Static you don't pass self, but a cls argument instead (or class_).
Please see an example below.
class City:
country = "USA" # This is a class level attribute which will be shared across all instances (and not created PER instance)
def __init__(self, name, location, population):
self.name = name
self.location = location
self.population = population
# This is an instance method which takes self as the first argument to refer to the instance
def print_population(self, some_nice_sentence_prefix):
print(some_nice_sentence_prefix +" In " +self.name + " lives " +self.population + " people!")
# This is a static (class) method which is marked with the #classmethod attribute
# All class methods must take a class argument as first param. The convention is to name is "cls" but class_ is also ok
#classmethod
def change_country(cls, new_country):
cls.country = new_country
Some tests just to make things more clear:
# Populate objects
city1 = City("New York", "East", "18,804,000")
city2 = City("Los Angeles", "West", "10,118,800")
#1) Use the instance method: No need to pass "self" - it is passed as the city1 instance
city1.print_population("Did You Know?") # Prints: Did You Know? In New York lives 18,804,000 people!
#2.A) Use the static method in the object
city2.change_country("Canada")
#2.B) Will be reflected in all objects
print("city1.country=",city1.country) # Prints Canada
print("city2.country=",city2.country) # Prints Canada
It occurs when you don't specify the no of parameters the __init__() or any other method looking for.
For example:
class Dog:
def __init__(self):
print("IN INIT METHOD")
def __unicode__(self,):
print("IN UNICODE METHOD")
def __str__(self):
print("IN STR METHOD")
obj = Dog("JIMMY", 1, 2, 3, "WOOF")
When you run the above programme, it gives you an error like that:
TypeError: __init__() takes 1 positional argument but 6 were given
How we can get rid of this thing?
Just pass the parameters, what __init__() method looking for
class Dog:
def __init__(self, dogname, dob_d, dob_m, dob_y, dogSpeakText):
self.name_of_dog = dogname
self.date_of_birth = dob_d
self.month_of_birth = dob_m
self.year_of_birth = dob_y
self.sound_it_make = dogSpeakText
def __unicode__(self, ):
print("IN UNICODE METHOD")
def __str__(self):
print("IN STR METHOD")
obj = Dog("JIMMY", 1, 2, 3, "WOOF")
print(id(obj))
If you want to call method without creating object, you can change method to static method.
class MyClass:
#staticmethod
def method(arg):
print(arg)
MyClass.method("i am a static method")
I get this error when I'm sleep-deprived, and create a class using def instead of class:
def MyClass():
def __init__(self, x):
self.x = x
a = MyClass(3)
-> TypeError: MyClass() takes 0 positional arguments but 1 was given
You should actually create a class:
class accum:
def __init__(self):
self.acc = 0
def accumulator(self, var2add, end):
if not end:
self.acc+=var2add
return self.acc
In my case, I forgot to add the ()
I was calling the method like this
obj = className.myMethod
But it should be is like this
obj = className.myMethod()