I am developing a Python3.8 project with usage of typing and dataclasses, and automatic tests include mypy. This brings me in a strange behavior that I do not really understand...
In short: Mypy seems not to understand dataclass attributes mechanics that, to my understanding, make them instance attributes.
Here's a minimal example, with a package and two modules:
__init__.py: void
app_events.py:
class AppEvent:
pass
main.py:
import dataclasses
import typing
from . import app_events
class B:
"""Class with *app_events* as instance attribute."""
def __init__(self):
self.app_events: typing.List[app_events.AppEvent] = []
def bar(self) -> app_events.AppEvent:
# no mypy complaint here: the import is correctly distinguished
# from the attribute
...
class C:
"""Class with *app_events* as class attribute."""
app_events: List[app_events.AppEvent]
def chew(self) -> app_events.AppEvent:
# mypy considers app_events to be the class attribute
...
#dataclasses.dataclass
class D:
app_events: typing.List[app_events.AppEvent] = \
dataclasses.field(default_factory=list)
def doo(self) -> app_events.AppEvent:
# same here: mypy considers app_events to be the class attribute
...
And the typecheck result:
PyCharm complains, for methods C.chew and D.doo: Unresolved attribute reference 'AppEvent' for class 'list'
mypy complains, still for methods C.chew and D.doo, that error: Name 'app_events.AppEvent' is not defined.
No issue for B.bar as written, though if app_events attribute is declared as a class attribute (instead of being defined in self.__init__, then mypy raise the same complaint.)
-> any idea how to understand/solve/circumvent this elegantly?
I'd really like not to rename my module and attributes, but if you have nice names in mind, please do not hesitate to propose :-)
Related
I have a class decorator, which adds a few functions and fields to decorated class.
#mydecorator
#dataclass
class A:
a: str = ""
Added (via setattr()) is a .save() function and a set of info for dataclass fields as a separate dict.
I'd like VScode and mypy to properly recognize that, so that when I use:
a=A()
a.save()
or a.my_fields_dict those 2 are properly recognized.
Is there any way to do that? Maybe modify class A type annotations at runtime?
TL;DR
What you are trying to do is not possible with the current type system.
1. Intersection types
If the attributes and methods you are adding to the class via your decorator are static (in the sense that they are not just known at runtime), then what you are describing is effectively the extension of any given class T by mixing in a protocol P. That protocol defines the method save and so on.
To annotate this you would need an intersection of T & P. It would look something like this:
from typing import Protocol, TypeVar
T = TypeVar("T")
class P(Protocol):
#staticmethod
def bar() -> str: ...
def dec(cls: type[T]) -> type[Intersection[T, P]]:
setattr(cls, "bar", lambda: "x")
return cls # type: ignore[return-value]
#dec
class A:
#staticmethod
def foo() -> int:
return 1
You might notice that the import of Intersection is conspicuously missing. That is because despite being one of the most requested features for the Python type system, it is still missing as of today. There is currently no way to express this concept in Python typing.
2. Class decorator problems
The only workaround right now is a custom implementation alongside a corresponding plugin for the type checker(s) of your choice. I just stumbled across the typing-protocol-intersection package, which does just that for mypy.
If you install that and add plugins = typing_protocol_intersection.mypy_plugin to your mypy configuration, you could write your code like this:
from typing import Protocol, TypeVar
from typing_protocol_intersection import ProtocolIntersection
T = TypeVar("T")
class P(Protocol):
#staticmethod
def bar() -> str: ...
def dec(cls: type[T]) -> type[ProtocolIntersection[T, P]]:
setattr(cls, "bar", lambda: "x")
return cls # type: ignore[return-value]
#dec
class A:
#staticmethod
def foo() -> int:
return 1
But here we run into the next problem. Testing this with reveal_type(A.bar()) via mypy will yield the following:
error: "Type[A]" has no attribute "bar" [attr-defined]
note: Revealed type is "Any"
Yet if we do this instead:
class A:
#staticmethod
def foo() -> int:
return 1
B = dec(A)
reveal_type(B.bar())
we get no complaints from mypy and note: Revealed type is "builtins.str". Even though what we did before was equivalent!
This is not a bug of the plugin, but of the mypy internals. It is another long-standing issue, that mypy does not handle class decorators correctly.
A person in that issue thread even mentioned your use case in conjunction with the desired intersection type.
DIY
In other words, you'll just have to wait until those two holes are patched. Or you can hope that at least the decorator issue by mypy is fixed soon-ish and write your own VSCode plugin for intersection types in the meantime. Maybe you can get together with the person behind that mypy plugin I mentioned above.
I'd like to type-annotate abstract class method witch behave as a constructor. For example in the code below, ElementBase.from_data is meant to be a abstract classmethod constructor.
tmp.py
from abc import abstractmethod, abstractclassmethod
import copy
from typing import TypeVar, Type
ElementT = TypeVar('ElementT', bound='ElementBase')
class ElementBase:
data: int
def __init__(self, data): self.data
##abstractmethod
def get_plus_one(self: ElementT) -> ElementT:
out = copy.deepcopy(self)
out.data = self.data + 1
return out
#abstractclassmethod
def from_data(cls: Type[ElementT], data: int) -> ElementT: # mypy error!!!
pass
class Concrete(ElementBase):
#classmethod
def from_data(cls, data: int) -> 'Concrete': # mypy error!!!
return cls(data)
However, applying mypy to this code shows the following erros.
tmp.py:18: error: The erased type of self "Type[tmp.ElementBase]" is not a supertype of its class "tmp.ElementBase"
tmp.py:23: error: Return type "Concrete" of "from_data" incompatible with return type <nothing> in supertype "ElementBase"
Do you have any idea to fix this error? Also, I'm specifically confused that the part of get_plus_one does not cause error, while only the part of abstractclassmethod does cause the error.
FYI, I want to make the abstract method constructor generic becaues I want to statically ensure that all subclass of ElementBase returns object with it's type when calling from_data.
[EDIT] comment out abstractmethod
It looks like mypy doesn't understand the abstractclassmethod decorator. That decorator has been deprecated since Python 3.3, as the abstractmethod and classmethod decorators were updated to play nice together. I think your code will work properly if you do:
#classmethod
#abstractmethod
def from_data(cls: Type[ElementT], data: int) -> ElementT:
pass
It's unrelated to your type checking issues, but you probably also want to change ElementBase to inherit from abc.ABC or to explicitly request the abc.ABCMeta metaclass if you want the abstractness of the class to be enforced by Python. Regular classes don't care about the abstractmethod decorator, and so as written, you'll be able to instantiate ElementBase (or you could if it's __init__ method didn't have an unrelated issue).
And another peripherally related note on this kind of type hinting... PEP 673 will add typing.Self in Python 3.11, which will be a convenient way for a method to refer to the type of object it's being called on. It should play nicely with classmethods without requiring you to jump through any hoops. With it you'd be able to write this much simpler version of the annotations:
#classmethod
#abstractmethod
def from_data(cls, data: int) -> Self:
pass
1. A problem with dataclass mix-ins, solved
To make abstract dataclasses that type-check under mypy, I've been breaking them into two classes, one that contains the abstract methods and one that contains the data members, as explained in this answer. The abstract class inherits from the dataclass. This runs into a problem, though, when another abstract-class-and-dataclass pair inherits from the first one: the "ancestor" dataclass's fields get wiped out by the "descendant". For example:
from dataclasses import dataclass
from abc import ABC, abstractmethod
#dataclass
class ADataclassMixin:
a_field: int = 1
class A(ADataclassMixin, ABC):
#abstractmethod
def method(self):
pass
#dataclass
#class BDataclassMixin(A): # works but fails mypy 0.931 type-check
class BDataclassMixin: # fails
b_field: int = 2
pass
class B(BDataclassMixin, A):
def method(self):
return self
o = B(a_field=5)
The last line fails, yielding this error message:
TypeError: BDataclassMixin.__init__() got an unexpected keyword argument 'a_field'
B's method-resolution order (B.__mro__) is (B, BDataclassMixin, A, ADataclassMixin, ABC, object), as expected. But a_field is not found.
A solution, shown in the commented-out line above, is to put the ancestor class explicitly in the descendant dataclass's declaration: class BDataclassMixin(A) instead of class BDataclassMixin. This fails type-checking, though, because a dataclass can only be a concrete class.
2. A problem with that solution, unsolved
The above solution breaks down if we add a third class, inheriting from B:
#dataclass
#class CDataclassMixin: # fails
class CDataclassMixin(A): # fails
#class CDataclassMixin(B, A): # works but fails type-check
c_field: int = 3
pass
class C(CDataclassMixin, B):
def method(self):
return "C's result"
pass
o = C(b_field=5)
Now, C has a_field and c_field but has lost b_field.
I have found that if I declare CDataclassMixin explicitly to inherit from B and A (in that order), b_field will be in the resulting class along with a_field_ and c_field`. However, explicitly stating the inheritance hierarchy in every mix-in defeats the purpose of mix-ins, which is to be able to code them independently of all the other mix-ins and to mix them easily and any way you like.
What is the correct way to make abstract dataclass mix-ins, so that classes that inherit from them include all the dataclass fields?
The correct solution is to abandon the DataclassMixin classes and simply make the abstract classes into dataclasses, like this:
#dataclass # type: ignore[misc]
class A(ABC):
a_field: int = 1
#abstractmethod
def method(self):
pass
#dataclass # type: ignore[misc]
class B(A):
b_field: int = 2
#dataclass
class C(B):
c_field: int = 3
def method(self):
return self
The reason for the failures is that, as explained in the documentation on dataclasses, the complete set of fields in a dataclass is determined when the dataclass is compiled, not when it is inherited from. The internal code that generates the dataclass's __init__ function can only examine the MRO of the dataclass as it is declared on its own, not when mixed in to another class.
It's necessary to add # type: ignore[misc] to each abstract dataclass's #dataclass line, not because the solution is wrong but because mypy is wrong. It is mypy, not Python, that requires dataclasses to be concrete. As explained by ilevkivskyi in mypy issue 5374, the problem is that mypy wants a dataclass to be a Type object and for every Type object to be capable of being instantiated. This is a known problem and awaits a resolution.
The behavior in the question and in the solution is exactly how dataclasses should behave. And, happily, abstract dataclasses that inherit this way (the ordinary way) can be mixed into other classes willy-nilly no differently than other mix-ins.
Putting the mixin as the last base class works without error:
#dataclass
class ADataclassMixin:
a_field: int = 1
class A(ABC, ADataclassMixin):
#abstractmethod
def method(self):
pass
#dataclass
class BDataclassMixin:
b_field: int = 2
class B(A, BDataclassMixin):
def method(self):
return self
o = B(a_field=5)
print((o.a_field, o.b_field)) # (5,2)
I am using importlib to import modules at runtime. These modules are plugins for my application and must implement 1 or more module-level functions. I have started adding type annotations to my applications and I get an error from mypy stating
Module has no attribute "generate_configuration"
where "generate_configuration" is one of the module functions.
In this example, the module is only required to have a generate_configuration function in it. The function takes a single dict argument.
def generate_configuration(data: Dict[str, DataFrame]) -> None: ...
I have been searching around for how to specify the interface of a module but all I can find are class interfaces. Can someone point me to some documentation showing how to do this? My google-fu is failing me on this one.
The code that loads this module is shown below. The error is generated by the last line.
plugin_directory = os.path.join(os.path.abspath(directory), 'Configuration-Generation-Plugins')
plugins = (
module_file
for module_file in Path(plugin_directory).glob('*.py')
)
sys.path.insert(0, plugin_directory)
for plugin in plugins:
plugin_module = import_module(plugin.stem)
plugin_module.generate_configuration(directory, points_list)
The type annotation for importlib.import_module simply returns types.ModuleType
From the typeshed source:
def import_module(name: str, package: Optional[str] = ...) -> types.ModuleType: ...
This means that the revealed type of plugin_module is Module -- which doesn't have your specific attributes.
Since mypy is a static analysis tool, it can't know that the return value of that import has a specific interface.
Here's my suggestion:
Make a type interface for your module (it doesn't have to be instantiated, it'll just help mypy figure things out)
class ModuleInterface:
#staticmethod
def generate_configuration(data: Dict[str, DataFrame]) -> None: ...
Make a function which imports your module, you may need to sprinkle # type: ignore, though if you use __import__ instead of import_module you may be able to avoid this limitation
def import_module_with_interface(modname: str) -> ModuleInterface:
return __import__(modname, fromlist=['_trash']) # might need to ignore the type here
Enjoy the types :)
The sample code I used to verify this idea:
class ModuleInterface:
#staticmethod
def compute_foo(bar: str) -> str: ...
def import_module_with_interface(modname: str) -> ModuleInterface:
return __import__(modname, fromlist=['_trash'])
def myf() -> None:
mod = import_module_with_interface('test2')
# mod.compute_foo() # test.py:12: error: Too few arguments for "compute_foo" of "ModuleInterface"
mod.compute_foo('hi')
I did some more research and eventually settled on a slightly different solution which uses typing.cast.
The solution still uses the static method definition from Anthony Sottile.
from typing import Dict
from pandas import DataFrame
class ConfigurationGenerationPlugin(ModuleType):
#staticmethod
def generate_configuration(directory: str, points_list: Dict[str, DataFrame]) -> None: ...
The code that imports the module then uses typing.cast() to set the correct type.
plugin_directory = os.path.join(os.path.abspath(directory), 'Configuration-Generation-Plugins')
plugins = (
module_file
for module_file in Path(plugin_directory).glob('*.py')
if not module_file.stem.startswith('lib')
)
sys.path.insert(0, plugin_directory)
for plugin in plugins:
plugin_module = cast(ConfigurationGenerationPlugin, import_module(plugin.stem))
plugin_module.generate_configuration(directory, points_list)
I am not sure how I feel about having to add the ConfigurationGenerationPlugin class or the cast() call to the code just to make mypy happy. However, I am going to stick with it for now.
I'm writing a library where I need a method that takes a (potentially) abstract type, and returns an instance of a concrete subtype of that type:
# script.py
from typing import Type
from abc import ABC, abstractmethod
class AbstractClass(ABC):
#abstractmethod
def abstract_method(self):
pass
T = TypeVar('T', bound=AbstractClass)
def f(c: Type[T]) -> T:
# find concrete implementation of c based on
# environment configuration
...
f(AbstractClass) # doesn't type check
Running mypy script.py yields:
error: Only concrete class can be given where "Type[AbstractClass]" is expected
I don't understand this error message and am having a hard time finding any documentation for it. Is there any way to annotate the function so that mypy will type check this?
As a side note, PyCharm's type checker, which is what I use the most, type checks f with no errors.
It does appear that mypy is a bit biased against using an abstract base class this way, though as you demonstrate there are valid use cases.
You can work around this by making your factory function a class method on your abstract class. If stylistically you'd like to have a top-level function as a factory, then you can create an alias to the class method.
from typing import TYPE_CHECKING
from abc import ABC, abstractmethod
class AbstractClass(ABC):
#abstractmethod
def abstract_method(self):
raise NotImplementedError
#classmethod
def make_concrete(cls) -> 'AbstractClass':
"""
find concrete implementation based on environment configuration
"""
return A()
class A(AbstractClass):
def abstract_method(self):
print("a")
# make alias
f = AbstractClass.make_concrete
x = f()
if TYPE_CHECKING:
reveal_type(x) # AbstractClass
Note that, without more work, mypy cannot know which concrete class is created by the factory function, it will only know that it is compatible with AbstractClass, as demonstrated by the output of reveal_type.
Alternately, if you're willing to give up the runtime checking provided by abc.ABC, you can get something even closer to your original design:
from typing import TYPE_CHECKING
from abc import abstractmethod
class AbstractClass: # do NOT inherit from abc.ABC
#abstractmethod
def abstract_method(self):
raise NotImplementedError
class A(AbstractClass):
def abstract_method(self):
print("a")
class Bad(AbstractClass):
pass
def f() -> AbstractClass:
"""
find concrete implementation based on environment configuration
"""
pass
b = Bad() # mypy displays an error here: Cannot instantiate abstract class 'Bad' with abstract attribute 'abstract_method'
x = f()
if TYPE_CHECKING:
reveal_type(x) # AbstractClass
This works because mypy checks methods marked with #abstractmethod even if the class does not inherit from abc.ABC. But be warned that if you execute the program using python, you will no longer get an error about instantiating the Bad class without implementing its abstract methods.