I am using importlib to import modules at runtime. These modules are plugins for my application and must implement 1 or more module-level functions. I have started adding type annotations to my applications and I get an error from mypy stating
Module has no attribute "generate_configuration"
where "generate_configuration" is one of the module functions.
In this example, the module is only required to have a generate_configuration function in it. The function takes a single dict argument.
def generate_configuration(data: Dict[str, DataFrame]) -> None: ...
I have been searching around for how to specify the interface of a module but all I can find are class interfaces. Can someone point me to some documentation showing how to do this? My google-fu is failing me on this one.
The code that loads this module is shown below. The error is generated by the last line.
plugin_directory = os.path.join(os.path.abspath(directory), 'Configuration-Generation-Plugins')
plugins = (
module_file
for module_file in Path(plugin_directory).glob('*.py')
)
sys.path.insert(0, plugin_directory)
for plugin in plugins:
plugin_module = import_module(plugin.stem)
plugin_module.generate_configuration(directory, points_list)
The type annotation for importlib.import_module simply returns types.ModuleType
From the typeshed source:
def import_module(name: str, package: Optional[str] = ...) -> types.ModuleType: ...
This means that the revealed type of plugin_module is Module -- which doesn't have your specific attributes.
Since mypy is a static analysis tool, it can't know that the return value of that import has a specific interface.
Here's my suggestion:
Make a type interface for your module (it doesn't have to be instantiated, it'll just help mypy figure things out)
class ModuleInterface:
#staticmethod
def generate_configuration(data: Dict[str, DataFrame]) -> None: ...
Make a function which imports your module, you may need to sprinkle # type: ignore, though if you use __import__ instead of import_module you may be able to avoid this limitation
def import_module_with_interface(modname: str) -> ModuleInterface:
return __import__(modname, fromlist=['_trash']) # might need to ignore the type here
Enjoy the types :)
The sample code I used to verify this idea:
class ModuleInterface:
#staticmethod
def compute_foo(bar: str) -> str: ...
def import_module_with_interface(modname: str) -> ModuleInterface:
return __import__(modname, fromlist=['_trash'])
def myf() -> None:
mod = import_module_with_interface('test2')
# mod.compute_foo() # test.py:12: error: Too few arguments for "compute_foo" of "ModuleInterface"
mod.compute_foo('hi')
I did some more research and eventually settled on a slightly different solution which uses typing.cast.
The solution still uses the static method definition from Anthony Sottile.
from typing import Dict
from pandas import DataFrame
class ConfigurationGenerationPlugin(ModuleType):
#staticmethod
def generate_configuration(directory: str, points_list: Dict[str, DataFrame]) -> None: ...
The code that imports the module then uses typing.cast() to set the correct type.
plugin_directory = os.path.join(os.path.abspath(directory), 'Configuration-Generation-Plugins')
plugins = (
module_file
for module_file in Path(plugin_directory).glob('*.py')
if not module_file.stem.startswith('lib')
)
sys.path.insert(0, plugin_directory)
for plugin in plugins:
plugin_module = cast(ConfigurationGenerationPlugin, import_module(plugin.stem))
plugin_module.generate_configuration(directory, points_list)
I am not sure how I feel about having to add the ConfigurationGenerationPlugin class or the cast() call to the code just to make mypy happy. However, I am going to stick with it for now.
Related
I have a class decorator, which adds a few functions and fields to decorated class.
#mydecorator
#dataclass
class A:
a: str = ""
Added (via setattr()) is a .save() function and a set of info for dataclass fields as a separate dict.
I'd like VScode and mypy to properly recognize that, so that when I use:
a=A()
a.save()
or a.my_fields_dict those 2 are properly recognized.
Is there any way to do that? Maybe modify class A type annotations at runtime?
TL;DR
What you are trying to do is not possible with the current type system.
1. Intersection types
If the attributes and methods you are adding to the class via your decorator are static (in the sense that they are not just known at runtime), then what you are describing is effectively the extension of any given class T by mixing in a protocol P. That protocol defines the method save and so on.
To annotate this you would need an intersection of T & P. It would look something like this:
from typing import Protocol, TypeVar
T = TypeVar("T")
class P(Protocol):
#staticmethod
def bar() -> str: ...
def dec(cls: type[T]) -> type[Intersection[T, P]]:
setattr(cls, "bar", lambda: "x")
return cls # type: ignore[return-value]
#dec
class A:
#staticmethod
def foo() -> int:
return 1
You might notice that the import of Intersection is conspicuously missing. That is because despite being one of the most requested features for the Python type system, it is still missing as of today. There is currently no way to express this concept in Python typing.
2. Class decorator problems
The only workaround right now is a custom implementation alongside a corresponding plugin for the type checker(s) of your choice. I just stumbled across the typing-protocol-intersection package, which does just that for mypy.
If you install that and add plugins = typing_protocol_intersection.mypy_plugin to your mypy configuration, you could write your code like this:
from typing import Protocol, TypeVar
from typing_protocol_intersection import ProtocolIntersection
T = TypeVar("T")
class P(Protocol):
#staticmethod
def bar() -> str: ...
def dec(cls: type[T]) -> type[ProtocolIntersection[T, P]]:
setattr(cls, "bar", lambda: "x")
return cls # type: ignore[return-value]
#dec
class A:
#staticmethod
def foo() -> int:
return 1
But here we run into the next problem. Testing this with reveal_type(A.bar()) via mypy will yield the following:
error: "Type[A]" has no attribute "bar" [attr-defined]
note: Revealed type is "Any"
Yet if we do this instead:
class A:
#staticmethod
def foo() -> int:
return 1
B = dec(A)
reveal_type(B.bar())
we get no complaints from mypy and note: Revealed type is "builtins.str". Even though what we did before was equivalent!
This is not a bug of the plugin, but of the mypy internals. It is another long-standing issue, that mypy does not handle class decorators correctly.
A person in that issue thread even mentioned your use case in conjunction with the desired intersection type.
DIY
In other words, you'll just have to wait until those two holes are patched. Or you can hope that at least the decorator issue by mypy is fixed soon-ish and write your own VSCode plugin for intersection types in the meantime. Maybe you can get together with the person behind that mypy plugin I mentioned above.
This is Python 3.10. My code is as follows:
from __future__ import annotations
from typing import Union
class Vehicle():
def __init__(self, components):
self.components = components
def getComponentWithFlag(self, flag: str) -> Union[Component,None]:
for component in self.components:
if getattr(component,flag,None):
return component
return None
class Component():
pass
class PassengerComponent(Component):
def __init__(self):
self.carriesPassengers = True
def ejectPassenger(self):
print('A passenger is tossed outside!')
class FreightComponent(Component):
def __init__(self):
self.carriesFreight = True
VW_Mini = Vehicle(components= [PassengerComponent()])
VW_Passat = Vehicle(components= [PassengerComponent(), FreightComponent()])
Truck = Vehicle(components= [FreightComponent()])
assert VW_Mini.getComponentWithFlag('carriesPassengers')
assert not VW_Mini.getComponentWithFlag('carriesFreight')
assert Truck.getComponentWithFlag('carriesFreight')
assert not Truck.getComponentWithFlag('carriesPassengers')
component = VW_Mini.getComponentWithFlag('carriesPassengers')
component.ejectPassenger()
Last line gives me a warning in PyCharm: Cannot find reference 'ejectPassenger' in 'Component | None'. I understand why it happens: there is no ejectPassenger method in Component class. Clearly the problem lies in how I typehint Vehicle.getComponentWithFlag method. Could you guys tell me how I should type hint its return object?
I know the object returned by that function:
may be None (if there's no appropriate Component),
may be an object of a subclass inheriting from Component class,
will never actually be an object of Component class itself.
Type hinting it explicitly like this: def getComponentWithFlag(self, flag: str) -> Union[PassengerComponent, FreightComponent, None] will not fly, because I will eventually have dozens of Components in my actual use-case and I would prefer to avoid typing them all out.
I would prefer to avoid typing them all out.
Unfortunately, I don't think it is possible, as you can't exclude root type only. (i.e. You can't hint all subclasses of T without T, as T is also subclass of T in static typing)
Instead, you can overload Vehicle.getComponentWithFlag.
from __future__ import annotations
from typing import Optional, Literal, Union, overload
class Vehicle():
def __init__(self, components):
self.components = components
#overload
def getComponentWithFlag(self, flag: Literal["carriesPassengers"]) -> Optional[PassengerComponent]:
...
#overload
def getComponentWithFlag(self, flag: Literal["carriesFreight"]) -> Optional[FreightComponent]:
...
def getComponentWithFlag(self, flag: str) -> Union[Component, None]:
for component in self.components:
if getattr(component,flag,None):
return component
return None
Try this code in your IDE. This code has limitation that you have to maintain overloaded variants as number of subclasses increases. However, I think this is optimal for now.
I'd like to type-annotate abstract class method witch behave as a constructor. For example in the code below, ElementBase.from_data is meant to be a abstract classmethod constructor.
tmp.py
from abc import abstractmethod, abstractclassmethod
import copy
from typing import TypeVar, Type
ElementT = TypeVar('ElementT', bound='ElementBase')
class ElementBase:
data: int
def __init__(self, data): self.data
##abstractmethod
def get_plus_one(self: ElementT) -> ElementT:
out = copy.deepcopy(self)
out.data = self.data + 1
return out
#abstractclassmethod
def from_data(cls: Type[ElementT], data: int) -> ElementT: # mypy error!!!
pass
class Concrete(ElementBase):
#classmethod
def from_data(cls, data: int) -> 'Concrete': # mypy error!!!
return cls(data)
However, applying mypy to this code shows the following erros.
tmp.py:18: error: The erased type of self "Type[tmp.ElementBase]" is not a supertype of its class "tmp.ElementBase"
tmp.py:23: error: Return type "Concrete" of "from_data" incompatible with return type <nothing> in supertype "ElementBase"
Do you have any idea to fix this error? Also, I'm specifically confused that the part of get_plus_one does not cause error, while only the part of abstractclassmethod does cause the error.
FYI, I want to make the abstract method constructor generic becaues I want to statically ensure that all subclass of ElementBase returns object with it's type when calling from_data.
[EDIT] comment out abstractmethod
It looks like mypy doesn't understand the abstractclassmethod decorator. That decorator has been deprecated since Python 3.3, as the abstractmethod and classmethod decorators were updated to play nice together. I think your code will work properly if you do:
#classmethod
#abstractmethod
def from_data(cls: Type[ElementT], data: int) -> ElementT:
pass
It's unrelated to your type checking issues, but you probably also want to change ElementBase to inherit from abc.ABC or to explicitly request the abc.ABCMeta metaclass if you want the abstractness of the class to be enforced by Python. Regular classes don't care about the abstractmethod decorator, and so as written, you'll be able to instantiate ElementBase (or you could if it's __init__ method didn't have an unrelated issue).
And another peripherally related note on this kind of type hinting... PEP 673 will add typing.Self in Python 3.11, which will be a convenient way for a method to refer to the type of object it's being called on. It should play nicely with classmethods without requiring you to jump through any hoops. With it you'd be able to write this much simpler version of the annotations:
#classmethod
#abstractmethod
def from_data(cls, data: int) -> Self:
pass
I am developing a Python3.8 project with usage of typing and dataclasses, and automatic tests include mypy. This brings me in a strange behavior that I do not really understand...
In short: Mypy seems not to understand dataclass attributes mechanics that, to my understanding, make them instance attributes.
Here's a minimal example, with a package and two modules:
__init__.py: void
app_events.py:
class AppEvent:
pass
main.py:
import dataclasses
import typing
from . import app_events
class B:
"""Class with *app_events* as instance attribute."""
def __init__(self):
self.app_events: typing.List[app_events.AppEvent] = []
def bar(self) -> app_events.AppEvent:
# no mypy complaint here: the import is correctly distinguished
# from the attribute
...
class C:
"""Class with *app_events* as class attribute."""
app_events: List[app_events.AppEvent]
def chew(self) -> app_events.AppEvent:
# mypy considers app_events to be the class attribute
...
#dataclasses.dataclass
class D:
app_events: typing.List[app_events.AppEvent] = \
dataclasses.field(default_factory=list)
def doo(self) -> app_events.AppEvent:
# same here: mypy considers app_events to be the class attribute
...
And the typecheck result:
PyCharm complains, for methods C.chew and D.doo: Unresolved attribute reference 'AppEvent' for class 'list'
mypy complains, still for methods C.chew and D.doo, that error: Name 'app_events.AppEvent' is not defined.
No issue for B.bar as written, though if app_events attribute is declared as a class attribute (instead of being defined in self.__init__, then mypy raise the same complaint.)
-> any idea how to understand/solve/circumvent this elegantly?
I'd really like not to rename my module and attributes, but if you have nice names in mind, please do not hesitate to propose :-)
I am trying to understand why pycharm warns me of wrong type when using an implementation of an abstract class with static method as parameter.
To demonstrate I will make a simple example. Let's say I have an abstract class with one method, a class that implements (inherits) this interface-like abstract class, and a method that gets the implementation it should use as parameter.
import abc
class GreetingMakerBase(abc.ABC):
#abc.abstractmethod
def make_greeting(self, name: str) -> str:
""" Makes greeting string with name of person """
class HelloGreetingMaker(GreetingMakerBase):
def make_greeting(self, name: str) -> str:
return "Hello {}!".format(name)
def print_greeting(maker: GreetingMakerBase, name):
print(maker.make_greeting(name))
hello_maker = HelloGreetingMaker()
print_greeting(hello_maker, "John")
Notice that in the type hinting of print_greeting I used GreetingMakerBase, and because isinstance(hello_maker, GreetingMakerBase) is True Pycharm is not complaining about it.
The problem is that I have many implementations of my class and dont want to make an instance of each, so I will make this make_greeting method static, like this:
class GreetingMakerBase(abc.ABC):
#staticmethod
#abc.abstractmethod
def make_greeting(name: str) -> str:
""" Makes greeting string with name of person """
class HelloGreetingMaker(GreetingMakerBase):
#staticmethod
def make_greeting(name: str) -> str:
return "Hello {}!".format(name)
def print_greeting(maker: GreetingMakerBase, name):
print(maker.make_greeting(name))
print_greeting(HelloGreetingMaker, "John")
This still works the same way, but apparently because the parameter in the function call is now the class name instead of an instance of it, Pycharm complains that:
Expected type 'GreetingMakerBase', got 'Type[HelloGreetingMaker]' instead.
Is there a way I can solve this warning without having to instantiate the HelloGreetingMaker class?
When you are doing print_greeting(HelloGreetingMaker, "John"), you are not trying to pass in an instance of HelloGreetingMaker. Rather, you're passing in the class itself.
The way we type this is by using Type[T], which specifies you want the type of T, rather then an instance of T. So for example:
from typing import Type
import abc
class GreetingMakerBase(abc.ABC):
#staticmethod
#abc.abstractmethod
def make_greeting(name: str) -> str:
""" Makes greeting string with name of person """
class HelloGreetingMaker(GreetingMakerBase):
#staticmethod
def make_greeting(name: str) -> str:
return "Hello {}!".format(name)
def print_greeting(maker: Type[GreetingMakerBase], name):
print(maker.make_greeting(name))
# Type checks!
print_greeting(HelloGreetingMaker, "John")
Note that Type[HelloGreetingMaker] is considered to be compatible with Type[GreetingMakerBase] -- Type[T] is covariant with respect to T.
The Python docs on the typing module and the mypy docs have more details and examples if you want to learn more.
You did not create an instance, and your type hints implies that the function only accepts instances (something of a type GreetingMakerBase, and not GreetingMakerBase itself or a subclass of it).
If you want to specify that only GreetingMakerBase itself is an acceptable argument, why have it as an argument at all? Just have the function call that class internally.
In any case, python 3.8 has some new typing improvements that can help you. You can specify a literal type hint:
from typing import Literal
def print_greeting(maker: Literal[GreetingMakerBase], name):
print(maker.make_greeting(name))
If you need to support this type-hint in other (earlier than 3.8) python versions, you will have to install the typing extensions:
pip install typing-extensions