I'm using pint to use and convert units. I wanted to create classes which restricts the quantities only to "[time]" or "[length]" dimensions, so as a first approach I did the following:
from pint import Quantity, DimensionalityError
class Time(Quantity):
def __new__(cls, v: str | Quantity) -> Quantity:
obj = Quantity(v)
if not obj.check("[time]"):
raise DimensionalityError(v, "[time]")
return obj
class Length(Quantity):
def __new__(cls, v: str | Quantity) -> Quantity:
obj = Quantity(v)
if not obj.check("[length]"):
raise DimensionalityError(v, "[length]")
return obj
At runtime it works as expected, i.e: I can do the following:
1hour = Time("1h") # Works ok, variable 1hour contains `<Quantity(1, 'hour')>`
bad = Time("1meter") # As expected, raises pint.errors.DimensionalityError: Cannot convert from '1meter' to '[time]'
1meter = Length("1meter") # Ok
bad_again = Length("1h") # Ok, raises DimensionalityError
However, from a typing perspective, something is wrong:
def myfunc(t: Time) -> str:
return f"The duration is {t}"
print(myfunc(Time("1h"))) # Ok
print(myfunc(Length("1m"))) # Type error?
The second call to myfunc() is a type error, since I'm passing a Length instead of a Time. However mypy is happy with the code. So I have some questions:
Why doesn't mypy catches the error?
How to do it properly?
For 1. I guess that something fishy is happening in pint's Quantity implementation. I tried:
foo = Quantity("3 pounds")
reveal_type(foo)
and the revealed type is Any instead of Quantity which is very suspicious.
So I tried removing the base class Quantity from my Time and Length classes (i.e: they derive now from object instead of Quantity), and in this case, mypy correctly manages the typing errors.
But it fails again as soon as I try something like Length("60km")/Time("1h"). mypy complains that Length object does not implement the required method for performing that division (although the code works ok at runtime because, after all, Length and Time __new__() method is returning a Quantity object which does implement the arithmetical operations).
So, again, is there any workaround for making the idea work both at run-time and for mypy?
Related
I am relatively new to Python. Using Python 3.7 in this example below. The Linter is not catching any of the coding errors nor, it throws any exception when wrong return types are returned. What is the best and formal way of handing such issues?
from typing import Tuple
from abc import ABC, abstractmethod
class MyAbc(ABC):
#abstractmethod
def get_hello(self) -> Tuple[bool, str, str]:
# Need the return to be a Tuple of 3 values: bool, str, str
pass
class ImplementedClass(MyAbc):
def get_hello(self):
return True, "Hello"
# But in the implementation I am returning only 2 values: bool, str
# This coding error is not caught here
ic: MyAbc = ImplementedClass()
print(ic.get_hello()) # Error escaped
resp1, resp2, resp3 = ic.get_hello()
# The issue is caught only here
# Pylint: Possible unbalanced tuple unpacking with sequence defined at line 15: left side has 3 label(s), right side has 2 value(s)
print(resp1, resp2, resp3)
print(ImplementedClass().get_hello())
def three_returns() -> Tuple[str, str, str]:
return "one", "two"
print(three_returns()) # Error escaped
def something(data: str) -> str:
print(type(data), data)
return 1 # Supposed to return str, returning int, not caught
value: str = something(2) # Expected str but int returned
print(value.upper()) # Pylint: Instance of 'int' has no 'upper' member
As mentioned in the code block, when incorrect object is returned, pylint or python will never throw any error. It is only when it is explicitly mapped or any class methods are invoked like str.upper() that's when the error is thrown. This would lead to testing all the paths thoroughly else, it can be sure that code block would work.
Is it how it is and we have live with it or there any better ways to handle it like what we get compile time errors in C++, Java?
What you seem to be searching for is a type checker like mypy, pyright or pyre.
Recently I had a problem reading information from a .ini file to a int type variable:
var:int = config['example']['var']
The .ini file is something like this:
Using configparser to read the value pass it as '3' instead of 3.
I must add that I know I can convert it to int by using int(var) (that's how I fixed it by the way), but that is not the point of this question.
The variable type is a int and the configparser read the value from the file as a string and successfully changes the variable type to a string, the code worked for multiple days until I had a bug, the interpreter indicates I was trying to compare a string type with an int type, by this point I was so far into the code I wasted 10 minutes pinpointing the source of the bug.
Here's a example code:
class testclass:
def __init__(self, var:int): #Specifying that var should be a int
self._storage:int = var #specifiying that _storage should be a int
#property
def get(self) -> int: #specifiying that get method should return a int
return self._storage
#Expected behaviour
correct = 1
test1 = testclass(correct)
#Passing a string when it should be a integer
wrong = '1'
test2 = testclass(wrong)
print('test1 type = ' + str(type(test1.get)))
print('test2 type = ' + str(type(test2.get)))
And it's output:
test1 type = <class 'int'>
test2 type = <class 'str'>
As you can see I made sure to specify the type of every single method and variable so I couldn't unknowingly pass the wrong type by mistake (which is exactly what I did), but it did not make any difference, whats is the point of :int and -> int and what should I do the lock arguments/variables/methods etc. to a variable type so I can't shoot myself on the foot in the future?
Python doesn't force types. You can pass other types even if you specified a type. However, you can use a Python Linter such as mypy and it will warn you when you pass the wrong type.
Note: The program will run, even if you use mypy, but you will see a warning while writing the code.
I have a function that is returning either an class instance or None depending
on some logic.
In some places of the code I know this function is for sure not returning None,
but mypy complains.
I made a minimal example that reproduces the situation described above.
I would like to avoid marking a_string as a_string: Optional[str] = "", I
know I can also overcome the problem using cast or type ignore, but somehow I
feel there might be a better way.
Any recommendations how to handle this situation?
For this example I am using mypy 0.641 and python 3.7
"""
Function returns either an object or none
"""
from typing import Optional, cast
RET_NONE = False
def minimal_example() -> Optional[str]:
if RET_NONE:
return None
else:
return "my string"
a_string = ""
maybe_string = minimal_example()
a_string = maybe_string
# mypy doesn't complain if I do the following
a_string = cast(str, maybe_string)
a_string = maybe_string # type: ignore
Mypy complains as follows:
❯❯❯ mypy mypy_none_or_object.py (chatsalot) ✘ 1
mypy_none_or_object.py:19: error: Incompatible types in assignment (expression has type "Optional[str]", variable has type "str")
Mypy is designed to treat function signatures as the "source of truth". If you indicate that some function returns an Optional[str], then mypy will assume that will always be the case. It won't attempt to see how any global variables may or may not alter that function signature.
The easiest possible way of working around this is to add an assert or isinstance check:
maybe_string = minimal_example()
reveal_type(maybe_string) # Revealed type is Optional[str]
assert maybe_string is not None # Or use 'if isinstance(maybe_string, str)
reveal_type(maybe_string) # Revealed type is str
(If you're not aware, mypy will special-case the reveal_type(...) function: whenever mypy encounters it, mypy prints out the type of whatever expression you provide. This is useful for debugging, but you should remember to delete the pseudo-function after you're done since it doesn't exist at runtime.)
Alternatively, you could redesign your code so that your function's return value is more normalized -- it always returns a string instead of sometimes returning one.
If RET_NONE is meant to be a more-or-less immutable global (e.g. something like "enable debug mode" or "assume we're running on Windows"), you could use is to take advantage of mypy's --always-true and --always-false flags and provide two different definitions of minimal_example. For example:
RET_NONE = False
if RET_NONE:
def minimal_example() -> None:
return None
else:
def minimal_example() -> str:
return str
You then invoke mypy using mypy --always-true RET_NONE or mypy --always-false RET_NONE to match how your variable is defined. You can find more info about these types here and maybe here.
A fourth alternative you could explore is using function overloads: https://mypy.readthedocs.io/en/latest/more_types.html#function-overloading
However, idk if that really works in your case: you can't define overloads where only the return type differs: the argument arity or types of each overload need to be distinguishable from each other in some way.
Both solutions: cast() and # type: ignore are effectively turning off mypy checks for the variable. This can shadow bugs and should be avoided when possible.
In your case mypy cannot know the value of RET_NONE, since it can be changed in runtime from False to anything else, thus the error.
I suggest adding an assertion:
a_string = ""
maybe_string = minimal_example()
assert maybe_string is not None # <- here
a_string = maybe_string
Now mypy is sure that on the next line maybe_string definitely won't be None. I covered this in the Constraining types section of my blog post about typing.
I am trying to extend pythons’ list with some custom methods, for that I
am creating an class inheriting from UserList.
I am not sure what is the right way and I would like to get mypy play
nicely with UserList.
I consulted cpython UserList docs and searched inside mypy for
UserList but couldn’t find anything.
Using:
mypy 0.641
python 3.7.0
This is a minimal example of what I am trying to achive
from collections import UserList
from typing import List, Optional, Union
class A:
...
class B:
...
class C:
...
TMessage = Union[A, B, C]
class MyList(UserList):
"""Minimal example"""
def __init__(self, data: Optional[List[TMessage]] = None) -> None:
self.data: List[TMessage] = []
if data:
self.data = data[:]
def get_last(self) -> TMessage:
return self.data[-1]
# other methods to be added ...
some_data = [A(), B(), C(), C(), B(), A()]
my_list_a = MyList(some_data)
my_list_b = MyList(some_data)
my_list_b = my_list_a[3:]
Mypy complains as follows
~/tmp ❯❯❯ mypy mypy_userlist.py
mypy_userlist.py:34: error: Argument 1 to "MyList" has incompatible type "List[object]"; expected "Optional[List[Union[A, B, C]]]"
mypy_userlist.py:35: error: Argument 1 to "MyList" has incompatible type "List[object]"; expected "Optional[List[Union[A, B, C]]]"
mypy_userlist.py:37: error: Incompatible types in assignment (expression has type "MutableSequence[Any]", variable has type "MyList")
I could add # type: ignore to the conflicting lines but I would like
to avoid that.
What is the right way to extend the python’s list with custom methods
and get mypy happy?
I'm fairly new at MyPy but I think you have two issues. The first is that lists are mutable, so although your list object some_data satisfies the required structure in your code, there's no reason that an object, not of type A, B or C couldn't be added to it later, meaning at compile tile, Mypy can't ensure that
my_list_a = MyList(some_data)
is a valid assignment. (have a look at the common issues section of the Mypy docs here for more discussion)
You can fix this by explicitly annotating some_data:
some_data : List[TMessage] = [A(), B(), C(), C(), B(), A()]
The second problem will pop up when you fix this, when you try and assign your two lists using slicing. MyPy won't know what your slice function will return and will complain about incompatible types.
To fix this, you can explicitly implement the slice functionality into your class.
def __getitem__(self, slice_indices) -> 'MyList':
return self.data[slice_indices]
I just started working with PyCharm Community Edition 2016.3.2 today. Every time I assign a value from my function at_square, it warns me that 'Function at_square doesn't return anything,' but it definitely does in every instance unless an error is raised during execution, and every use of the function is behaving as expected. I want to know why PyCharm thinks it doesn't and if there's anything I can do to correct it. (I know there is an option to suppress the warning for that particular function, but it does so by inserting a commented line in my code above the function, and I find it just as annoying to have to remember to take that out at the end of the project.)
This is the function in question:
def at_square(self, square):
""" Return the value at the given square """
if type(square) == str:
file, rank = Board.tup_from_an(square)
elif type(square) == tuple:
file, rank = square
else:
raise ValueError("Expected tuple or AN str, got " + str(type(square)))
if not 0 <= file <= 7:
raise ValueError("File out of range: " + str(file))
if not 0 <= rank <= 7:
raise ValueError("Rank out of range: " + str(rank))
return self.board[file][rank]
If it matters, this is more precisely a method of an object. I stuck with the term 'function' because that is the language PyCharm is using.
My only thought is that my use of error raising might be confusing PyCharm, but that seems too simple. (Please feel free to critique my error raising, as I'm not sure this is the idiomatic way to do it.)
Update: Humorously, if I remove the return line altogether, the warning goes away and returns immediately when I put it back. It also goes away if I replace self.board[file][rank] with a constant value like 8. Changing file or rank to constant values does not remove the warning, so I gather that PyCharm is somehow confused about the nature of self.board, which is a list of 8 other lists.
Update: Per the suggestion of #StephenRauch, I created a minimal example that reflects everything relevant to data assignment done by at_square:
class Obj:
def __init__(self):
self.nested_list = [[0],[1]]
#staticmethod
def tup_method(data):
return tuple(data)
def method(self,data):
x,y = Obj.tup_method(data)
return self.nested_list[x][y]
def other_method(self,data):
value = self.method(data)
print(value)
x = Obj()
x.other_method([1,2])
PyCharm doesn't give any warnings for this. In at_square, I've tried commenting out every single line down to the two following:
def at_square(self, square):
file, rank = Board.tup_from_an(square)
return self.board[file][rank]
PyCharm gives the same warning. If I leave only the return line, then and only then does the warning disappear. PyCharm appears to be confused by the simultaneous assignment of file and rank via tup_from_an. Here is the code for that method:
#staticmethod
def tup_from_an(an):
""" Convert a square in algebraic notation into a coordinate tuple """
if an[0] in Board.a_file_dict:
file = Board.a_file_dict[an[0]]
else:
raise ValueError("Invalid an syntax (file out of range a-h): " + str(an))
if not an[1].isnumeric():
raise ValueError("Invalid an syntax (rank out of range 1-8): " + str(an))
elif int(an[1]) - 1 in Board.n_file_dict:
rank = int(an[1]) - 1
else:
raise ValueError("Invalid an syntax (rank out of range 1-8): " + str(an))
return file, rank
Update: In its constructor, the class Board (which is the parent class for all these methods) saves a reference to the instance in a static variable instance. self.at_square(square) gives the warning, while Board.instance.at_square(square) does not. I'm still going to use the former where appropriate, but that could shed some light on what PyCharm is thinking.
PyCharm assumes a missing return value if the return value statically evaluates to None. This can happen if initialising values using None, and changing their type later on.
class Foo:
def __init__(self):
self.qux = [None] # infers type for Foo().qux as List[None]
def bar(self):
return self.qux[0] # infers return type as None
At this point, Foo.bar is statically inferred as (self: Foo) -> None. Dynamically changing the type of qux via side-effects does not update this:
foo = Foo()
foo.qux = [2] # *dynamic* type of foo.bar() is now ``(self: Foo) -> int``
foo_bar = foo.bar() # Function 'bar' still has same *static* type
The problem is that you are overwriting a statically inferred class attribute by means of a dynamically assigned instance attribute. That is simply not feasible for static analysis to catch in general.
You can fix this with an explicit type hint.
import typing
class Foo:
def __init__(self):
self.qux = [None] # type: typing.List[int]
def bar(self):
return self.qux[0] # infers return type as int
Since Python 3.5, you can also use inline type hints. These are especially useful for return types.
import typing
class Foo:
def __init__(self):
# initial type hint to enable inference
self.qux: typing.List[int] = [None]
# explicit return type hint to override inference
def bar(self) -> int:
return self.qux[0] # infers return type as int
Note that it is still a good idea to rely on inference where it works! Annotating only self.qux makes it easier to change the type later on. Annotating bar is mostly useful for documentation and to override incorrect inference.
If you need to support pre-3.5, you can also use stub files. Say your class is in foomodule.py, create a file called foomodule.pyi. Inside, just add the annotated fields and function signatures; you can (and should) leave out the bodies.
import typing
class Foo:
# type hint for fields
qux: typing.List[int]
# explicit return type hint to override inference
def bar(self) -> int:
...
Type hinting as of Python 3.6
The style in the example below is now recommended:
from typing import typing
class Board:
def __init__(self):
self.board: List[List[int]] = []
Quick Documentation
Pycharm's 'Quick Documentation' show if you got the typing right. Place the cursor in the middle of the object of interest and hit Ctrl+Q. I suspect the types from tup_from_an(an) is not going to be as desired. You could, try and type hint all the args and internal objects, but it may be better value to type hint just the function return types. Type hinting means I don't need to trawl external documentation, so I focus effort on objects that'll be used by external users and try not to do too much internal stuff. Here's both arg and return type hinting:
#staticmethod
def tup_from_an(an: List[int]) -> (int, int):
...
Clear Cache
Pycharm can lock onto out dated definitions. Doesn't hurt to go help>find action...>clear caches
No bodies perfect
Python is constantly improving (Type hinting was updated in 3.7) Pycharm is also constantly improving. The price for the fast pace of development on these relatively immature advanced features means checking or submitting to their issue tracker may be the next call.