I am following this tutorial on named tuple with specification of variable types. However, I modified the code (below), and even if I enter values of wrong types, there was no error message or programming break as a result. I understand you can write your own try/except to raise error exception, but is there a readily-available solution/syntax to enforce users entering the right type of variables.
from typing import NamedTuple
class Pet(NamedTuple):
pet_name: str
pet_type: str
def __repr__(self):
return f"{self.pet_name}, {self.pet_type}"
cleons_pet = Pet('Cotton', 'owl')
print('cleons_pet: ', cleons_pet)
cleons_pet_v2 = Pet(222, 1)
print('cleons_pet_v2: ', cleons_pet_v2)
# Output
cleons_pet: Cotton, owl
cleons_pet_v2: 222, 1
[Finished in 0.1s]
The type hints in python will not be evaluated by python itself! See PEP484
While these annotations are available at runtime through the usual annotations attribute, no type checking happens at runtime. Instead, the proposal assumes the existence of a separate off-line type checker which users can run over their source code voluntarily.
There are at least two projects which offer offline type checking (mypy and pyre). You should definitely use them if you are using type hints in your project.
If you want to validate the input while running the application, you have to either convince the offline type checkers by validating the data by yourself or use a third-party library. I know of attrs, where you can use validators or type annotations for online validation.
Related
Is there a neat solution to raise an error if a value is passed to the NamedTuple field that does not match the declared type?
In this example, I intentionally passed page_count str instead of int. And the script will work on passing the erroneous value forward.
(I understand that linter will draw your attention to the error, but I encountered this in a case where NamedTuple fields were filled in by a function getting values from config file).
I could check the type of each value with a condition, but it doesn't look really clean. Any ideas? Thanks.
from typing import NamedTuple
class ParserParams(NamedTuple):
api_url: str
page_count: int
timeout: float
parser_params = ParserParams(
api_url='some_url',
page_count='3',
timeout=10.0,
)
By design, Python is a dynamically typed language which means any value can be assigned to any variable. Typing is only supported as hints - the errors might be highlighted in your IDE, but they do not enforce anything.
This means that if you need type checking you have to implement it yourself. On the upside, this can probably be automated, i.e. implemented only once instead of separately for every field. However, NamedTuple does not provide such checking out of the box.
PyCharm Version: 2019.1.2
Python Version: 3.7
I am trying to use least code to reproduce the problem. And this is the snippet of my code:
def sql_reader():
def outer(func):
def wrapped_function(*args, **kwargs):
func(*args, **kwargs)
return [{"a": 1, "b": 2}]
return wrapped_function
return outer
#sql_reader()
def function_read():
return "1"
result = function_read()
for x in result:
print(x['a'])
print(result)
Basically, what I am doing is to "decorate" some function to output different types. For example, in this snippet, the function being decorated is returning 1 which is int. However, in decorator, I change the behavior and return list of dict.
Functionally speaking, it works fine. But it looks like my IDE always complains about it which is annoying as below:
Is there anyway I can get rid of this warning message?
With all due respect, you are using an over 3 year old version of PyCharm. I struggle to see a reason for this. The community edition is free and requires no root privileges to unpack and run on Linux systems. You should seriously consider upgrading to the latest version.
Same goes for Python by the way. You can install any version (including the latest) via Pyenv without root privileges. Although the Python version requirement may be subject to external restrictions for the code you are working on, so that is just a suggestion. But for the IDE I see no reason to use such an outdated version.
Since I am not using your PyCharm version, I can not reproduce your problem. Version 2022.2.3 has no such issues with your code. Be that as it may, there are a few things you can do to make life easier for static type checkers (and by extension for yourself).
The first thing I would always suggest is to use functools.wraps, when you are wrapping functions via a decorator. This preserves a lot of useful metadata about the wrapped function and even stores a reference to it in the wrapper's __wrapped__ attribute.
The second is proper type annotations. This should go for any code you write, unless it really is just a quick-and-dirty draft script that you will only use once and then throw away. The benefits of proper annotations especially in conjunction with modern IDEs are huge. There are many resources out there explaining them, so I won't go into details here.
In this concrete case, proper type hints will remove ambiguity about the return types of your functions and should work with any IDE (bugs non withstanding). In my version of PyCharm the return type of your wrapper function is inferred to be Any because no annotations are present, which prevents any warning like yours to pop up, but also doesn't allow any useful auto-suggestions to be provided.
Here is what I would do with your code: (should be compatible with Python 3.7)
from functools import wraps
from typing import Any, Callable, Dict, List
AnyFuncT = Callable[..., Any]
ResultsT = List[Dict[str, int]]
def sql_reader() -> Callable[[AnyFuncT], Callable[..., ResultsT]]:
def outer(func: AnyFuncT) -> Callable[..., ResultsT]:
#wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> ResultsT:
func(*args, **kwargs)
return [{"a": 1, "b": 2}]
return wrapper
return outer
#sql_reader()
def function_read() -> str:
return "1"
Adding reveal_type(function_read()) underneath and calling mypy on this file results in the following:
note: Revealed type is "builtins.list[builtins.dict[builtins.str, builtins.int]]"
Success: no issues found in 1 source file
As you can see, at least mypy now correctly infers the type returned by the wrapper function we put around function_read. Your IDE should also correctly infer the types involved, but as I said I cannot verify this with my version.
Moreover, now PyCharm will give you auto-suggestions for methods available on the types involved:
results = function_read()
first = results[0]
value = first["a"]
If I now start typing results., PyCharm will suggest things like append, extend etc. because it recognizes result as a list. If I type first., it will suggest keys, values etc. (inferring it as a dictionary) and if I type value. it will give options like imag, real and to_bytes, which are available for integers.
More information: typing module docs
I've been using Python's type annotations in an unusual way: I have some code that inspects the annotations of a function's arguments at run-time, searches for values that match the types of the arguments, and calls the function with values of appropriate types (if found).
This has been working wonderfully, and was remarkably easy to implement, but the source file has grown fairly large, so today I tried breaking it into multiple files. I found that get_type_hints() can't evaluate a type annotation if it's from a module other than the current one, at least not without being given a global namespace where all the needed types are defined.
I'm using from __future__ import annotations everywhere, so the type annotations in each function's .__annotations__ attribute are stored as strings in need of evaluation. To evaluate them, I need the globals from the module where the function was defined. How can I get that? Or will that even work? I'm using if TYPE_CHECKING: to avoid circular imports; consequently some annotations won't be available at run-time in each module where they're applied to a function.
Here's the code that extracts the types of the function arguments, if that helps:
def params_of(func: Callable) -> Iterable[Tuple[str, TypeAnnotation]]:
type_hints = get_type_hints(func)
for param_name in inspect.signature(func).parameters:
if param_name == 'return':
continue # disregard return type
yield (param_name, type_hints.get(param_name, Any))
(TypeAnnotation is only for readability; it's defined to Any.)
The following code works as expected, but the os.path.join produces a type error using pyright in VSCode, where shown.
# python 3.6.9
# pyright 1.1.25
# windows 10
# vscode 1.42.1
import os
import tempfile
with tempfile.TemporaryDirectory() as tmpfolder:
name = "hello.txt"
path = os.path.join(tmpfolder, name)
# No overloads for 'os.path.join(tmpfolder, name)' match parameters
# Argument types: (TypeVar['AnyStr', str, bytes], Literal['hello.txt'])
print(path)
I think I understand the immediate cause of the problem, but contend it should not be happening. Given that, I have some questions:
Is this the idiomatic way to write this code?
Is the problem in tempfile, os, pyright, or me?
If I cannot upgrade Python, what is the best (i.e. least clunky) way to suppress the error?
This seems like a limitation of pyright.
In short, the tempfile.TemporaryDirectory class is typed to be generic with respect to AnyStr. However, your example code omits specifying the generic type, leaving it up to the type checker to infer something appropriate.
In this case, I think there are several reasonable things for a type checker to do:
Pick some default generic type based on the typevar, such as 'str' or 'Union[str, bytes]'. For example, mypy ends up picking 'str' by default, giving 'tmpfolder' a type of 'str'.
Pick some placeholder type like either 'Any', the dynamic type, or NoReturn (aka 'bottom' aka 'nothing'). Both types are a valid subtype of every type, so are guaranteed to be valid placeholders and not cause downstream errors. This is what pyre and pytype does -- they deduce 'tmpfolder' has type 'Any' and 'nothing' respectively.
Attempt to infer the correct type based on context. Some type checkers may attempt to do this, but I don't know of any that handles this particular case perfectly.
Report an error and ask the user to specify the desired generic type.
What pyright seems to do instead is to just "leak" the generic variable. There is perhaps a principled reason why pyright is deciding to do this that I'm overlooking, but IMO this seems like a bug.
To answer your other questions, your example program is idiomatic Python, and type checkers should ideally support it without modification.
Adding a # type: ignore comment to the line with the error is the PEP 484 sanctioned way of suppressing error messages. I'm not familiar enough with pyright to know if it has a different preferred way of suppressing errors.
I ran into an issue when using mypy and have been able to find any help/reports regarding it. The following simplified code and error message should be self-explanatory:
#!/usr/bin/env python3
from typing import List, Union
class Corpus:
path: List[str]
def __init__(self, path:Union[str,List[str]]) -> None:
if type(path) == str:
self.path = [path]
else:
self.path = path
Mypy gives the following errors:
simplified.py:10: error: List item 0 has incompatible type "Union[str, List[str]]"; expected "str"
simplified.py:12: error: Incompatible types in assignment (expression has type "Union[str, List[str]]", variable has type "List[str]")
Even though the type of path variable is checked so that self.path should always result in string list, mypy complains about incompatible types.
Am I overlooking something or is this a bug in mypy?
(It it is a bug, should I go with #type: ignore annotation or is there a better work-around?)
(Some background: I decided to ease my life by writing a module which would take care of some repeating work. The argument in question should be a path to text data and I expect it to be only one string most of the time so I don't want to force putting it in a list. However, I wish to allow specifying more paths too. Internally, I store it as a list anyway as iterator over the class is always initialized with such list (and then possibly extends it further by "unpacking" directories)).
Try using isinstance(path, str) instead of type(path) == str. The former makes mypy typecheck your code without reporting an error.
Mypy really ought to support the latter form though -- there's an open feature request about it. The reason why this isn't implemented yet is almost certainly due to lack of time -- mypy's core team is pretty small, and there's an easy workaround in this case, so the feature was deprioritized.
(But hey, mypy is open source, so if you have some spare time...)