Can't use Pydantic model attributes on type hinting - python-3.x

Like I used to do with FastAPI routes, I want to make a function that is expecting a dict. I want to type hint like in FastAPI with a Pydantic model.
Note that I am just using FastAPI as a reference here and this app serves a total different purpose.
What I did:
models.py
from pydantic import BaseModel
class Mymodel(BaseModel):
name:str
age:int
main.py
def myfunc(m:Mymodel):
print(m)
print(m.name)
myfunc({"name":"abcd","age":3})
It prints m as a normal dict and not Mymodel and m.name just throws an AttributeError.
I don't understand why it is behaving like this because the same code would work in FastAPI. Am I missing something here? What should I do to make this work.
I am expecting a dict arg in the func, I want to type hint with a class inherited from pydantic BaseModel. Then I want to acccess the attributes of that class.
I don't want to do:
def myfunc(m):
m = Mymodel(**m)
Thank You.

from pydantic import BaseModel
from pydantic import validate_arguments
class Mymodel(BaseModel):
name:str
age:int
#validate_arguments
def myfunc(m:Mymodel):
print(m)
print(m.name)
myfunc({"name":"abcd","age":3})
This might be what you are looking for: https://pydantic-docs.helpmanual.io/usage/validation_decorator/

Since you pass a dict to your custom function, the attribute should be accessed in the following way:
print(m['name'])
# or
print(m.get('name'))
Otherwise, to use m.name instead, you need to parse the dict to the corresponding Pydantic model, before passing it to the function, as shwon below:
data = {"name":"abcd", "age":3}
myfunc(Mymodel(**data))
# or
myfunc(Mymodel.parse_obj(data))
The reason that passing {"name":"abcd", "age":3} in FastAPI and later accessing the attributes using the dot operator (e.g., m.name) works, is that FastAPI does the above parsing and validation internally, as soon as a request arrives. This is the reason that you can then convert it back to a dictionary in your endpoint, using m.dict(). Try, for example, passing an incorrect key, e.g., myfunc(Mymodel(**{"name":"abcd","MYage":3}))—you would get a field required (type=value_error.missing) error (as part of Pydantic's Error Handling), similar to what FastAPI would return (as shown below), if a similar request attempted to go through (you could also test that through Swagger UI autodocs at http://127.0.0.1:8000/docs). Otherwise, any dictionary passed by the user (in the way you show in the question) would go through without throwing an error, in case it didn't match the Pydantic model.
{
"detail": [
{
"loc": [
"body",
"age"
],
"msg": "field required",
"type": "value_error.missing"
}
]
}
You could alternatively use Pydantic's validation decorator (i.e., #validate_arguments) on your custom function. As per the documentation:
The validate_arguments decorator allows the arguments passed to a
function to be parsed and validated using the function's annotations
before the function is called. While under the hood this uses the same
approach of model creation and initialisation; it provides an
extremely easy way to apply validation to your code with minimal
boilerplate.
Example:
from pydantic import validate_arguments
from pydantic import BaseModel
class Model(BaseModel):
name: str
age: int
#validate_arguments
def myfunc(m: Model):
print(m)
print(m.name)
myfunc({"name":"abcd","age":3})

Related

What is the correct type hint to use when exporting a pydantic model as a dict?

I'm writing an abstraction module which validates an excel sheet against a pydantic schema and returns the row as a dict using dict(MyCustomModel(**sheet_row))
. I would like to use type hinting so any function that uses the abstraction methods gets a type hint for the returned dictionary with its keys instead of just getting an unhelpful dict. Basically I'd like to return the keys of the dict that compose the schema so I don't have to keep referring to the schema for its fields and to catch any errors early on.
My current workaround is having my abstraction library return the pydantic model directly and type hint using the Model itself. This means every field has to be accessed using a dot notation instead of accessing it like a regular dictionary. I cannot annotate the dict has being the model itself as its a dict, not the actual pydantic model which has some extra attributes as well.
I tried type hinting with the type MyCustomModel.__dict__(). That resulted in the error TypeError: Parameters to generic types must be types. Got mappingproxy({'__config__': <class 'foo.bar.Config'>, '__fields__': {'lab.. Is there a way to send a type hint about the fields in the schema, but as a dictionary? I don't omit any keys during the dict export. All the fields in the model is present in the final dict being returned
I am going to try and abstract that question and create minimal reproducible example for you.
Question
Consider this working example:
from typing import Any
from pydantic import BaseModel
class Foo(BaseModel):
x: str
y: int
def validate(data: dict[str, Any], model: type[BaseModel]) -> dict[str, Any]:
return dict(model.parse_obj(data))
def test() -> None:
data = {"x": "spam", "y": "123"}
validated = validate(data, Foo)
print(validated)
# reveal_type(validated["x"])
# reveal_type(validated["y"])
if __name__ == "__main__":
test()
The code works fine and outputs {'x': 'spam', 'y': 123} as expected. But if you uncomment the reveal_type lines and run mypy over it, obviously the type it sees is just Any for both.
Is there a way to annotate validate, so that a type checker knows, which keys will be present in the returned dictionary, based on the model provided to it?
Answer
Python dictionaries have no mechanism built into them for distinguishing their type via specific keys. The generic dict type is parameterized by exactly two type parameters, namely the key type and the value type.
You can utilize the typing.TypedDict class to define a type based on the specific keys of a dictionary. However (as pointed out by #hernán-alarcón in the comments) the __dict__ method still returns just a dict[str, Any]. You can always cast the output of course and for this particular Foo model this would work:
from typing import Any, TypedDict, cast
from pydantic import BaseModel
class Foo(BaseModel):
x: str
y: int
class FooDict(TypedDict):
x: str
y: int
def validate(data: dict[str, Any], model: type[BaseModel]) -> FooDict:
return cast(FooDict, dict(model.parse_obj(data)))
def test() -> None:
data = {"x": "spam", "y": "123"}
validated = validate(data, Foo)
print(validated)
reveal_type(validated["x"]) # "builtins.str"
reveal_type(validated["y"]) # "builtins.int"
if __name__ == "__main__":
test()
But it is not very helpful, if validate should be able to deal with any model, not just Foo.
The easiest way to generalize this that I can think of is to make your own base model class that is generic in terms of the corresponding TypedDict. Binding the type argument in a dedicated private attribute should be enough. You won't actually have to set it or interact with it at any point. It is enough to specify it, when you subclass your base class. Here is a working example:
from typing import Any, Generic, TypeVar, TypedDict, cast
from pydantic import BaseModel as PydanticBaseModel, PrivateAttr
T = TypeVar("T")
class BaseModel(PydanticBaseModel, Generic[T]):
__typed_dict__: type[T] = PrivateAttr(...)
class FooDict(TypedDict):
x: str
y: int
class Foo(BaseModel[FooDict]):
x: str
y: int
def validate(data: dict[str, Any], model: type[BaseModel[T]]) -> T:
return cast(T, model.parse_obj(data).dict())
def test() -> None:
data = {"x": "spam", "y": "123"}
validated = validate(data, Foo)
print(validated)
reveal_type(validated["x"]) # "builtins.str"
reveal_type(validated["y"]) # "builtins.int"
reveal_type(validated) # "TypedDict('FooDict', {'x': builtins.str, 'y': builtins.int})"
if __name__ == "__main__":
test()
This works well enough to convey the dictionary keys and corresponding types.
If you are wondering, whether there is a way to just dynamically infer the TypedDict rather than just duplicating the model fields manually, the answer is no.
Static type checkers do not execute your code, they just read it.
This brings me to the final consideration. I don't know, why you would even want to use a dictionary over a model instance in the first place. It seems that for the purposes of dealing with structured data, the model is superior in every aspect, if you already are using Pydantic anyway.
The fact that you access the fields as attributes (via dot-notation) is a feature IMHO and not a drawback of this approach. If you for some reason do need to have dynamic attribute access via field names as strings, you can always just use getattr on the model instance.

How do I use FastAPI's "response_model_exclude_none" as a variable option

In FastApi you can set response_model_exclude_none=True to exclude fields with value None. As an example, I could set up an endpoint like so:
from fastapi import APIRouter
from pydantic import BaseModel
class TestModel(BaseModel):
model_option: str
router = APIRouter(prefix="/hello", tags=["test"])
#router.get(
"/world",
response_model=TestModel,
response_model_exclude_none=True,
)
def return_model(model_option: str):
return TestModel(model_option=model_option)
As a user, the arguments I can pass are exclusively in the return_model function. Is there a way to allow the user to also pass the response_model_exclude_none argument in the router.get() decorator so that they can decide whether or not they want to exclude None values or not?

How would you use asyncpg with FastAPI to map returned values from a select query to pydantic models for output and validation?

I want to use FastAPI without an ORM (using asyncpg) and map the returned values from a select query to a pydantic model. This way the returned values are validated with pydantic and the response that is returned is structured like the pydantic model/schema.
I’ve tried looking for documentation on this but it’s pretty hard to find/not clear. I’d appreciate any help!
Every pydantic model inherits a couple of utility helpers to create objects. One is parse_obj which takes a dict and creates the model object from that.
parse_obj: this is very similar to the __init__ method of the model, except it takes a dict rather than keyword arguments. If the object passed is not a dict a ValidationError will be raised.
From the example on the linked section above:
class User(BaseModel):
id: int
name = 'John Doe'
signup_ts: datetime = None
m = User.parse_obj({'id': 123, 'name': 'James'})
print(m)
#> id=123 signup_ts=None name='James'
You might be able to give parse_obj a Record directly since it implements dict-like accessors, so just try it and see if it works. If not you can use dict(<row record from asyncpg>) to convert the record to an actual dict.

How to specify the type of a unittest.mock.sentinel?

I am using unittest.mock.sentinel to provide dumb values to my test functions and then assert calls.
I'd like to be able to specify the type of the sentinel so that it passes type checking in the methods.
MWE:
import collections
from unittest.mock import sentinel
def fun(x):
if not isinstance(x, collections.Iterable):
raise TypeError('x should be iterable')
pass
def test_fun_pass_if_x_is_instance_iterable():
# this does not work and raise because sentinel is not iterable
assert fun(sentinel.x) is None
EDIT
I have tried to do sentinel.x = collections.Iterable() but got the error:
TypeError: Can't instantiate abstract class Iterable with abstract methods __iter__
So far I can do sentinel.x = tuple() or sentinel.x = list() for instance, but these are special case of an iterable
I think the problem here is that collections.Iterable is an abstract base class (ABC) and cannot be instantiated directly. That's what the error message says, the method __iter__ is abstract, without body. You have to use a derived class or write one on your own.

Can't pickle <class 'a class'>: attribute lookup inner class on a class failed

I was using PySpark to process some calls data. As you see, I added some inner classes to class GetInfoFromCalls dynamically by using metaclass.
code below located in package for_test that existed in all nodes:
class StatusField(object):
"""
some alias.
"""
failed = "failed"
succeed = "succeed"
status = "status"
getNothingDefaultValue = "-999999"
class Result(object):
"""
Result that store result and some info about it.
"""
def __init__(self, result, status, message=None):
self.result = result
self.status = status
self.message = message
structureList = [
("user_mobile", str, None),
("real_name", str, None),
("channel_attr", str, None),
("channel_src", str, None),
("task_data", dict, None),
("bill_info", list, "task_data"),
("account_info", list, "task_data"),
("payment_info", list, "task_data"),
("call_info", list, "task_data")
]
def inner_get(self, defaultValue=StatusField.getNothingDefaultValue):
try:
return self.holder.get(self)
except Exception as e:
return Result(defaultValue, StatusField.failed)
print(e)
class call_meta(type):
def __init__(cls, name, bases, attrs):
for name_str, type_class, pLevel_str in structureList:
setattr(cls, name_str, type(
name_str,
(object,),
{})
)
class GetInfoFromCalls(object, metaclass = call_meta):
def __init__(self, call_deatails):
for name_str, type_class, pLevel_str in structureList:
inn = getattr(self.__class__, name_str)()
object_dict = {
"name": name_str,
"type": type_class,
"pLevel": None if pLevel_str is None else getattr(self, pLevel_str),
"context": None,
"get": inner_get,
"holder": self,
}
for attr_str, real_attr in object_dict.items():
setattr(inn, attr_str, real_attr)
setattr(self, name_str, inn)
self.call_details = call_deatails
when I ran
import pickle
pickle.dumps(GetInfoFromCalls("foo"))
it raised error like this:
Traceback (most recent call last):
File "<ipython-input-11-b2d409e35eb4>", line 1, in <module>
pickle.dumps(GetInfoFromCalls("foo"))
PicklingError: Can't pickle <class '__main__.user_mobile'>: attribute lookup user_mobile on __main__ failed
It seemed that I can't pickle inner classes because them were added dynamically by code. When classes were pickled, inner classes were not existed, is it right?
Really I don't want to write these classes that were nearly same to each other. Does someone has good way to avoid this problem?
Python's pickle actually does not serializes classes: it does serialize instances, and put in the serialization a reference to each instance's class - and that reference is based on the class being bound to a name in a well defined module. So, instances of classes that don't have a module name, but rather live as attribute in other classes, or data inside lists and dictionaries, typically will not work.
One straight forward thing one can try to do is try to use dill instead of pickle. It is a third party package that works like "pickle" but has extensions to actually serialize arbitrary dynamic classes.
While using dill may help other people reaching here, it is not your case, because in order to use dill, you'd have to monkey patch the underlying RPC mechanism PySpark is using to make use of dill instead of pickle, and that might not be trivial nor consistent enough for production use.
If the problem is really about dynamically created classes being unpickable, what you can do is to create extra meta-classes, for the dynamic classes themselves, instead of using "type", and on these metaclasses, create proper __getstate__ and __setstate__ (or other helper methods as it is on pickle documentation) - that might enable these classes to be pickled by ordinary Pickle. That is, a separate metaclass with Pickler helper methods to be used instead of type(..., (object, ), ...) in your code.
However, "unpickable object" is not the error you are getting - it is an attribute lookup error, which suggests the structure you are building is not good enough for Pickle to introspect into it and get all the members from one of your instances - it is not related (yet) to the unpickleability of the class object. Since your dynamic classes live as attributes on the class (which is not itself pickled) and not of the instance, it is very well possible that pickle does not care about it. Check the docs on pickle above, and maybe all you need there is proper helper-method to pickle on you class, nothing different on the the metaclass for all that you have there to work properly.

Resources