I'm trying to figure out how to serialize an object with Pickle to a save file. My example is an object called World and this object has a list (named objects) of potentially hundreds of instantiated objects of different class types.
The problem is that Pickle won't let me serialize the items within the World.objects list because they aren't instantiated as attributes of World.
When I attempt to serialize with:
with open('gsave.pkl', 'wb') as output:
pickle.dump(world.objects, output, pickle.DEFAULT_PROTOCOL)
I get the following error:
_pickle.PicklingError: Can't pickle <class 'world.LargeHealthPotion'>:
attribute lookup LargeHealthPotion on world failed
So, my question is: what is an alternative way of storing the world.objects list items so that they are attributes of world rather than list items that don't get saved?
UPDATE
I think my issue isn't where the objects are stored; but rather that the class LargeHealthPotion (and many others) are dynamically created within the World class by operations such as this:
def __constructor__(self, n, cl, d, c, h, l):
# initialize super of class type
super(self.__class__, self).__init__(name=n, classtype=cl, description=d, cost=c,
hp=h, level=l)
# create the object class dynamically, utilizing __constructor__ for __init__ method
item = type(item_name,
(eval("{}.{}".format(name,row[1].value)),),
{'__init__':__constructor__})
# add new object to the global _objects object to be used throughout the world
self._objects[item_name] = item(obj_name, obj_classtype, obj_description, obj_cost,
obj_hp, obj_level)
When this finishes, I will have a new object like <world.LargeHealthPotion object at 0x103690ac8>. I do this dynamically because I don't want to explicitly have to create hundreds of different types of classes for each different type of object in my world. Instead, I create the class dynamically while iterating over the item name (with it's stats) that I want to create.
This introduces a problem though, because when pickling, it can't find the static reference to the class in order to deconstruct, or reconstruct the object...so it fails.
What else can I do? (Besides creating literal class references for each, and every, type of object I'm going to instantiate into my world.)
Pickle does not pickle classes, it instead relies on references to classes which doesn't work if the class was dynamically generated. (this answer has appropriate exert and bolding from documentation)
So pickle assumes that if your object is from the class called world.LargeHealthPotion then it check that that name actually resolves to the class that it will be able to use when unpickling, if it doesn't then you won't be able to reinitialize the object since it doesn't know how to reference the class. There are a few ways of getting around this:
Define __reduce__ to reconstruct object
I'm not sure how to demo this method to you, I'd need much more information about your setup to suggest how to implement this but I can describe it:
First you'd make a function or classmethod that could recreate one object based on the arguments (probably take class name, instance variables etc.) Then define __reduce__ on the object base class that would return that function along with the arguments needed to pass to it when unpickling.
Put the dynamic classes in the global scope
This is the quick and dirty solution. Assuming the class names do not conflict with other things defined in the world module you could theoretically insert the classes into the global scope by doing globals()[item_name] = item_type, but I do not recommend this as long term solution since it is very bad practice.
Don't use dynamic classes
This is definitely the way to go in my opinion, instead of using the type constructor, just define your own class named something like ObjectType that:
Is not a subclass of type so the instances would be pickle-able.
When an instance is it called constructs a new game-object that has a reference to the object type.
So assuming you have a class called GameObject that takes cls=<ObjectType object> you could setup the ObjectType class something like this:
class ObjectType:
def __init__(self, name, description):
self.item_name = name
self.base_item_description = description
#other qualities common to all objects of this type
def __call__(self, cost, level, hp):
#other qualities that are specific to each item
return GameObject(cls=self, cost=cost, level=level, hp=hp)
Here I am using the __call__ magic method so it uses the same notation as classes cls(params) to create instances, the cls=self would indicate to the (abstracted) GameObject constructor that the class (type) of GameObject is based on the ObjectType instance self. It doesn't have to be a keyword argument, but I'm not sure how else to make a coherent example code without knowing more about your program.
Related
I'm writing a library which permits expansion by registering custom render elements.
I had the idea of using a metaclass on AbstractElement to automatically register all non-abstract classes subsequently derived from it. This is what I came up with:
import abc
import inspect
class AbstractElementMeta(type):
def __new__(mcs, name, basses, attrs):
cls = super(M, mcs).__new__(mcs, name, basses, attrs)
if not inspect.isabstract(cls):
register_element(cls) # Defined elsewhere in the module
class AbstractElement(abc.ABC, metaclass=AbstractElementMeta):
pass
But this results in the following error:
TypeError: metaclass conflict: the metaclass of a derived class must be a
(non-strict) subclass of the metaclasses of all its bases
Still sort of wrapping my head around the internal class structure and mechanisms of python. Is what I want to do possible, and if so, how can I go about it? I want to avoid making my users use the metaclass themselves (I'd just make a decorator if I had to).
My attempt was to create the default instance from inside of a metaclass, but to no avail. At least the reported class is the singleton in the example bellow.
EDIT: Clarifying requirements here: a singleton comparable by using the is keyword, without having to instantiate/call it. Unfortunately, this well known question-answer here doesn't seem to address that.
class MyNormalClass:
def __init__(self, values):
self.values = values
class MySingleton(MyNormalClass, type):
def __new__(mcs, *args, **kwargs):
return MyNormalClass(["default"])
print(MySingleton)
# <class '__main__.MySingleton'>
print(MySingleton.values)
# AttributeError: type object 'MySingleton' has no attribute 'values'
Metaclasses for singletons are overkill. (Search my answers for that, and there should be about 10 occurrences of this phrase).
In your example code in the question, the code inside the class and the metaclass methods is not even being run, not once. There is no black-magic in Python - the program just runs, and there are a few special methods marked with __xx__ that will be called intrinsically by the language runtime. In this case, the metaclass __new__ will be called whenever you create a new class using it as the metaclass, which you code does not show. And also, it would create a new instance of your "singleton" class each time it were used.
In this case, if all you need is a single instance, just create that instance, and let that one be public, instead of its class. You can even us the instance itself to shadow the class from the module namespace, so no one can instantiate it again by accident. If you want values to be immutable, well, you can't
ensure that with pure Python code in any way, but you can make changing values do not work casually with = so that people will know they should not be changing it:
class MySingleton:
__slots__ = ("values",)
def __init__(self, values):
self.values = values
def lock(self, name, value):
raise TypeError("Singleton can't change value")
self.__class__.__setitem__ = lock
MySingleton = MySingleton(["values"])
Given the following example:
class Container:
def __init__(self, var):
self.var = var
class Test:
def __init__(self):
self.var = Container("123")
Is it possible to overload the type() method such as type(Test().var) would yield string rather than Container ?
EDIT : I am using the Container class in order to place restrictions on Test.var.
The idea is that Test is a class that contains many variables, some of witch have similar names. The Container class is there to ensure that the right types are used ( __eq__(), __str__(), __add__(), ... are overloaded in order to make the Container class as discreet as possible ) so that issues are diagnosed as fast as possible ( the code will by used by people with a very wide variety of expertise in python )
The other solution would have been to use the #property but as there are many variables, the code ends up being way bigger than it would otherwise and not as simple to maintain ( there is close to a hundred classes witch will have to implement the properties )
I would like to overload type(Test().var) in order to return type(Test().var.var) so that it would be as easy to use as possible
The short answer is "no."
From this official Python doc, it states:
Every object has an identity, a type and a value... The type() function returns an object’s type (which is an object itself). Like its identity, an object’s type is also unchangeable.
I am currently developing a piece of software where the I have class instamces that are generated from dictionaries. The way these dictionariea file are structured is as follows:
layer_dict = {
"layer_type": "Conv2D",
"name": "conv1",
"kernel_size": 3,
...
}
Then, the following code is ran
def create_layer(layer_dict):
LayerType = getattr(layers, layer_dict['layer_type']
del layer_dict['layer_type']
return LayerType(**layer_dict)
Now, I want to support the creation of new layer types (by subclassing the BaseLayer class). I've thought of a few ways to do this and thought I'd ask which way is best and why as I don't have much experience developing software (finishing an MSc in comp bio).
Method 1: Metaclasses
The first method I thought of was to have a metaclass that registers every subclass of BaseLayer in a dict and do a simple lookup of this dict instead of using getattr.
class MetaLayer(type)
layers = {}
def __init__(cls, name, bases, dct):
if name in MetaLayer.layers:
raise ValueError('Cannot have more than one layer with the same name')
MetaLayer.layers[name] = cls
Benefit: The metaclass can make sure that no two classes have the same name. The user doesn't need to think about anything but subclassing when creating new layers.
Downside: Metaclasses are difficult to understand and often frowned upon
Method 2: Traversing the __subclasses__ tree
The second method I thought of was to use the __subclassess__ function of BaseLayer to get a list of all subclasses, then create a dict with Layer.__name__ as keys and Layer as values. See example code below:
def get_subclasses(cls):
"""Returns all classes that inherit from `cls`
"""
subclasses = {
sub.__name__: sub for sub in cls.__subclasses__()
}
subsubclasses = (
get_subclasses(sub) for sub in subclasses.values()
)
subsubclasses = {
name: sub for subs in subsubclasses for name, sub in subs.items()
}
return {**subclasses, ** subsubclasses}
Benefit: Easy to explain how this works.
Downside: We might end up with two layers having the same name.
Method 3: Using a class decorator
The final method is my favourite as it doesn't hide any implementation details in a metaclass, and still manages to prevent multiple classes with the same name.
Here the layers module has a global variable named layers and a decorator named register_layer, which simply adds the decorated classes to the layers dict. See code below.
layers = {}
def register_layer(cls):
if cls.__name__ in layers:
raise ValueError('Cannot have two layers with the same name')
layers[cls.__name__] = cls
return cls
Benefit: No metaclasses and no way of having two layers with the same name.
Downside: Requires a global variable, which is often frowned upon.
So, my question is, which method is preferable? And more importantly, why?
Actually - that is the kind of things metaclases are designed for. As you can see from the options you stated above, it is the simpler and more straightforward design.
They are sometimes "frowned upon" because of two things: (1) people don't understand then and don't care for understanding; (2) people misuse then when they are actually not needed; (3) they are hard to combine - so if any of your classes is to be used with a mixn that have a different metaclass (say abc.ABC), you have also to produce a combining metaclass.
Method 4: __init_subclass__
Now, that said, from Python 3.6, there is a new feature that can cover your usecase without the need for metaclasses: the class __init_subclass__ method:
it is called as a classmethod on the base class when subclasses of it are created.
All you need is to write a proper __init_subclass__ method on your BaseLayer class and have all the benefits you'd have from the implementation in the metaclasses and none of the downsides
Like you, I like the class decorator approach as it is more readable.
You can avoid using a global variable by making the class decorator itself a class, and making layers a class variable instead. You can also avoid possible name collision by joining the target class' name with its module name:
class register_layer:
layers = {}
def __new__(cls, target):
cls.layers['.'.join((target.__module__, target.__name__))] = target
return target
I have a QtGui.QAbstractItemModel object. I'd like to create a new QtGui.QStandardItemModel object based on the QtGui.QAbstractItemModel. Because QtGui.QStandardItemModel is sub-classed from QtGui.QAbstractItemModel I should be able to copy all data from one object to another. How do you do so?
Usually you would something like this:
data_model = QtGui.QAbstractItemModel()
new_data_model = QtGui.QAbstractItemModel(data_model)
but the Constructor does not support instantiating with that type of argument.
Any Ideas?
QAbstractItemModel is an abstract class that can not and should not be instantiated. Its usefulness is mainly for inheritance since it serves as a basis for any type of model such as QStandardItemModel.
In addition to passing as a parent a model to another does not imply that the data will be copied, and finally QAbstractItemModel is in the submodule QtCore, not in QtGui.