This is Python 3.10. I use a PriorityQueue as a way to track Actors' turn order in my game. It's just a simple roguelike. I don't use the synchronization features of the PriorityQueue. My code:
import pickle
from queue import PriorityQueue
class GameEngine():
def __init__(self):
self.pqueue = PriorityQueue()
def save_to_file(self):
with open('save.pkl', 'wb') as file:
pickle.dump(self, file, pickle.HIGHEST_PROTOCOL)
class Monster():
pass
engine = GameEngine()
orc1 = Monster()
orc2 = Monster()
engine.pqueue.put((20,orc1))
engine.pqueue.put((10,orc2))
engine.save_to_file()
It returns TypeError: cannot pickle '_thread.lock' object. From what I understand PriorityQueue is not pickle-able. I've read here that Queue.Queue has a pickle-able alternative of collections.deque if the synchronization stuff is not necessary. Is there such an alternative to PriorityQueue, or is there a way to pickle it anyway? Other than implementing my own simplified version of PriorityQueue?
As you don't need the synchronisation features of PriorityQueue, just use the light-weight heapq module. It provides functions (not methods) to work on a plain list:
import pickle
from heapq import heappush, heappop
class GameEngine():
def __init__(self):
self.pqueue = []
def save_to_file(self):
with open('save.pkl', 'wb') as file:
pickle.dump(self, file, pickle.HIGHEST_PROTOCOL)
class Monster():
pass
engine = GameEngine()
orc1 = Monster()
orc2 = Monster()
heappush(engine.pqueue, (20,orc1))
heappush(engine.pqueue, (10,orc2))
engine.save_to_file()
Related
From this, I was able to make this:
import os
import types
import zipfile
import sys
import io
class ZipImporter(object):
def __init__(self, zip_file):
self.zfile = zip_file
self._paths = [x.filename for x in self.zfile.filelist]
def _mod_to_paths(self, fullname):
# get the python module name
py_filename = fullname.replace(".", os.sep) + ".py"
# get the filename if it is a package/subpackage
py_package = fullname.replace(".", os.sep) + "/__init__.py"
print(py_package)
if py_filename in self._paths:
return py_filename
elif py_package in self._paths:
return py_package
else:
return None
def find_module(self, fullname, path):
if self._mod_to_paths(fullname) is not None:
return self
return None
def load_module(self, fullname):
filename = self._mod_to_paths(fullname)
if not filename in self._paths:
raise ImportError(fullname)
new_module = types.ModuleType(fullname)
new_module.__name__=fullname
print(fullname)
exec(self.zfile.open(filename, 'r').read(),new_module.__dict__)
new_module.__file__ = filename
new_module.__loader__ = self
if filename.endswith("__init__.py"):
new_module.__path__ = []
new_module.__package__ = fullname
else:
new_module.__package__ = fullname.rpartition('.')[0]
sys.modules[fullname]=new_module
return new_module
module_zip=zipfile.ZipFile(io.BytesIO(),"w")
for key in module_dict:
module_zip.writestr(key,module_dict[key])
sys.meta_path.append(ZipImporter(module_zip))
import pyparsing
Using the source code of pyparsing as a test. However, it fails with ImportError: attempted relative import with no known parent package. Even if I replace all the relative imports with absolute imports, it fails with RecursionError: maximum recursion depth exceeded while calling a Python object, as it tries to import pyparsing repeatedly. Is there something fundamental I'm not understanding about the way Python's import system works?
I found the answer --- PEP 302 says that:
Note that the module object must be in sys.modules before the loader executes the module code. This is crucial because the module code may (directly or indirectly) import itself; adding it to sys.modules beforehand prevents unbounded recursion in the worst case and multiple loading in the best.
I am using multiprocessing.Pool to speed up computation, as I call one function multiple times, and then collate the result. Here is a snippet of my code:
import multiprocessing
from functools import partial
def Foo(id:int,constant_arg1:str, constant_arg2:str):
custom_class_obj = CustomClass(constant_arg1, constant_arg2)
custom_class_obj.run() # this changes some attributes of the custom_class_obj
if(something):
return None
else:
return [custom_class_obj]
def parallel_run(iters:int, a:str, b:str):
pool = multiprocessing.Pool(processes=k)
## create the partial function obj before passing it to pool
partial_func = partial(Foo, constant_arg1=a, constant_arg2=b)
## create the variable id list
iter_list = list(range(iters))
all_runs = pool.map(partial_func, iter_list)
return all_runs
This throws the following error in the multiprocessing module:
multiprocessing.pool.MaybeEncodingError: Error sending result: '[[<CustomClass object at 0x1693c7070>], [<CustomClass object at 0x1693b88e0>], ....]'
Reason: 'TypeError("cannot pickle 'module' object")'
How can I resolve this?
I was able to replicate the error message with a minimal example of an un-picklable class. The error basically states the instance of your class can't be pickled because it contains a reference to a module, and modules are not picklable. You need to comb through CustomClass to make sure instances don't hold things like open file handles, module references, etc.. If you need to have those things, you should use __getstate__ and __setstate__ to customize the pickle and unpickle process.
distilled example of your error:
from multiprocessing import Pool
from functools import partial
class klass:
def __init__(self, a):
self.value = a
import os
self.module = os #this fails: can't pickle a module and send it back to main process
def foo(a, b, c):
return klass(a+b+c)
if __name__ == "__main__":
with Pool() as p:
a = 1
b = 2
bar = partial(foo, a, b)
res = p.map(bar, range(10))
print([r.value for r in res])
using Python 3.6 or newer, I want to type hint a function myfunc that returns an object of MyClass.
How can I hint, that myqueue is a deque filled with MyClass objects?
from collections import deque
global_queue = deque()
class MyClass:
pass
def myfunc(myqueue=global_queue) -> MyClass:
return myqueue.popleft()
for i in range(10):
global_queue.append(MyClass())
In Python 3.9, you can use deque['MyClass']() directly.
If you are using Python 3.6.1 or higher, you can use typing.Deque:
from typing import Deque
from collections import deque
global_queue: Deque['MyClass'] = deque()
class MyClass:
pass
def myfunc(myqueue: Deque[MyClass] = global_queue) -> MyClass:
return myqueue.popleft()
for i in range(10):
global_queue.append(MyClass())
Alternatively, you can do global_queue = Deque['MyClass']() instead -- at runtime, that'll construct a collections.deque object.
If you need to support Python 3.5, install the typing_extensions 3rd party library and do from typing_extensions import Deque. That library contains backports of types that were added after the typing module was first added to the standard library.
im using multithreading in python3 with Flask as below.
Would like to know if there is any issue in below code, and if this is efficient way of using threads
import _thread
COUNT = 0
class Myfunction(Resource):
#staticmethod
def post():
global GLOBAL_COUNT
logger = logging.getLogger(__name__)
request_json = request.get_json()
logger.info(request_json)
_thread.start_new_thread(Myfunction._handle_req, (COUNT, request_json))
COUNT += 1
return Response("Request Accepted", status=202, mimetype='application/json')
#staticmethod
def _handle_req(thread_id, request_json):
with lock:
empID = request_json.get("empId", "")
myfunction2(thread_id,empID)
api.add_resource(Myfunction, '/Myfunction')
I think the newer module threading would be better suited for python 3. Its more powerful.
import threading
threading.Thread(target=some_callable_function).start()
or if you wish to pass arguments
threading.Thread(target=some_callable_function,
args=(tuple, of, args),
kwargs={'dict': 'of', 'keyword': 'args'},
).start()
Unless you specifically need _thread for backwards compatibility. Not specifically related to how efficient your code is but good to know anyways.
see What happened to thread.start_new_thread in python 3 and https://www.tutorialspoint.com/python3/python_multithreading.htm
I'm trying to use pickle to save one of my objects but I face this error when trying to dump it:
TypeError: can't pickle _thread.lock objects
It is not clear to me, because I'm not using any locks inside my code. I tried to reproduce this error:
import threading
from time import sleep
import pickle
class some_class:
def __init__(self):
self.a = 1
thr = threading.Thread(target=self.incr)
self.lock = threading.Lock()
thr.start()
def incr(self):
while True:
# with self.lock:
self.a += 1
print(self.a)
sleep(0.5)
if __name__ == "__main__":
a = some_class()
val = pickle.dumps(a, pickle.HIGHEST_PROTOCOL)
print("pickle done!")
pickle_thread.py", line 22, in
val = pickle.dumps(a, pickle.HIGHEST_PROTOCOL) TypeError: can't pickle _thread.lock objects
If I define a thread lock inside my object I can't pickle it, right?
I think the problem here is using threading.lock but is there any workaround for this?
Actually, in my main project, I can't find any locks but I've used lots of modules that I can't trace them. What should I look for?
Thanks.
You can try to customize the pickling method for this class by excluding unpicklable objects from the dictionary:
def __getstate__(self):
state = self.__dict__.copy()
del state['lock']
return state
When unpickling, you can recreate missing objects manually, e.g.:
def __setstate__(self, state):
self.__dict__.update(state)
self.lock = threading.Lock() # ???
I don't know enough about the threading module to predict if this is gonna be sufficient.