I'm trying klepto as a replacement for shelve to persist class instances but am seeing a lengthy traceback when attempting to do so.
My platform is:
python 3.7.3
klepto 0.1.6
The example #
Pickle versus shelve storing large dictionaries in Python
... works fine on the above platform.
from dataclasses import dataclass
#dataclass (frozen=True)
class Target:
name : str
#dataclass (frozen=True)
class Image:
target :Target
image_1 = Image (target=Target(name='Crab Nebula'))
image_2 = Image (target=Target(name='Orion Nebula'))
##keys = image_1.name, image_2.name
keys = image_1.target.name, image_2.target.name
values = image_1, image_2
d = dict ( zip (keys,values))
import klepto
a = klepto.archives.dir_archive (dict=d)
a.dump()
a.dump() initiates a lengthy traceback beginning with ...
File "C:\Users\jgv\Documents\Projects\AstroPix_ODB (klepto)\stackoverflow.py", line 20, in <module>
a.dump()
File "C:\Users\jgv\AppData\Local\Programs\Python\Python37\lib\site-packages\klepto\_archives.py", line 165, in dump
self.archive.update(self)
and concluding with ...
File "C:\Users\jgv\AppData\Local\Programs\Python\Python37\lib\site-packages\dill\_dill.py", line 1148, in save_dictproxy
raise ReferenceError("%s does not reference a class __dict__" % obj)
ReferenceError: {} does not reference a class __dict__
Related
Goal: read in parameters from .yaml to pass to functions during runtime.
I've seen no reference to this error online; so decided to make a post.
I have a user-defined params.yaml:
my_params:
host: txt
project: txt
roi_term_id: 123
# ...
That is read-in by params.py:
import os
from dataclasses import dataclass
from pathlib import Path
import yaml
from decouple import config
from typed_json_dataclass import TypedJsonMixin
#dataclass
class MyParams(TypedJsonMixin):
host: str
project: str
roi_term: str
def __post_init__(self):
self.public_key = config('KEY')
assert isinstance(self.public_key, str)
self.private_key = config('SECRET')
assert isinstance(self.private_key, str)
super().__post_init__()
# ...
#dataclass
class Params(TypedJsonMixin):
my_params: MyParams
# ...
def load_params_dict():
parameter_file = 'params.yaml'
cwd = Path(os.getcwd())
params_path = cwd / parameter_file
if params_path.exists():
params = yaml.safe_load(open(params_path))
else: # If this script is being called from the path directory
params_path = cwd.parent / parameter_file
params = yaml.safe_load(open(params_path))
return params
params_dict = load_params_dict()
print(params_dict)
project_params = Params.from_dict(params_dict)
Traceback:
File "/home/me/miniconda3/envs/myvenv/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 152, in __post_init__
expected_type(**field_value)
TypeError: __init__() got an unexpected keyword argument 'roi_term_id'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "path/main.py", line 7, in <module>
from params import project_params
File "/home/me/PycharmProjects/project/path/params.py", line 89, in <module>
project_params = Params.from_dict(params_dict)
File "/home/me/miniconda3/envs/myvenv/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 248, in from_dict
return cls(**raw_dict)
File "<string>", line 9, in __init__
File "/home/me/miniconda3/envs/myvenv/lib/python3.7/site-packages/typed_json_dataclass/typed_json_dataclass.py", line 155, in __post_init__
raise TypeError(f'{class_name}.{field_name} '
TypeError: Params.my_params is expected to be <class 'params.MyParams'>, but value {'host': 'txt', 'project': 'txt', 'roi_term_id': 123} is a dict with unexpected keys
2 things, name and dtype.
Subject:
roi_term: str
Name:
Keys' names in params.yaml must be the exact same as the attributes' names in a class (and I assume order).
dtype:
Attribute in class says str, but when the params.yaml file gets parsed it is considered int - since the value in the file is a whole number. I changed str to int in MyParams class.
Thus, roi_term_id from params.yaml conflicted with roi_term from MyParams class.
I'm creating a new testing frameworks, I started to implement my own functions in *.py files, but when I try to run test, I've got following stack:
(venv) PLAMWS0024:OAT user$ robot -v CONFIG_FILE:"/variables-config.robot" ./catalog/tests/test1.robot
Traceback (most recent call last):
File "/Users/user/PycharmProjects/OAT/venv/bin/robot", line 5, in <module>
from robot.run import run_cli
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/__init__.py", line 44, in <module>
from robot.rebot import rebot, rebot_cli
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/rebot.py", line 45, in <module>
from robot.run import RobotFramework
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/run.py", line 44, in <module>
from robot.running.builder import TestSuiteBuilder
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/running/__init__.py", line 98, in <module>
from .builder import TestSuiteBuilder, ResourceFileBuilder
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/running/builder/__init__.py", line 16, in <module>
from .builders import TestSuiteBuilder, ResourceFileBuilder
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/running/builder/builders.py", line 20, in <module>
from robot.parsing import SuiteStructureBuilder, SuiteStructureVisitor
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/parsing/__init__.py", line 380, in <module>
from .model import ModelTransformer, ModelVisitor
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/parsing/model/__init__.py", line 18, in <module>
from .statements import Statement
File "/Users/user/PycharmProjects/OAT/venv/lib/python3.8/site-packages/robot/parsing/model/statements.py", line 453, in <module>
class Error(Statement, Exception):
TypeError: multiple bases have instance lay-out conflict
I suspect it's because in one of my files I'm trying to get variables from Robot Framework built in functionalities.
and I'm thinking it's because I'm trying to use protected methods, but I am not sure.
I found issue TypeError: multiple bases have instance lay-out conflict and it shows that there might be a mismatch in naming convention (or am I wrong?), but my project is a bit small, so the only option is that Robot can't see the function itself.
What can I miss?
Some code:
Test itself:
*** Settings ***
Documentation TO BE CHANGED
... SET IT TO CORRECT DESCRIPTION
Library ${EXECDIR}/file.py
Library String
*** Test Cases ***
User can do stuff
foo bar
from datetime import datetime
from robot.api import logger
from robot.libraries.BuiltIn import _Variables
from robot.parsing.model.statements import Error
import json
import datetime
from catalog.resources.utils.clipboardContext import get_value_from_clipboard
Vars = _Variables()
def foo_bar(params):
# Get all variables
country = get_value_from_clipboard('${COUNTRY}')
address = get_value_from_clipboard('${ADDRESS}')
city = get_value_from_clipboard('${CITY}')
postcode = get_value_from_clipboard('${POSTALCODE}')
And calling Vars:
from robot.libraries.BuiltIn import _Variables
from robot.parsing.model.statements import Error
Vars = _Variables()
def get_value_from_clipboard(name):
"""
Returns value saved inside variables passed in Robot Framework
:param name: name of the variable, needs to have ${} part
as example: ${var} passed in config file
:return: value itself, passed as string
"""
try:
return Vars.get_variable_value(name)
except Error as e:
raise Error('Missing parameter in the clipboard, stack:' + str(e))
What fixed issue:
uninstall all requirements from requirements.txt file and install all one-by-one.
Additional steps I tried:
comment out all files one-by-one and run only robot command - failed, got same errors
cleaned vnenv as described here: How to reset virtualenv and pip? (failed)
check out if any variable has same naming as described in python3.8/site-packages/robot/parsing/model/statements.py - none
So looks like there was some clash in installing requirements by PyCharm IDE
I'm trying to edit the metadata Title field of PDFs, to include the ASCII equivalents when possible. I'm using Python3 and the module pdfrw.
How can I do string operations that replace the metadata fields?
My test code is here:
from pdfrw import PdfReader, PdfWriter, PdfString
import unicodedata
def edit_title_metadata(inpdf):
trailer = PdfReader(inpdf)
# this statement is breaking pdfrw
trailer.Info.Title = unicode_normalize(trailer.Info.Title)
# also have tried:
#trailer.Info.Title = PdfString(unicode_normalize(trailer.Info.Title))
PdfWriter("test.pdf", trailer=trailer).write()
return
def unicode_normalize(s):
return unicodedata.normalize('NFKD', s).encode('ascii', 'ignore')
if __name__ == "__main__":
edit_title_metadata('Anadon-2011-Scientific Opinion on the safety e.pdf')
And the traceback is:
Traceback (most recent call last):
File "get_metadata.py", line 68, in <module>
main()
File "get_metadata.py", line 54, in main
edit_title_metadata(pdf)
File "get_metadata.py", line 11, in edit_title_metadata
trailer.Info.Title = PdfString(unicode_normalize(trailer.Info.Title))
File "get_metadata.py", line 18, in unicode_normalize
return unicodedata.normalize('NFKD', s).encode('ascii', 'ignore')
File "/path_to_python/python3.7/site-packages/pdfrw/objects/pdfstring.py", line 550, in encode
if isinstance(source, uni_type):
TypeError: isinstance() arg 2 must be a type or tuple of types
Notes:
This issue at GitHub may be related.
FWIW, Also getting same error with Python3.6
I've shared the pdf (which has non-ascii hyphens, unicode char \u2010)
.
wget https://gist.github.com/philshem/71507d4e8ecfabad252fbdf4d9f8bdd2/raw/cce346ab39dd6ecb3a718ad3f92c9f546761e87b/Anadon-2011-Scientific%2520Opinion%2520on%2520the%2520safety%2520e.pdf
You have to use the .decode() method on the metadata fields:
trailer.Info.Title = unicode_normalize(trailer.Info.Title.decode())
And full working code:
from pdfrw import PdfReader, PdfWriter, PdfReader
import unicodedata
def edit_title_metadata(inpdf):
trailer = PdfReader(inpdf)
trailer.Info.Title = unicode_normalize(trailer.Info.Title.decode())
PdfWriter("test.pdf", trailer=trailer).write()
return
def unicode_normalize(s):
return unicodedata.normalize('NFKD', s).encode('ascii', 'ignore')
if __name__ == "__main__":
edit_title_metadata('Anadon-2011-Scientific Opinion on the safety e.pdf')
I was trying to use the code snippet from this tutorial:
http://newcoder.io/scrape/part-4/
The following is the file with the class definition:
from sqlalchemy.orm import sessionmaker
from .models import Deals, db_connect, create_deals_table
class Test(object):
def __init__(self, args):
"""
Initializes database connection and sessionmaker.
Creates deals table.
"""
engine = db_connect()
create_deals_table(engine)
self.Session = sessionmaker(bind=engine)
def add_item(self):
session = self.Session()
tester = Deals(title="test 3 deal",location='here', price=2.00)
session.add(tester)
session.commit()
When I call Test.add_item(args), the following error is generated.
(sql) [bucket#localhost heap]$ python heap.py test Traceback (most
recent call last): File "heap.py", line 13, in <module>
main(sys.argv) File "heap.py", line 8, in main
Test.add_item(args) File "/home/bucket/src/heap/game/engine.py", line 17, in add_item
session = self.Session() AttributeError: 'list' object has no attribute 'Session'
The code worked fine and added a row when I ran the the same code as a python script with no class definition.
You're trying to call an instance method on the class itself: Test.add_item(args)
You define the method as only having self and no parameters. This means that you need to call it on an instance of the class, and pass it no arguments
test = Test()
test.add_item()
The reason you're getting that particular error is because you're calling the function with the list argument args getting used as the parameter self, and then the self.Session() line is attempting to get the attribute Session from the args object. You need to fix either the method definition or your call to it (e.g., def add_item(self, args):)
I have a file called entities.py, which contains the following code:
class EntityClass:
entities = {}
def __init__(self, parent=None):
.......
def show_all(self):
......
But then, when I run python 3 and type the command as follows:
>>> import entities
>>> ent = entities.EntityClass()
>>> ent.show_all()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'EntityClass' object has no attribute 'show_all'
show_all should clearly be an attribute for EntityClass.
This of course worked perfectly in Python 2, and I'm assuming it's a Python 3 issue...
Is there a work around for this?
From the code posted, it looks like your indentation levels are wrong, you have declared the show_all() method on the module, not the class.
def_show_all(self): Should be indented to the same level as entities = {}
class EntityClass:
entities ={}
def __init__(self,parent=None):
.......
def show_all(self):
......