The unbearable opaqueness of time.struct_time - python-3.x

Why do pylint and the intellisense features of IDEs have trouble recognizing instances of time.struct_time? The following code contains some trivial tests of existent/non-existent attributes of classes, named tuples and the named-tuple-like time.struct_time. Everything works as expected in pylint, IntelliJ and VSCode - the access to missing attributes is reported in each case except for time.struct_time - it generates no warnings or errors in any of these tools. Why can't they tell what it is and what its attributes are?
import time
from collections import namedtuple
t = time.localtime()
e = t.tm_mday
e = t.bad # this is not reported by linters or IDEs.
class Clz:
cvar = 'whee'
def __init__(self):
self.ivar = 'whaa'
o = Clz()
e = Clz.cvar
e = o.ivar
e = Clz.bad
e = o.bad
Ntup = namedtuple('Ntup', 'thing')
n = Ntup(thing=3)
e = n.thing
e = n.bad
The context of the question is the following recent bug in pipenv -
# Halloween easter-egg.
if ((now.tm_mon == 10) and (now.tm_day == 30))
Obviously, the pass path was never tested but it seems the typical static analysis tools would not have helped here either. This is odd for a type from the standard library.
(Fix can be seen in full at https://github.com/kennethreitz/pipenv/commit/033b969d094ba2d80f8ae217c8c604bc40160b03)

time.struct_time is an object defined in C, which means it can't be introspected statically. The autocompletion software can parse Python code and make a reasonable guess as to what classes and namedtuples support, but they can't do this for C-defined objects.
The work-around most systems use is to generate stub files; usually by introspecting the object at runtime (importing the module and recording the attributes found). For example, CodeIntel (part of the Komodo IDE), uses an XML file format called CIX. However, this is a little more error-prone so such systems then err on the side of caution, and will not explicitly mark unknown attributes as wrong.
If you are coding in Python 3, you could look into using type hinting. For C extensions you still need stub files, but the community is pretty good at maintaining these now. The standard library stub files are maintained in a project called typeshed.
You'd have to add type hints to your project:
#!/usr/bin/env python3
import time
from collections import namedtuple
t: time.struct_time = time.localtime()
e: int = t.tm_mday
e = t.bad # this is not reported by linters or IDEs.
class Clz:
cvar: str = 'whee'
ivar: str
def __init__(self) -> None:
self.ivar = 'whaa'
o = Clz()
s = Clz.cvar
s = o.ivar
s = Clz.bad
s = o.bad
Ntup = namedtuple('Ntup', 'thing')
n = Ntup(thing=3)
e = n.thing
e = n.bad
but then the flake8 tool combined with the flake8-mypy plugin will detect the bad attributes:
$ flake8 test.py
test.py:8:5: T484 "struct_time" has no attribute "bad"
test.py:22:5: T484 "Clz" has no attribute "bad"
test.py:23:5: T484 "Clz" has no attribute "bad"
test.py:28:5: T484 "Ntup" has no attribute "bad"
PyCharm builds on this work too, and perhaps can detect the same invalid use. It certainly directly supports pyi files.

Related

Sphinx: Share function Docstring [duplicate]

I have a class like this:
class MyBase(object):
x = 3
"""Documentation for property x"""
and another class that inherits it:
class MyObj(MyBase):
x = 0
When I use sphinx's autodoc to generate documentation, MyObj.x is not documented. Is there any way to inherit the docstring from MyBase.x? I found DocInherit but since this uses a decorator, it only works for class methods. Any way to do this with properties?
I found a workaround using the property function:
class MyBase(object):
_x = 3
x = property( lambda s: s._x, doc="Documentation for property x")
class MyObj(MyBase):
_x = 0
This is nice in that given an instance variable:
>>> m = MyObj()
>>> m.x
0
one can call help(m) and get proper documentation of property x and sphinx also picks this up correctly.
As far as I know, docstrings for attributes are not part of Python. When I try it, MyBase.x.__doc__ does not get set to the string beneath it. Docstrings only work on classes, functions and methods. If Sphinx picks up the string underneath x = 3 as a docstring, it's probably doing its own processing of the source code to get that.
If you only care for building Documentation via Sphinx. you can use:
":inherited-members:"
.. autoclass:: Noodle
:members:
:inherited-members:
This will also add the doc strings of inherited members in Sphinx Documentation.
http://sphinx-doc.org/ext/autodoc.html
As Thomas already stated, attributes do not have docstrings in Python. Sphinx however provides it's own processing allowing for attributes to be documented.
class Test(object):
#: This is an attibute docstring.
test_attr = 'test'
#property
def test_prop(self):
"""This is a property docstring."""
This results in:
class Test
Bases: object
test_attr = 'test'
This is an attibute docstring.
test_prop
This is a property docstring.

Instance attributes in a subclass of Chem.Atom in rdkit cannot be accessed

I defined a subclass of Atom in rdkit.Chem. I also defined an instance attribute in it but I could not get that instance from RWMol object in rdkit.
Below there is a sample code for my problem:
from rdkit import Chem
class MyAtom(Chem.Atom):
def __init__(self, symbol, **kwargs):
super().__init__(symbol, **kwargs)
self.my_attribute = 0
def get_my_attribute(self):
return self.my_attribute
if __name__ == '__main__':
rw_mol = Chem.RWMol()
# I created MyAtom class object then added to RWMol. But I couldn't get it again.
my_atom = MyAtom('C')
my_atom.my_attribute = 3
rw_mol.AddAtom(my_atom)
atom_in_mol = rw_mol.GetAtoms()[0]
# I can access my_atom new defined attributes.
print(my_atom.get_my_attribute())
# below two line gives error: AttributeError: 'Atom' object has no attribute 'get_my_attribute'
print(atom_in_mol.get_my_attribute())
print(atom_in_mol.my_attribute)
# type(atom1): <class '__main__.MyAtom'>
# type(atom_in_mol): <class 'rdkit.Chem.rdchem.Atom'>
# Why below atom types are different? Thanks to polymorphism, that two object types must be same.
Normally this code must run but it gives error due to last line because atom_in_mol object type is Chem.Atom. But should it be MyAtom? I also cannot access my_attribute directly.
rdkit Python library is a wrapper of C++. So is the problem this? Cannot I use inheritance for this library?
Note: I researched rdkit documentation and there is a SetProp method for saving values in atoms. It uses dictionary to save values. It runs fine but it is too slow for my project. I want to use instance attributes to save my extra values. Is there any solution for that inheritance problem, or faster different solution?
Python RDKit library is a C++ wrapper, so sometimes it does not follows the conventional Python object handling.
To go deeper, you will have to dig through the source code:
rw_mol.AddAtom(my_atom)
Above will execute AddAtom method in rdkit/Code/GraphMol/Wrap/Mol.cpp, which, in turn, calls addAtom method in rdkit/Code/GraphMol/RWMol.h, which then calls addAtom method in rdkit/Code/GraphMol/ROMol.cpp with default argument of updateLabel = true and takeOwnership = false.
The takeOwnership = false condition makes the argument atom to be duplicated,
// rdkit/Code/GraphMol/ROMol.cpp
if (!takeOwnership)
atom_p = atom_pin->copy();
else
atom_p = atom_pin;
Finally, if you look into what copy method do in rdkit/Code/GraphMol/Atom.cpp
Atom *Atom::copy() const {
auto *res = new Atom(*this);
return res;
}
So, it reinstantiate Atom class and returns it.

Access Violation when using Ctypes to Interface with Fortran DLL

I have a set of dlls created from Fortran that I am running from python. I've successfully created a wrapper class and have been running the dlls fine for weeks.
Today I noticed an error in my input and changed it, but to my surprise this caused the following:
OSError: exception: access violation reading 0x705206C8
If seems that certain input values somehow cause me to try to access illegal data. I created the following MCVE and it does repeat the issue. Specifically an error is thrown when 338 < R_o < 361. Unfortunately I cannot publish the raw Fortran code, nor create an MCVE which replicates the problem and is sufficiently abstracted such that I could share it. All of the variables are either declared as integer or real(8) types in the Fortran code.
import ctypes
import os
DLL_PATH = "C:\Repos\CASS_PFM\dlls"
class wrapper:
def __init__(self,data):
self.data = data
self.DLL = ctypes.CDLL(os.path.join(DLL_PATH,"MyDLL.dll"))
self.fortran_subroutine = getattr(self.DLL,"MyFunction_".lower())
self.output = {}
def run(self):
out = (ctypes.c_longdouble * len(self.data))()
in_data = []
for item in self.data:
item.convert_to_ctypes()
in_data.append(ctypes.byref(item.c_val))
self.fortran_subroutine(*in_data, out)
for item in self.data:
self.output[item.name] = item.convert_to_python()
class FortranData:
def __init__(self,name,py_val,ctype,some_param=True):
self.name = name
self.py_val = py_val
self.ctype = ctype
self.some_param = some_param
def convert_to_ctypes(self):
ctype_converter = getattr(ctypes,self.ctype)
self.c_val = ctype_converter(self.py_val)
return self.c_val
def convert_to_python(self):
self.py_val = self.c_val.value
return self.py_val
def main():
R_o = 350
data = [
FortranData("R_o",R_o,'c_double',False),
FortranData("thick",57.15,'c_double',False),
FortranData("axial_c",100,'c_double',False),
FortranData("sigy",235.81,'c_double',False),
FortranData("sigu",619.17,'c_double',False),
FortranData("RO_alpha",1.49707,'c_double',False),
FortranData("RO_sigo",235.81,'c_double',False),
FortranData("RO_epso",0.001336,'c_double',False),
FortranData("RO_n",6.6,'c_double',False),
FortranData("Resist_Jic",116,'c_double',False),
FortranData("Resist_C",104.02,'c_double',False),
FortranData("Resist_m",0.28,'c_double',False),
FortranData("pressure",15.51375,'c_double',False),
FortranData("i_write",0,'c_int',False),
FortranData("if_flag_twc",0,'c_int',),
FortranData("i_twc_ll",0,'c_int',),
FortranData("i_twc_epfm",0,'c_int',),
FortranData("i_err_code",0,'c_int',),
FortranData("Axial_TWC_ratio",0,'c_double',),
FortranData("Axial_TWC_fail",0,'c_int',),
FortranData("c_max_ll",0,'c_double',),
FortranData("c_max_epfm",0,'c_double',)
]
obj = wrapper(data)
obj.run()
print(obj.output)
if __name__ == "__main__": main()
It's not just the R_o value either; there are some combinations of values that cause the same error (seemingly without rhyme or reason). Is there anything within the above Python that might lead to an access violation depending on the values passed to the DLL?
Python version is 3.7.2, 32-bit
I see 2 problems with the code (and a potential 3rd one):
argtypes (and restype) not being specified. Check [SO]: C function called from Python via ctypes returns incorrect value (#CristiFati's answer) for more details
This may be a consequence (or at least it's closely related to) the previous. I can only guess without the Fortran (or better: C) function prototype, but anyway there is certainly something wrong. I assume that for the input data things should be same as for the output data, so the function would take 2 arrays (same size), and the input one's elements would be void *s (since their type is not consistent). Then, you'd need something like (although I can't imagine how would Fortran know which element contains an int and which a double):
in_data (ctypes.c_void_p * len(self.data))()
for idx, item in enumerate(self.data):
item.convert_to_ctypes()
in_data[index] = ctypes.addressof(item.c_val)
Since you're on 032bit, you should also take calling convention into account (ctypes.CDLL vs ctypes.WinDLL)
But again, without the function prototype, everything is just a speculation.
Also, why "MyFunction_".lower() instead of "myfunction_"?

Module run inside module returns - TypeError: 'numpy.float64' object cannot be interpreted as an integer

I am running a function inside another function in a loop and having a problem when the first module (Mod1) uses values defined in the second module (Mod2) to generate the result for the equation called Overall.
The error returned is thus: TypeError: 'numpy.float64' object cannot be interpreted as an integer
The code is as follows
def Mod1(A,
B,
C):
As = pd.read_csv('AProps.csv',index_col='AName')
As = As.dropna(0,how='all',thresh=None,subset=None,inplace=False)
AvailAs = As.index.tolist()
Bs = pd.read_csv('BProps.csv',index_col='BName')
Bs = Matrices.dropna(0,how='all',thresh=None, subset=None,inplace=False)
AvailBs = Bs.index.tolist()
Prop1_A = As['Prop1'][A]
Prop2_A = As['Prop2'][A]
Prop3_A = As['Prop3'][A]
Prop4_A = Prop1_A/(2*(1+Prop3_A))
Prop1_B = Bs['Prop1'][B]
Prop3_B = Bs['Prop3'][B]
Prop4 = Prop1_B/(2*(1+Prop3_B)
Overall = Prop1_A*C+Prop1_B*(1-C)
Return Overall
def Mod2(NumItems):
A_List = []
B_List = []
C_List = []
for i in range(NumItems):
A_List.append(input('Choose the A_Input for item {0}: '.format(i+1)))
B_List.append(input('Choose the B_input for item {0}: '.format(i+1)))
C_List.append(input('Choose the C_input for item {0}: '.format(i+1)))
for i in range (NumLams):
Func1(A_List[i],B_List[i],C_List[i])
Just to be clear, the intention of the code is that Mod2 creates lists of inputs which are then used within Mod1. In Mod1 inputs A and B are used to pull outputs from csv files which are read into As and Bs. Input C is used in functions within Mod1, ie Overall.
When I run Mod1 manually, there are no issues at all. But when run in conjunction with Mod2 as outlined above...
The code in Mod1 runs through using the values Prop1_A and Prop1_B pulled from As and Bs without issue, but when it comes to run the function for Overall the aforementioned error is returned within Python.
I am certain that the issue is with the way the value for C is interpreted, ie as a float64, but don’t understand why python might be expecting to see an integer there.
I am using Spyder 3.2.6 as installed with the Anaconda package.
Any help given is gratefully received.
I solved the problem by explicitly defining all the selections from the csv, Prop_1 etc, in Mod1 as floats. A simple answer really.

lxml - use default class element lookup and TreeBuilder parser target at the same time

With lxml, is there a way to use a ElementDefaultClassLookup and a parser target derived from the TreeBuilder class at the same time?
I have tried:
from lxml import etree
class MyElement(etree.ElementBase):
pass
class MyComment(etree.CommentBase):
pass
class MyTreeBuilder(etree.TreeBuilder):
pass
parser_lookup = etree.ElementDefaultClassLookup(element=MyElement, comment=MyComment)
parser = etree.XMLParser(target=MyTreeBuilder()) # (My)TreeBuilder accepts a `parser` argument keyword (which will then use the class lookup from that parser), but we haven't created the parser yet!
parser.set_element_class_lookup(parser_lookup)
xml_string = '<root xmlns:test="hello"><element test:foobar="1" /><!-- world --></root>'
root = etree.fromstring(xml_string, parser=parser)
print(type(root), type(root.xpath('//comment()[1]')[0]), etree.tostring(root))
which results in:
<class 'lxml.etree._Element'> <class 'lxml.etree._Comment'> b'<root><element xmlns:ns0="hello" ns0:foobar="1"/><!-- world --></root>'
I want:
<class 'MyElement'> <class 'MyComment'> b'<root xmlns:test="hello"><element test:foobar="1"/><!-- world --></root>'
Notice the XML namespace differences as well as the Python classes.
I'm using lxml 3.4.4.
I can get the correct namespace prefix on the attribute by using:
parser = etree.XMLParser(target=etree.TreeBuilder())
which doesn't make any sense to me - why doesn't my derived class behave the same way? (I realize that omitting the target argument will by default use a TreeBuilder anyway.)
and I can get the correct class and namespace prefix by using:
parser = etree.XMLParser()
but I specifically want to use my own "target" to create a tree, and ideally don't want to have to reinvent the wheel.
I have tried to set the target after the parser is initialized, like so:
parser.target = MyTreeBuilder(parser=parser)
but this gives an error:
AttributeError: attribute 'target' of 'lxml.etree._BaseParser' objects is not writable
I checked the source code for the TreeBuilder class, and tried:
class MyTreeBuilder(etree.TreeBuilder):
def set_parser(self, parser):
super()._parser = parser
parser_lookup = etree.ElementDefaultClassLookup(element=MyElement, comment=MyComment)
tb = MyTreeBuilder()
parser = etree.XMLParser(target=tb)
parser.set_element_class_lookup(parser_lookup)
tb.set_parser(parser)
which gives:
AttributeError: 'super' object has no attribute '_parser'
and I tried:
parser_lookup = etree.ElementDefaultClassLookup(element=MyElement, comment=MyComment)
fake_parser = etree.XMLParser()
fake_parser.set_element_class_lookup(parser_lookup)
tb = MyTreeBuilder(parser=fake_parser)
parser = etree.XMLParser(target=tb)
parser.set_element_class_lookup(parser_lookup)
which gives the correct classes, but not still the correct attribute namespaces. I therefore imagine it needs some information from the correct parser to be able to build the tree correctly.
I tried setting the element_factory keyword argument instead of, and as well as, parser - but got the same result:
tb = MyTreeBuilder(element_factory=fake_parser.makeelement)
# and
tb = MyTreeBuilder(element_factory=fake_parser.makeelement, parser=fake_parser)
EDIT: looking at the lxml class lookup code, it seems that one can set a class attribute called PARSER on the custom Element.
I tried that:
MyElement.PARSER = parser
but the result was the same as without it.
However, the makeelement method of the parser works as expected:
test = parser.makeelement('root', attrib={ '{hello}foobar': '1' }, nsmap={ 'test': 'hello' })
print(type(test), etree.tostring(test))
as it gives (obviously the attribute is on a different node than when I parse the string):
<class 'MyElement'> b'<root xmlns:test="hello" test:foobar="1"/>'
Combining these two approaches using:
def m(tag, attrib=None,nsmap=None, *children):
return MyElement.PARSER.makeelement(tag, attrib=attrib, nsmap=nsmap, *children)
parser = etree.XMLParser(target=MyTreeBuilder(element_factory=m))
gives the correct classes but still incorrect namespaces.
Is what I want possible? Why do the namespaces "go wrong" as soon as I use a custom target that isn't a pure TreeBuilder? Is there something I need to do differently somewhere, to manually correct the namespace behavior? Will I have to make my own tree builder implementation?

Resources