What is the relationship between Vala VAPI and GObject Introspection? - introspection

First, some context: I'm a Python developer who has written a medium-sized application using PyGObject, taking advantage of GObject Introspection to access things like GSettings, etc. Some of my Python objects actually subclass GObject.GObject, so I'm using GObject quite extensively.
Recently, a certain library has come to my attention that wraps a C library in GObject (gexiv2, used by Shotwell/Vala), however it doesn't currently support introspection. I'm interested in adding introspection support to gexiv2 so that I can access it from Python, but I don't even know where to begin on this topic.
When I research introspection and VAPI, I see lots of documentation referencing the fact that the VAPI can be automatically generated from the introspection annotations... but what about a project that already has a VAPI, but no introspection? Is it possible to automatically generate the introspection annotations given the VAPI?
Thanks.

VAPI bindings are not necessarily related to GObject introspection. For instance, there are VAPI bindings for POSIX, Linux, libudev, and other things that are definitely not GObject-based. There isn't a direct way to convert a VAPI to a GObject binding.
However, if you have C header files and a working library, then you can generally build a GObject introspection file from the library. For gexiv2, download and build the source, then execute:
g-ir-scanner -I gexiv2 gexiv2/gexiv2-{metadata,managed-stream,preview-properties,preview-image,log,startup}.h -n GExiv2 --library libgexiv2.la --pkg gobject-2.0
And this will produce a GIR binding (XML) that you can use in Python.

Well, after getting sick of the tediousness of hand-copying VAPI definitions into introspection annotations, I wrote this (crude) script to do it for me:
#!/bin/env python
import sys
from collections import defaultdict
ANNOTATION = """/**
* %s:
%s *
* Returns:%s
*/
"""
PARAMETER = """ * #%s:%s
"""
methods = defaultdict(set)
attrs = defaultdict(dict)
with open(sys.argv[1]) as vapi:
for line in vapi:
tokens = line.split()
try:
names = tuple(tokens[0].split('.'))
except IndexError:
continue
attrs[names] = {}
for attribute in tokens[1:]:
key, val = attribute.split('=')
if val == '"1"': val = True
if val == '"0"': val = False
attrs[names][key] = val
methods[names[0]]
if len(names) > 1:
methods[names[0]].add(names[-1])
for method in methods:
params = ''
for param in methods[method]:
param_attributes = ''
param_attrs = attrs[(method, param)]
if param_attrs.get('hidden'):
param_attributes += ' (skip)'
if param_attrs.get('is_out'):
param_attributes += ' (out)'
if param_attrs.get('transfer_ownership'):
param_attributes += ' (transfer full)'
elif 'transfer_ownership' in param_attrs:
param_attributes += ' (transfer none)'
if param_attrs.get('array_null_terminated'):
param_attributes += ' (array zero-terminated=1)'
if param_attrs.get('array_length_pos'):
param_attributes += ' (array length=FIXME)'
if param_attributes:
param_attributes += ':'
params += PARAMETER % (param, param_attributes)
attributes = ''
method_attrs = attrs[(method,)]
if method_attrs.get('transfer_ownership'):
attributes += ' (transfer full)'
elif 'transfer_ownership' in method_attrs:
attributes += ' (transfer none)'
if method_attrs.get('nullable'):
attributes += ' (allow-none)'
if method_attrs.get('array_null_terminated'):
attributes += ' (array zero-terminated=1)'
if attributes:
attributes += ':'
print ANNOTATION % (method, params, attributes)
This obviously has some disadvantages: It doesn't insert the annotations into the code, it simply prints them, so you have to do quite a bit of copy & pasting to get everything into the right place. It also doesn't handle arrays very well, but it at least lets you know when there's an array you need to fix manually. All in all, it was significantly less work to run this script and then massage the results than it was to parse by hand. I'm posting it here in the hopes that it gets picked up by google and somebody else may benefit one day (although I dearly hope that all GObject-based projects from here on out simply start with annotations and then use vapigen).

Related

Access Violation when using Ctypes to Interface with Fortran DLL

I have a set of dlls created from Fortran that I am running from python. I've successfully created a wrapper class and have been running the dlls fine for weeks.
Today I noticed an error in my input and changed it, but to my surprise this caused the following:
OSError: exception: access violation reading 0x705206C8
If seems that certain input values somehow cause me to try to access illegal data. I created the following MCVE and it does repeat the issue. Specifically an error is thrown when 338 < R_o < 361. Unfortunately I cannot publish the raw Fortran code, nor create an MCVE which replicates the problem and is sufficiently abstracted such that I could share it. All of the variables are either declared as integer or real(8) types in the Fortran code.
import ctypes
import os
DLL_PATH = "C:\Repos\CASS_PFM\dlls"
class wrapper:
def __init__(self,data):
self.data = data
self.DLL = ctypes.CDLL(os.path.join(DLL_PATH,"MyDLL.dll"))
self.fortran_subroutine = getattr(self.DLL,"MyFunction_".lower())
self.output = {}
def run(self):
out = (ctypes.c_longdouble * len(self.data))()
in_data = []
for item in self.data:
item.convert_to_ctypes()
in_data.append(ctypes.byref(item.c_val))
self.fortran_subroutine(*in_data, out)
for item in self.data:
self.output[item.name] = item.convert_to_python()
class FortranData:
def __init__(self,name,py_val,ctype,some_param=True):
self.name = name
self.py_val = py_val
self.ctype = ctype
self.some_param = some_param
def convert_to_ctypes(self):
ctype_converter = getattr(ctypes,self.ctype)
self.c_val = ctype_converter(self.py_val)
return self.c_val
def convert_to_python(self):
self.py_val = self.c_val.value
return self.py_val
def main():
R_o = 350
data = [
FortranData("R_o",R_o,'c_double',False),
FortranData("thick",57.15,'c_double',False),
FortranData("axial_c",100,'c_double',False),
FortranData("sigy",235.81,'c_double',False),
FortranData("sigu",619.17,'c_double',False),
FortranData("RO_alpha",1.49707,'c_double',False),
FortranData("RO_sigo",235.81,'c_double',False),
FortranData("RO_epso",0.001336,'c_double',False),
FortranData("RO_n",6.6,'c_double',False),
FortranData("Resist_Jic",116,'c_double',False),
FortranData("Resist_C",104.02,'c_double',False),
FortranData("Resist_m",0.28,'c_double',False),
FortranData("pressure",15.51375,'c_double',False),
FortranData("i_write",0,'c_int',False),
FortranData("if_flag_twc",0,'c_int',),
FortranData("i_twc_ll",0,'c_int',),
FortranData("i_twc_epfm",0,'c_int',),
FortranData("i_err_code",0,'c_int',),
FortranData("Axial_TWC_ratio",0,'c_double',),
FortranData("Axial_TWC_fail",0,'c_int',),
FortranData("c_max_ll",0,'c_double',),
FortranData("c_max_epfm",0,'c_double',)
]
obj = wrapper(data)
obj.run()
print(obj.output)
if __name__ == "__main__": main()
It's not just the R_o value either; there are some combinations of values that cause the same error (seemingly without rhyme or reason). Is there anything within the above Python that might lead to an access violation depending on the values passed to the DLL?
Python version is 3.7.2, 32-bit
I see 2 problems with the code (and a potential 3rd one):
argtypes (and restype) not being specified. Check [SO]: C function called from Python via ctypes returns incorrect value (#CristiFati's answer) for more details
This may be a consequence (or at least it's closely related to) the previous. I can only guess without the Fortran (or better: C) function prototype, but anyway there is certainly something wrong. I assume that for the input data things should be same as for the output data, so the function would take 2 arrays (same size), and the input one's elements would be void *s (since their type is not consistent). Then, you'd need something like (although I can't imagine how would Fortran know which element contains an int and which a double):
in_data (ctypes.c_void_p * len(self.data))()
for idx, item in enumerate(self.data):
item.convert_to_ctypes()
in_data[index] = ctypes.addressof(item.c_val)
Since you're on 032bit, you should also take calling convention into account (ctypes.CDLL vs ctypes.WinDLL)
But again, without the function prototype, everything is just a speculation.
Also, why "MyFunction_".lower() instead of "myfunction_"?

How do I set an instance attribute in Python when the instance is determined by a function?

I would like to iterate through a selection of class instances and set a member variable equal to a value. I can access the members value with:
for foo in range(1,4): #class members: pv1, pv2, pv3
bar[foo] ='{0}'.format(locals()['pv' + str(foo)+'.data'])
However when I try to set/mutate the values like so:
for foo in range(1,4): #class members:
'{0}'.format(locals()['pv' + str(foo)+'.data']) = bar[foo]
I obviously get the error:
SyntaxError: can't assign to function call
I have tried a few methods to get it done with no success. I am using many more instances than 3 in my actual code(about 250), but my question is hopefully clear. I have looked at several stack overflow questions, such as Automatically setting class member variables in Python -and- dynamically set an instance property / memoized attribute in python? Yet none seem to answer this question. In C++ I would just use a pointer as an intermediary. What's the Pythonic way to do this?
An attr is a valid assignment target, even if it's an attr of the result of an expression.
for foo in range(1,3):
locals()['pv' + str(foo)].data = bar[foo]
Another developer wrote a few lines about setattr(), mostly about how it should be avoided.
setattr is unnecessary unless the attribute name is dynamic.
But they didn't say why. Do you mind elaborating why you switched your answer away from setattr()?
In this case, the attr is data, which never changes, so while
for i in range(1, 3):
setattr(locals()['pv' + str(i)], 'data', bar[i])
does the same thing, setattr isn't required here. The .data = form is both good enough and typically preferred--it's faster and has clearer intent--which is why I changed it. On the other hand, if you needed to change the attr name every loop, you'd need it, e.g.
for i in range(1,3):
setattr(locals()['pv' + str(i)], 'data' + str(i), bar[i])
The above code sets attrs named data1, data2, data3, unrolled, it's equivalent to
pv1.data1 = bar[1]
pv2.data2 = bar[2]
pv3.data3 = bar[3]
I originally thought your question needed to do something like this, which is why I used setattr in the first place. Once I tested it and got it working I just posted it without noticing that the setattr was no longer required.
If the attr name changes at runtime like that (what the other developer meant by "dynamic") then you can't use the dot syntax, since you have a string object rather than a static identifier. Another reason to use setattr might be if you need a side effect in an expression. Unlike in C, assignments are statements in Python. But function calls like setattr are expressions.
Here is an example of creating a class which explicitly allows access through index or attribute calls to change internal variables. This is not generally promoted as 'good programming' though. It does not explicitly define the rules by which people should be expected to interact with the underlying variables.
the definition of __getattr__() function allows for the assignment of (object).a .
the definition of __getitem__() function allows for the assignment of
(object)['b']
class Foo(object):
def __init__(self, a=None,b=None,c=None):
self.a=a
self.b=b
self.c=c
def __getattr__(self, x):
return self.__dict__.get(x, None)
def __getitem__(self, x):
return self.__dict__[x]
print
f1 = Foo(3,2,4)
print 'f1=', f1.a, f1['b'], f1['c']
f2 = Foo(4,6,2)
print 'f2=', f2.a, f2['b'], f2['c']
f3 = Foo(3,5,7)
print 'f3=', f3.a, f3['b'], f3['c']
for x in range(1, 4):
print 'now setting f'+str(x)
locals()['f'+str(x)].a=1
locals()['f'+str(x)].b=1
locals()['f'+str(x)].c=1
print
print 'f1=', f1.a, f1['b'], f1['c']
print 'f2=', f2.a, f2['b'], f2['c']
print 'f3=', f3.a, f3['b'], f3['c']
The result is
f1= 3 2 4
f2= 4 6 2
f3= 3 5 7
now setting f1
now setting f2
now setting f3
f1= 1 1 1
f2= 1 1 1
f3= 1 1 1

Array of Structs in Python

I cannot use multiprocessing, I need shared memory among entirely separate python processes on Windows using python 3. I've figured out how to do this using mmap, and it works great...when I use simple primitive types. However, I need to pass around more complex information. I've found the ctypes.Structure and it seems to be exactly what I need.
I want to create an array of ctypes.Structure and update an individual element within that array, write it back to memory as well as read an individual element.
import ctypes
import mmap
class Person(ctypes.Structure):
_fields_ = [
('name', ctypes.c_wchar * 10),
('age', ctypes.c_int)
]
if __name__ == '__main__':
num_people = 5
person = Person()
people = Person * num_people
mm_file = mmap.mmap(-1, ctypes.sizeof(people), access=mmap.ACCESS_WRITE, tagname="shmem")
Your people is not an array yet, it's still a class. In order to have your array, you need to initialize the class using from_buffer(), just like you were doing before with c_int:
PeopleArray = Person * num_people
mm_file = mmap.mmap(-1, ctypes.sizeof(PeopleArray), ...)
people = PeopleArray.from_buffer(mm_file)
people[0].name = 'foo'
people[0].age = 27
people[1].name = 'bar'
people[1].age = 42
...

The unbearable opaqueness of time.struct_time

Why do pylint and the intellisense features of IDEs have trouble recognizing instances of time.struct_time? The following code contains some trivial tests of existent/non-existent attributes of classes, named tuples and the named-tuple-like time.struct_time. Everything works as expected in pylint, IntelliJ and VSCode - the access to missing attributes is reported in each case except for time.struct_time - it generates no warnings or errors in any of these tools. Why can't they tell what it is and what its attributes are?
import time
from collections import namedtuple
t = time.localtime()
e = t.tm_mday
e = t.bad # this is not reported by linters or IDEs.
class Clz:
cvar = 'whee'
def __init__(self):
self.ivar = 'whaa'
o = Clz()
e = Clz.cvar
e = o.ivar
e = Clz.bad
e = o.bad
Ntup = namedtuple('Ntup', 'thing')
n = Ntup(thing=3)
e = n.thing
e = n.bad
The context of the question is the following recent bug in pipenv -
# Halloween easter-egg.
if ((now.tm_mon == 10) and (now.tm_day == 30))
Obviously, the pass path was never tested but it seems the typical static analysis tools would not have helped here either. This is odd for a type from the standard library.
(Fix can be seen in full at https://github.com/kennethreitz/pipenv/commit/033b969d094ba2d80f8ae217c8c604bc40160b03)
time.struct_time is an object defined in C, which means it can't be introspected statically. The autocompletion software can parse Python code and make a reasonable guess as to what classes and namedtuples support, but they can't do this for C-defined objects.
The work-around most systems use is to generate stub files; usually by introspecting the object at runtime (importing the module and recording the attributes found). For example, CodeIntel (part of the Komodo IDE), uses an XML file format called CIX. However, this is a little more error-prone so such systems then err on the side of caution, and will not explicitly mark unknown attributes as wrong.
If you are coding in Python 3, you could look into using type hinting. For C extensions you still need stub files, but the community is pretty good at maintaining these now. The standard library stub files are maintained in a project called typeshed.
You'd have to add type hints to your project:
#!/usr/bin/env python3
import time
from collections import namedtuple
t: time.struct_time = time.localtime()
e: int = t.tm_mday
e = t.bad # this is not reported by linters or IDEs.
class Clz:
cvar: str = 'whee'
ivar: str
def __init__(self) -> None:
self.ivar = 'whaa'
o = Clz()
s = Clz.cvar
s = o.ivar
s = Clz.bad
s = o.bad
Ntup = namedtuple('Ntup', 'thing')
n = Ntup(thing=3)
e = n.thing
e = n.bad
but then the flake8 tool combined with the flake8-mypy plugin will detect the bad attributes:
$ flake8 test.py
test.py:8:5: T484 "struct_time" has no attribute "bad"
test.py:22:5: T484 "Clz" has no attribute "bad"
test.py:23:5: T484 "Clz" has no attribute "bad"
test.py:28:5: T484 "Ntup" has no attribute "bad"
PyCharm builds on this work too, and perhaps can detect the same invalid use. It certainly directly supports pyi files.

Another Name error for my text adventure game Python

So i am making a text adventure game, and currently i am making the enemies. My class random_enemies makes trash mobs for your character to fight and i have a function in it called weak, normal, strong, etc... that scales with your character depending on which one it is. When i call random_enemies.weak it says (Name Error: global variable "p" is not defined) even though it should be.
import random
from character import *
from player import *
class random_enemies(character):
def __init__(self,name,hp,maxhp,attack_damage,ability_power,exp):
super(random_enemies,self).__init__(name,hp,maxhp)
self.attack_damage = attack_damage
self.ability_power = ability_power
self.exp = exp
def weak():
self.hp = random.randint(p.maxhp/10, p.maxhp/5)
self.attack_damage = None
self.ability_power = None
self.exp = None
from character import*
class player(character):
def __init__(self,name,hp,maxhp,attack_damage,ability_power):
super(player,self).__init__(name, hp, maxhp)
self.attack_damage = attack_damage
self.ability_power = ability_power
This is my player class and below is the class that player gets "maxhp" from.
class character(object):
def __init__(self,name,hp,maxhp):
self.name = name
self.hp = hp
self.maxhp = maxhp
def attack(self,other):
pass
p=player(Players_name, 100, 100, 10, 5,)
while (p.hp>0):
a=input("What do you want to do?")
if a=="Instructions":
Instructions()
elif a=="Commands":
Commands()
elif a=="Fight":
print("Level",wave,"Wave Begins")
if wave < 6:
b = random_enemies.weak()
print("A",b,"Appeared!")
print("Stats of",b, ": \n Health=", b.hp,"Attack Damage=",b.attack_damage)
continue
I just made this really quickly just to test if everything I had was working until I got the error. This is also the place where random_enemies.weak() was called. Also in this is where I defined what "p" was.
So, first of all, follow a naming convention. For python code I recommend that you use pep8 as a convention.
You have a problem with classes vs. instances in your code. First, you need an instance of a class before you can use it:
enemy = random_enemy() # a better name would be RandomEnemy
In Python, all methods start with self, and you need to pass to the method the arguments that it needs to do its work. weak is a method, so it should be more like this:
def weak(self, player):
# the method for weak ... weak attack ?
# remember to change p to player, which is more meaningful
...
Now that you have your instance and it has a method weak which receives a player as argument, you can use it as follows:
# you can't use random_enemy here as you tried because it is a class
# you need a random_enemy instance, the enemy that I talked about earlier
b = enemy.weak(player) # renamed p to player because it is more meaningful
For this all to work, you will need one more thing. weak() needs to return something. Right now you are using what it returns, nothing! The code that you posted is b = random_enemies.weak(). Because weak() does not have a return clause, b will always be None.
Some notes: Avoid one-letter variables unless there is a long standing convention (like using i for loop counter). It is easier to understand what you are trying to do if you define player instead of just p.
Python has a really great tutorial for all this stuff.

Resources