I am trying to use Collections-C in Cython.
I noticed that some structures are defined in the .c file, and an alias for them is in the .h file. When I try to define those structures in a .pxd file and use them in a .pyx file, gcc throws an error: storage size of ‘[...]’ isn’t known.
I was able to reproduce my issue to a minimum setup that replicates the external library and my application:
testdef.c
/* Note: I can't change this */
struct bogus_s {
int x;
int y;
};
testdef.h
/* Note: I can't change this */
typedef struct bogus_s Bogus;
cytestdef.pxd
# This is my code
cdef extern from 'testdef.h':
struct bogus_s:
int x
int y
ctypedef bogus_s Bogus
cytestdef.pyx
# This is my code
def fn():
cdef Bogus n
n.x = 12
n.y = 23
print(n.x)
If I run cythonize, I get
In function ‘__pyx_pf_7sandbox_9cytestdef_fn’:
cytestdef.c:1106:9: error: storage size of ‘__pyx_v_n’ isn’t known
Bogus __pyx_v_n;
^~~~~~~~~
I also get the same error if I use ctypedef Bogus: [...] notation as indicated in the Cython manual.
What am I doing wrong?
Thanks.
Looking at the documentation for your Collections-C library these are opaque structures that you're supposed to use purely through pointers (don't need to know the size to have a pointer, while you do to allocate on the stack). Allocation of these structures is done in library functions.
To change your example to match this case:
// C file
int bogus_s_new(struct bogus_s** v) {
*v = malloc(sizeof(struct bogus_s));
return (v!=NULL);
}
void free_bogus_s(struct bogus_s* v) {
free(v);
}
Your H file would contain the declarations for those and your pxd file would contain wrappers for the declarations. Then in Cython:
def fn():
cdef Bogus* n
if not bogus_s_new(&n):
return
try:
# you CANNOT access x and y since the type is
# designed to be opaque. Instead you should use
# the acessor functions defined in the header
# n.x = 12
# n.y = 23
finally:
free_bogus_s(n)
Related
Given a ctypes-pointer, for example double**:
import ctypes
data=(ctypes.POINTER(ctypes.c_double)*4)() # results in [NULL, NULL, NULL, NULL]
is it possible to obtain a format string, which describes the memory layout of the data?
Right now, I create a memoryview to get this information, which feels somewhat silly:
view=memoryview(data)
print(view.format) # prints: &<d
Is there a more direct way with less overhead? Maybe through using the C-API?
One could fill data with meaningful values, if this is of any help:
import ctypes
data=(ctypes.POINTER(ctypes.c_double)*2)(
(ctypes.c_double*2)(1.0,2.0),
(ctypes.c_double*1)(3.0))
# results in [
# ptr0 -> [1,2],
# ptr1 -> [3]
# ]
print(data[1][0]) # prints 3.0
It seems as if there is nothing fundamentally better than memoryview(data).format. However, one could speed-up this a little bit by using C-API.
The format-string (which extends the struct format-string-syntax as described in PEP3118) is calculated recursively and is stored in the format-member of the StgDictObject-object, which can be found in the tp_dict-field of
ctypes-arrays/pointers:
typedef struct {
PyDictObject dict; /* first part identical to PyDictObject */
...
/* pep3118 fields, pointers neeed PyMem_Free */
char *format;
int ndim;
Py_ssize_t *shape;
...
} StgDictObject;
This format-field is accessed only during the recursive calculation and when a buffer is exported - that is how memoryview gets this info:
static int PyCData_NewGetBuffer(PyObject *myself, Py_buffer *view, int flags)
{
...
/* use default format character if not set */
view->format = dict->format ? dict->format : "B";
...
return 0;
}
Now we could use C-API to populate a buffer (without creating the actual memoryview), here implemented in Python:
%%cython
from cpython cimport buffer
def get_format_via_buffer(obj):
cdef buffer.Py_buffer view
buffer.PyObject_GetBuffer(obj, &view, buffer.PyBUF_FORMAT|buffer.PyBUF_ANY_CONTIGUOUS)
cdef bytes format = view.format
buffer.PyBuffer_Release(&view)
return format
This version is about 3 times faster than via memoryview:
import ctypes
c=(ctypes.c_int*3)()
%timeit get_format_via_buffer(c) # 295 ns ± 10.3
%timeit memoryview(c).format # 936 ns ± 7.43 ns
On my machine, there are about 160 ns overhead of calling a def-function and about 50 ms for creating a bytes-object.
Even if it doesn't make much sense to optimize it further due to the unavoidable overhead, there is still at least theoretical interest of how it could be speed up.
If one really wants to shave off also the cost of filling out the Py_buffer-struct, than there is no clean way: ctypes-module isn't part of Python-C-API (it is not in the include-directory), so the way forward is to repeat the solution Cython uses with the array.array, i.e. hardcoding the memory layout of the object (which makes this solution brittle because the memory-layout of StgDictObject can get out of sync).
Here with Cython and without error-checking:
%%cython -a
from cpython cimport PyObject
# emulate memory-layout (i.e. copy definitions from ctypes.h)
cdef extern from *:
"""
#include <Python.h>
typedef struct _ffi_type
{
size_t size;
unsigned short mem[2];
struct _ffi_type **elements;
} ffi_type;
typedef struct {
PyDictObject dict; /* first part identical to PyDictObject */
Py_ssize_t size[3]; /* number of bytes,alignment requirements,number of fields */
ffi_type ffi_type_pointer;
PyObject *proto; /* Only for Pointer/ArrayObject */
void *setfunc[3];
/* Following fields only used by PyCFuncPtrType_Type instances */
PyObject *argtypes[4];
int flags; /* calling convention and such */
/* pep3118 fields, pointers neeed PyMem_Free */
char *format;
int ndim;
} StgDictObject;
"""
ctypedef struct StgDictObject:
char *format
def get_format_via_hack(obj):
cdef PyObject *p =<PyObject *>obj
cdef StgDictObject *dict = <StgDictObject *>(p.ob_type.tp_dict)
return dict.format
And it is as fast as it gets:
%timeit get_format_via_hack(c) # 243 ns ± 14.5 ns
in f1.h header using typedef for structure. sample code snippet shown below
typedef struct{
int a;
union u
{
int x;
char y;
}xyz;
}mystruct;
In f2.h header using the structure mysturct to get the offset. Code snippet shown below
static mystruct ktt
//#define OFFSET_T(b, c) ((int*)((&((mystruct*)0)->b)) - (int*)((&((mystruct*)0)->c)))
#define OFFSET_T(b, c) ((char*) &ktt.b - (char *) &ktt.c)
static struct Mystruct1{
int n;
}mystruct1 = {OFFSET_T(xyz,a)};
when i'm doing compilation in AIX machine using xlc compiler it is throwing the error as "1506-221(S) Initializer must be a valid constant expression".
i tried both the macro's but both are getting same error. Is there anything wrong in f2.h macro while performing size of structure to get offset ??
The expression in question needs to be an arithmetic constant expression in order to be portable. Neither macro qualifies, since operands of pointer type are involved and arithmetic constant expressions are restricted such that those operands are not allowed. In C11, this is found in subclause 6.6 paragraph 8.
That said, the code using the first macro (source reproduced below) does compile on multiple versions of the xlc compiler on AIX.
typedef struct{
int a;
union u
{
int x;
char y;
}xyz;
}mystruct;
static mystruct ktt;
#define OFFSET_T(b, c) ((int*)((&((mystruct*)0)->b)) - (int*)((&((mystruct*)0)->c)))
//#define OFFSET_T(b, c) ((char*) &ktt.b - (char *) &ktt.c)
static struct Mystruct1{
int n;
}mystruct1 = {OFFSET_T(xyz,a)};
The compiler invocation I used was:
xlc offsetcalc.c -c -o /dev/null
The version information for one of the older versions I tried is:
IBM XL C/C++ for AIX, V10.1
Version: 10.01.0000.0021
The version information for one of the newest versions I tried is:
IBM XL C/C++ for AIX, V13.1.3 (5725-C72, 5765-J07)
Version: 13.01.0003.0004
I am searching for a memory leak in code of someone else. I found:
def current(self):
...
data = PyBuffer_New(buflen)
PyObject_AsCharBuffer(data, &data_ptr, &buflen)
...
return VideoFrame(data, self.frame_size, self.frame_mode,
timestamp=<double>self.frame.pts/<double>AV_TIME_BASE,
frameno=self.frame.display_picture_number)
cdef class VideoFrame:
def __init__(self, data, size, mode, timestamp=0, frameno=0):
self.data = data
...
In function current() is no free or similar, neither in VideoFrame. Is the PyBuffer automatically freed when the VideoFrame object gets deleted?
The answer is: "it depends; we don't have enough code to tell from your question." It's governed by what type you've told Cython that PyBuffer_New returns. I'll give two simplified illustrating cases and hopefully you should be able to work it out for your more complicated case.
If you tell Cython that it's a PyObject* it has no innate knowledge of that type, and doesn't do anything to keep track of the memory:
# BAD - memory leak!
cdef extern from "Python.h":
ctypedef struct PyObject
PyObject* PyBuffer_New(int size)
def test():
cdef int i
for i in range(100000): # call lots of times to allocate lots of memory
# (type of a is automatically inferred to be PyObject*
# to match the function definition)
a = PyBuffer_New(1000)
and the generated code for the loop pretty much looks like:
for (__pyx_t_1 = 0; __pyx_t_1 < 1000; __pyx_t_1+=1) {
__pyx_v_i = __pyx_t_1;
__pyx_v_a = PyBuffer_New(1000);
}
i.e. memory is being allocated but never freed. If you run test() and look at a task manager you can see the memory usage jump up and not return.
Alternatively, if you tell Cython it's an object that lets Cython deal with it like any other Python object, and manage the reference count correctly:
# Good - no memory leak
cdef extern from "Python.h":
object PyBuffer_New(int size)
def test():
cdef int i
for i in range(100000):# call lots of times to allocate lots of memory
a = PyBuffer_New(1000)
The generated code for the loop is then
for (__pyx_t_1 = 0; __pyx_t_1 < 100000; __pyx_t_1+=1) {
__pyx_v_i = __pyx_t_1;
__pyx_t_2 = PyBuffer_New(1000); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 7; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
__Pyx_GOTREF(__pyx_t_2);
__Pyx_XDECREF_SET(__pyx_v_a, __pyx_t_2);
__pyx_t_2 = 0;
}
Note the DECREF, which will be where the object is deallocated. If you run test() here you see no long-term jump in memory usage.
It may be possible to jump between these two cases by using cdef for variable (for example in the definition of VideoFrame). If they use PyObject* without careful DECREFs then they're probably leaking memory...
I came across a problem recently.
I have three files, A.h, B.cpp, C.cpp:
A.h
#ifndef __A_H__
#define __A_H__
int M()
{
return 1;
}
#endif // __A_H__
B.cpp
#include "A.h"
C.cpp
#include "A.h"
As I comile the three files by MSVC, there is a error:
C.obj : error LNK2005: "int __cdecl M(void)" (?M##YAHXZ) already defined in B.obj
It is easy understanding, as we know, B.obj has a symbol named "M", also C.obj has a "M".
Here the error comes.
However, if I change M method to a class which contain a method M like this below:
A.h
#ifndef __A_H__
#define __A_H__
class CA
{
public:
int M()
{
return 1;
}
};
#endif // __A_H__
there is no more errors!! Could somebody tell me what is happening?
If B.cpp and C.cpp include A.h, then both are compiled with your definition of M, so both object files will contain code for M. When the linker gathers all the functions, he sees that M is defined in two object files and does not know which one to use. Thus the linker raises an LNK2005.
If you put your function M into a class declaration, then the compiler marks/handles M as an inline function. This information is written into the object file. The linker sees that both object files contain a definition for an inline version of CA::M, so he assumes that both are equal and picks up randomly one of the two definitions.
If you had written
class CA {
public:
int M();
};
int CA::M()
{
return 1;
}
this would have caused the same problems (LNK2005) as your initial version, because then CA::M would not have been inline any more.
So as you might guess by now, there are two solutions for you. If you want M to be inlined, then change your code to
__inline int M()
{
return 1;
}
If you don't care about inlining, then please do it the standard way and put the function declaration into the header file:
extern int M();
And put the function definition into a cpp file (for A.h this would ideally be A.cpp):
int M()
{
return 1;
}
Please note that the extern is not really necessary in the header file.
Another user suggested that you write
static int M()
{
return 1;
}
I'd not recommend this. This would mean that the compiler puts M into both of your object files and marks M as being a function that is only visible in each object file itself. If the linker sees that a function in B.cpp calls M, it finds M in B.obj and in C.obj. Both have M marked as static, so the linker ignores M in C.obj and picks the M from B.obj. Vice versa if a function in C.cpp calls M, the linker picks the M from C.obj. You will end up with multiple definitions of M, all with the same implementation. This is a waste of space.
See http://faculty.cs.niu.edu/~mcmahon/CS241/c241man/node90.html how to do ifdef guards. You have to start with ifndef before the define.
Edit: Ah no, while your guard is wrong that's not the issue. Put static in front of your function to make it work. Classes are different because they define types.
I don't know what's under the hood, but if you don't need a class I guess that the compiler will automatically add the "extern" key to your functions, so you'll get the error including the header 2 times.
You can add the static keyword to M() method so you'll have only one copy of that function in memory and no errors at compile time.
By the way: I see you have a #endif, but not a #ifdef or #ifndef, is it a copy/paste error?
In my data.h file I have:
typedef struct {
double ***grid;
} Solver;
In my .c file I have
static Solver _solver;
which first makes a call to a function to do some allocation on grid such as
_solver.grid = malloc(....);
//then makes a call to
GS_init(_solver.grid);
The GS_init function is declared in GS.h as:
void GS_init(double ***grid);
When I try to compile, I get two errors:
the struct "<unnamed>" has no field "grid"
GS_init(_solver.grid)
^
and
too many arguments in function call
GS_init(_solver.grid)
^
Any ideas what is going wrong here?
This code compiles with 'gcc -Wall -Werror -c':
data.h
typedef struct
{
double ***grid;
} Solver;
gs.h
extern void GS_init(double ***grid);
gs.c
#include "data.h"
#include "gs.h"
#include <stdlib.h>
static Solver _solver;
void anonymous(void)
{
_solver.grid = malloc(32 * sizeof(double));
GS_init(_solver.grid);
}
Derek asked:
Why does this work? Is it because of the extern keyword?
The 'extern' is not material to making it work, though I always use it.
When I have to flesh out GS_init() in, say compute.c, would I write void GS_init(double ***grid){ //loop over grid[i][j][k] setting to zero }
Sort of...yes, the GS_init() code could do that if the data structure is set up properly, which is going to need more information than there is currently visible in the structure.
For the compiler to process:
grid[i][j][k] = 0.0;
the code has to know the valid ranges for each of i, j, and k; assume the number of rows in each dimension are Ni, Nj, Nk. The data 'structure' pointed to by grid must be an array of Ni 'double **' values - which must be allocated. Each of those entries must point to Nj 'double *' values. So, you have to do more allocation than a single malloc(), and you have to do more initialization than just setting everything to zero.
If you want to use a single array of doubles only, you will have to write a different expression to access the data:
grid[(i * Ni + j) * Nj + k] = 0.0;
And under this scenario, grid would be a simple double * and not a triple pointer.