I'm wrapping a C library to Pyhon 3 using Cython and i'm looking for a way of converting wchar_t string to python object which i want to return from a function. There's an answer in this question, but it involves encoding the string as multibyte str, and decoding it back to unicode. I hope for more straightforward solution. I tried using PyUnicode_FromWideChar from Python C API, but i'm getting a segfault. Here's my .pyx code:
from cpython.ref cimport PyObject
from libc.stddef cimport wchar_t
cdef extern from "Python.h":
PyObject* PyUnicode_FromWideChar(wchar_t *w, Py_ssize_t size)
cdef extern from "../../src/my_c_lib.h":
wchar_t * myObjToStr(wchar_t * result, size_t size, myObj * obj)
cdef class MyClass:
cdef myObj * ptr
...
def to_str(self):
cdef wchar_t buf[64]
cdef wchar_t * result = myObjToStr(buf, 64, self.ptr)
if result is NULL:
raise Exception("Error converting object to string.")
cdef PyObject * pystr = PyUnicode_FromWideChar(result, 64)
return <object>pystr
result is actually the pointer to buf. What's wrong with this? Is there another way without encoding/decoding?
Edit: I found that PyUnicode_FromWideChar() returns NULL, but why? I checked, that result is a valid wchar_t * string.
Using -1 as second argument to PyUnicode_FromWideChar(result, -1)(so that the function uses wcslen internally), fixed the problem.
So PyUnicode_FromWideChar really is the way to go.
Related
I am trying to make sense of the ctypes/callback documentation:
https://docs.python.org/3/library/ctypes.html#callback-functions
My c code defines a LogFunction as:
typedef void (*LogFunction)(int logLevel, const char* loggerName, const char* logMessage);
So my naive attempt (following the documentation) has been to expose the API using:
LOGFUNC = CFUNCTYPE(None, c_int, c_char_p, c_char_p)
acme_listener_configure = _lib.acme_listener_configure
acme_listener_configure.restype = None
acme_listener_configure.argtypes = [LOGFUNC]
However when using it (python3):
def py_log_func(a, b, c):
print("py_log_func", a, b, c)
return
acme_listener_configure(LOGFUNC(py_log_func))
Here is what I see (Debian):
py_log_func 5 b'acme.logger.name' b'invalid input'
Is there a way to declare LOGFUNC so that conversion from bytes into proper utf-8 strings is done correctly (I know the library passes utf-8 buffer in this function).
bytes are how UTF-8 strings are represented. It sounds like you want to decode the received UTF-8 byte strings to Unicode strings. If so, then .decode() defaults to using 'utf8' to decode strings:
from ctypes import *
LOGFUNC = CFUNCTYPE(None, c_int, c_char_p, c_char_p)
acme_listener_configure = _lib.acme_listener_configure
acme_listener_configure.argtypes = LOGFUNC,
acme_listener_configure.restype = None
#LOGFUNC
def py_log_func(a, b, c):
print("py_log_func", a.decode(), b.decode(), c.decode())
acme_listener_configure(py_log_func)
Note that using LOGFUNC as a decorator of the callback can prevent some issues if the callback is continued to be used beyond the .configure function; otherwise, the LOGFUNC(py_log_func) object is destroyed after the function call returns.
I'm using Python3.6 with GStreamer-1.0 and PyGObject (for python access) to read video frames from a camera (tiscamera).
The frames are gotten via python code and eventually I get a GstBuffer:
import gi
gi.require_version("Gst", "1.0")
from gi.repository import
# Set up
Gst.init([])
pipeline = Gst.parse_launch("tcambin serial=12345678 name=source ! video/x-raw,format=GRAY8,width=1920,height=1080,framerate=18/1 ! appsink name=sink")
sink = pipeline.get_by_name("sink")
sink.set_property("max-buffers", 10)
sink.set_property("drop", 1)
sink.set_property("emit-signals", 1)
pipeline.set_state(Gst.State.PLAYING)
# Get sample
sample = sink.emit("pull-sample")
buffer = sample.get_buffer()
meta = buffer.get_meta("TcamStatisticsMetaApi")
The type of meta is gi.repository.Gst.Meta, but in C it's actually TcamStatisticsMeta*, as can be understood by looking at tiscamera's c code example 10-metadata.c. The C code there is:
GstBuffer* buffer = gst_sample_get_buffer(sample);
GstMeta* meta = gst_buffer_get_meta(buffer, g_type_from_name("TcamStatisticsMetaApi"));
GstStructure* struc = ((TcamStatisticsMeta*)meta)->structure;
My problem is that in Python, I can't access struct attributes defined in TcamStatisticsMeta. I'm simply missing the casting bit from GstMeta* to TcamStatisticsMeta* and the translation of TcamStatisticsMeta* into a PyObject.
Does anyone have a direction of how this can be done without needing to modify/recompile the gstreamer-1.0 C code? Using Cython perhaps?
I've started using Cython to try and call a C function with data I get from Python. The python object is of type gi.repository.Gst.Buffer and the function should get a GstBuffer*, but I can't find a way to get the struct pointer from the Python object.
Here is my .pxd file:
cdef extern from "gstreamer-1.0/gstmetatcamstatistics.h":
ctypedef unsigned long GType
ctypedef struct GstBuffer:
pass
ctypedef struct GstMeta:
pass
GstMeta* gst_buffer_get_meta(GstBuffer* buffer, GType api)
GType g_type_from_name(const char* name)
And my .pyx file:
from my_pxd_file cimport GType, g_type_from_name, GstMeta, gst_buffer_get_meta
cdef void c_a(buffer):
cdef char* tcam_statistics_meta_api = "TcamStatisticsMetaApi"
cdef GType gt = g_type_from_name(tcam_statistics_meta_api)
cdef GstMeta* meta = gst_buffer_get_meta(buffer, gt)
def a(buffer):
c_a(buffer)
And my python file:
# No need to use pyximport as I've cythonized the code in setup.py
from . import my_pyx_file
...
buffer = sample.get_buffer()
my_pyx_file.a(buffer)
This results in a SIGSEGV error:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
The problem is that I can't cast buffer into a GstBuffer*. Does anyone know how to do that?
Debugging in Pycharm, I can actually see the GstBuffer* address:
<Gst.Buffer object at 0x7f3a0b4fc348 (GstBuffer at 0x7f39ec007060)>
But how do I get this address so I can pass it to gst_buffer_get_meta?
Is there a canonical Cython way to do that?
I am trying to use Collections-C in Cython.
I noticed that some structures are defined in the .c file, and an alias for them is in the .h file. When I try to define those structures in a .pxd file and use them in a .pyx file, gcc throws an error: storage size of ‘[...]’ isn’t known.
I was able to reproduce my issue to a minimum setup that replicates the external library and my application:
testdef.c
/* Note: I can't change this */
struct bogus_s {
int x;
int y;
};
testdef.h
/* Note: I can't change this */
typedef struct bogus_s Bogus;
cytestdef.pxd
# This is my code
cdef extern from 'testdef.h':
struct bogus_s:
int x
int y
ctypedef bogus_s Bogus
cytestdef.pyx
# This is my code
def fn():
cdef Bogus n
n.x = 12
n.y = 23
print(n.x)
If I run cythonize, I get
In function ‘__pyx_pf_7sandbox_9cytestdef_fn’:
cytestdef.c:1106:9: error: storage size of ‘__pyx_v_n’ isn’t known
Bogus __pyx_v_n;
^~~~~~~~~
I also get the same error if I use ctypedef Bogus: [...] notation as indicated in the Cython manual.
What am I doing wrong?
Thanks.
Looking at the documentation for your Collections-C library these are opaque structures that you're supposed to use purely through pointers (don't need to know the size to have a pointer, while you do to allocate on the stack). Allocation of these structures is done in library functions.
To change your example to match this case:
// C file
int bogus_s_new(struct bogus_s** v) {
*v = malloc(sizeof(struct bogus_s));
return (v!=NULL);
}
void free_bogus_s(struct bogus_s* v) {
free(v);
}
Your H file would contain the declarations for those and your pxd file would contain wrappers for the declarations. Then in Cython:
def fn():
cdef Bogus* n
if not bogus_s_new(&n):
return
try:
# you CANNOT access x and y since the type is
# designed to be opaque. Instead you should use
# the acessor functions defined in the header
# n.x = 12
# n.y = 23
finally:
free_bogus_s(n)
Given a ctypes-pointer, for example double**:
import ctypes
data=(ctypes.POINTER(ctypes.c_double)*4)() # results in [NULL, NULL, NULL, NULL]
is it possible to obtain a format string, which describes the memory layout of the data?
Right now, I create a memoryview to get this information, which feels somewhat silly:
view=memoryview(data)
print(view.format) # prints: &<d
Is there a more direct way with less overhead? Maybe through using the C-API?
One could fill data with meaningful values, if this is of any help:
import ctypes
data=(ctypes.POINTER(ctypes.c_double)*2)(
(ctypes.c_double*2)(1.0,2.0),
(ctypes.c_double*1)(3.0))
# results in [
# ptr0 -> [1,2],
# ptr1 -> [3]
# ]
print(data[1][0]) # prints 3.0
It seems as if there is nothing fundamentally better than memoryview(data).format. However, one could speed-up this a little bit by using C-API.
The format-string (which extends the struct format-string-syntax as described in PEP3118) is calculated recursively and is stored in the format-member of the StgDictObject-object, which can be found in the tp_dict-field of
ctypes-arrays/pointers:
typedef struct {
PyDictObject dict; /* first part identical to PyDictObject */
...
/* pep3118 fields, pointers neeed PyMem_Free */
char *format;
int ndim;
Py_ssize_t *shape;
...
} StgDictObject;
This format-field is accessed only during the recursive calculation and when a buffer is exported - that is how memoryview gets this info:
static int PyCData_NewGetBuffer(PyObject *myself, Py_buffer *view, int flags)
{
...
/* use default format character if not set */
view->format = dict->format ? dict->format : "B";
...
return 0;
}
Now we could use C-API to populate a buffer (without creating the actual memoryview), here implemented in Python:
%%cython
from cpython cimport buffer
def get_format_via_buffer(obj):
cdef buffer.Py_buffer view
buffer.PyObject_GetBuffer(obj, &view, buffer.PyBUF_FORMAT|buffer.PyBUF_ANY_CONTIGUOUS)
cdef bytes format = view.format
buffer.PyBuffer_Release(&view)
return format
This version is about 3 times faster than via memoryview:
import ctypes
c=(ctypes.c_int*3)()
%timeit get_format_via_buffer(c) # 295 ns ± 10.3
%timeit memoryview(c).format # 936 ns ± 7.43 ns
On my machine, there are about 160 ns overhead of calling a def-function and about 50 ms for creating a bytes-object.
Even if it doesn't make much sense to optimize it further due to the unavoidable overhead, there is still at least theoretical interest of how it could be speed up.
If one really wants to shave off also the cost of filling out the Py_buffer-struct, than there is no clean way: ctypes-module isn't part of Python-C-API (it is not in the include-directory), so the way forward is to repeat the solution Cython uses with the array.array, i.e. hardcoding the memory layout of the object (which makes this solution brittle because the memory-layout of StgDictObject can get out of sync).
Here with Cython and without error-checking:
%%cython -a
from cpython cimport PyObject
# emulate memory-layout (i.e. copy definitions from ctypes.h)
cdef extern from *:
"""
#include <Python.h>
typedef struct _ffi_type
{
size_t size;
unsigned short mem[2];
struct _ffi_type **elements;
} ffi_type;
typedef struct {
PyDictObject dict; /* first part identical to PyDictObject */
Py_ssize_t size[3]; /* number of bytes,alignment requirements,number of fields */
ffi_type ffi_type_pointer;
PyObject *proto; /* Only for Pointer/ArrayObject */
void *setfunc[3];
/* Following fields only used by PyCFuncPtrType_Type instances */
PyObject *argtypes[4];
int flags; /* calling convention and such */
/* pep3118 fields, pointers neeed PyMem_Free */
char *format;
int ndim;
} StgDictObject;
"""
ctypedef struct StgDictObject:
char *format
def get_format_via_hack(obj):
cdef PyObject *p =<PyObject *>obj
cdef StgDictObject *dict = <StgDictObject *>(p.ob_type.tp_dict)
return dict.format
And it is as fast as it gets:
%timeit get_format_via_hack(c) # 243 ns ± 14.5 ns
I am struggling to initialise thread-local ndarrays with cython.parallel:
Pseudo-code:
cdef:
ndarray buffer
with nogil, parallel():
buffer = np.empty(...)
for i in prange(n):
with gil:
print "Thread %d: data address: 0x%x" % (threadid(), <uintptr_t>buffer.data)
some_func(buffer.data) # use thread-local buffer
cdef void some_func(char * buffer_ptr) nogil:
(... works on buffer contents...)
My problem is that in all threads buffer.data points to the same address. Namely the address of the thread that last assigned buffer.
Despite buffer being assigned within the parallel() (or alternatively prange) block, cython does not make buffer a private or thread-local variable but keeps it as a shared variable.
As a result, buffer.data points to the same memory region wreaking havoc on my algorithm.
This is not a problem exclusively with ndarray objects but seemingly with all cdef class defined objects.
How do I solve this problem?
I think I have finally found a solution to this problem that I like.
The short version is that you create an array that has shape:
(number_of_threads, ...<whatever shape you need in the thread>...)
Then, call openmp.omp_get_thread_num and use that to index the array to get a "thread-local" sub-array. This avoids having a separate array for every loop index (which could be enormous) but also prevents threads overwriting each other.
Here's a rough version of what I did:
import numpy as np
import multiprocessing
from cython.parallel cimport parallel
from cython.parallel import prange
cimport openmp
cdef extern from "stdlib.h":
void free(void* ptr)
void* malloc(size_t size)
void* realloc(void* ptr, size_t size)
...
cdef int num_items = ...
num_threads = multiprocessing.cpu_count()
result_array = np.zeros((num_threads, num_items), dtype=DTYPE) # Make sure each thread uses separate memory
cdef c_numpy.ndarray result_cn
cdef CDTYPE ** result_pointer_arr
result_pointer_arr = <CDTYPE **> malloc(num_threads * sizeof(CDTYPE *))
for i in range(num_threads):
result_cn = result_array[i]
result_pointer_arr[i] = <CDTYPE*> result_cn.data
cdef int thread_number
for i in prange(num_items, nogil=True, chunksize=1, num_threads=num_threads, schedule='static'):
thread_number = openmp.omp_get_thread_num()
some_function(result_pointer_arr[thread_number])