Undefined behavior when using Cython and external multithreaded C library - python-3.x

Question

I am new to Cython (as well as Python), and I am trying to understand where I am doing wrong when I try to expose the C-API of an external multithreaded library to Python. To illustrate my problem, I will go over a hypothetical MWE.
Let's assume that I have the following directory structure
.
├── app.py
├── c_mylib.pxd
├── cxx
│ ├── CMakeLists.txt
│ ├── include
│ │ └── mylib.h
│ └── src
│ └── reduce_cb.cpp
├── mylib.pyx
└── setup.py
Here, cxx contains the external multithreaded library as follows (header and implementation files are concatenated):
/* cxx/include/mylib.h */
#ifndef MYLIB_H_
#define MYLIB_H_
#ifdef __cplusplus
extern "C" {
#endif
typedef double (*func_t)(const double *, const double *, void *);
double reduce_cb(const double *, const double *, func_t, void *);
#ifdef __cplusplus
}
#endif
#endif
/* cxx/src/reduce_cb.cpp */
#include <iterator>
#include <mutex>
#include <thread>
#include <vector>
#include "mylib.h"
extern "C" {
double reduce_cb(const double *xb, const double *xe, func_t func, void *data) {
const auto d = std::distance(xb, xe);
const auto W = std::thread::hardware_concurrency();
const auto split = d / W;
const auto remain = d % W;
std::vector<std::thread> workers(W);
double res{0};
std::mutex lock;
const double *xb_w{xb};
const double *xe_w;
for (unsigned int widx = 0; widx < W; widx++) {
xe_w = widx < remain ? xb_w + split + 1 : xb_w + split;
workers[widx] = std::thread(
[&lock, &res, func, data](const double *xb, const double *xe) {
const double partial = func(xb, xe, data);
std::lock_guard<std::mutex> guard(lock);
res += partial;
},
xb_w, xe_w);
xb_w = xe_w;
}
for (auto &worker : workers)
worker.join();
return res;
}
}
with the accompanying cxx/CMakeLists.txt file as follows:
cmake_minimum_required(VERSION 3.9)
project(dummy LANGUAGES CXX)
add_library(mylib
include/mylib.h
src/reduce_cb.cpp
)
target_compile_features(mylib
PRIVATE
cxx_std_11
)
target_include_directories(mylib
PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
)
set_target_properties(mylib
PROPERTIES PUBLIC_HEADER include/mylib.h
)
install(TARGETS mylib
ARCHIVE DESTINATION lib
LIBRARY DESTINATION lib
PUBLIC_HEADER DESTINATION include
)
The corresponding Cython files are as follows (this time definition and implementation files are concatenated):
# c_mylib.pxd
cdef extern from "include/mylib.h":
ctypedef double (*func_t)(const double *, const double *, void *)
double reduce_cb(const double *, const double *, func_t, void *)
# mylib.pyx
# cython: language_level = 3
cimport c_mylib
cdef double func(const double *xb, const double *xe, void *data):
cdef int d = (xe - xb)
func = <object>data
return func(<double[:d]>xb)
def reduce_cb(double [::1] arr not None, f):
cdef int d = arr.shape[0]
data = <void*>f
return c_mylib.reduce_cb(&arr[0], &arr[0] + d, func, data)
# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules=cythonize([
Extension("mylib", ["mylib.pyx"], libraries=["mylib"])
])
)
Building the C++ library, and building the Cython extension module and linking it against the C++ library following the instructions, I get undefined behavior when I try to run
import mylib
from numpy import array
def cb(x):
partial = 0
for idx in range(x.shape[0]):
partial += x[idx]
return partial
arr = array([val + 1 for val in range(100)], "d")
print("sum(arr): ", mylib.reduce_cb(arr, cb))
By undefined behavior, I mean that I get either of
SIGSEGV (Address boundary error),
"Fatal Python error: GC object already tracked" with SIGABRT, or,
(rarely) the correct result.
I have checked the documentation of Cython thoroughly (I guess), and I have searched both SO and Google for this problem, but I could not find a proper solution to this problem.
Basically, I would like to have a C library, which is unaware of Python and which uses callback functions from multiple threads, that is integrated inside Python. Is this at all possible? I tried nogil signatures and with gil: blocks as discussed in Cython's documentation, but I got compilation errors. Moreover, gc related functionality in Cython seems to be valid only for extension types, which does not apply to my case.
I am stuck and I would appreciate any hint/help.

Answer 1

That happens when you use Python-objects/functionality without lock. Your critical section is not only the summation but also the call to function func, i.e.:
workers[widx] = std::thread(
[&lock, &res, func, data](const double *xb, const double *xe) {
std::lock_guard<std::mutex> guard(lock);
const double partial = func(xb, xe, data); // must be guarded
res += partial;
},
xb_w, xe_w);
which makes the parallelization senseless in the first place, doesn't it? Probably, from software-engineering point of view, a better place for guard would be in the wrapper function func - but I have put it into worker because the consequences are seen much better this way.
Python uses reference counting for memory management - similar to std::shared_ptr. However it doesn't lock with fine granularity as shared_ptr, which locks only when changing the reference counter, but uses a more coarse lock - the global interpreter lock. That has the consequence, that when one changes the reference count of an python-object from open-MP-thread or other threads not registered in Python-interpreter the reference-counter is not protected/guarded and race conditions arise. What you are observing, are the possible results of such race conditions.
The GIL makes your endeavor more or less impossible: you need to lock every call to possible python but than you serialize the calls to this functionality!