How to use Cython with pytest? - python-3.x

Question

The goal is to use the pytest unit test framework for a Python3 project that uses Cython. This is not a plug-and-play thing, because pytest by default is not able to import the Cython modules.
One unsuccessful solution would be to use the pytest-cython plugin, but it simply does not work for me:
> py.test --doctest-cython
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --doctest-cython
inifile: None
rootdir: /censored/path/to/my/project/dir
To verify that I have the package installed:
> pip freeze | grep pytest-cython
pytest-cython==0.1.0
UPDATE:
I'm using PyCharm and it seems that it is not using my pip-installed packages but rather uses a custom(?) pycharm repository for packages used by my project. Once I added pytest-cython to that repository, the command runs but strange enough it doesn't recognize the Cython module anyway, although the package/add-on is specifically designed for that purpose:
> pytest --doctest-cython
Traceback:
tests/test_prism.py:2: in <module>
from cpc_naive.prism import readSequence, processInput
cpc_naive/prism.py:5: in <module>
from calculateScore import calculateScore, filterSortAlphas,
calculateAlphaMatrix_c#, incrementOverlapRanges # cython code
E ImportError: No module named 'calculateScore'
Another unsuccessful solution I got here is to use pytest-runner, but this yields:
> python3 setup.py pytest
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'pytest'
UPDATE:
I first had forgotten to add setup_requires=['pytest-runner', ...] and tests_require=['pytest', ...] to the setup script. Once i did that, I got another error:
> python3 setup.py pytest
Traceback (most recent call last):
File "setup.py", line 42, in <module>
tests_require=['pytest']
(...)
AttributeError: type object 'test' has no attribute 'install_dists'
UPDATE 2 (setup.py):
from distutils.core import setup
from distutils.extension import Extension
from setuptools import find_packages
from Cython.Build import cythonize
import numpy
try: # try to build the .c file
from Cython.Distutils import build_ext
except ImportError: # if the end-user doesn't have Cython that's OK; you should have shipped the .c files anyway.
use_cython = False
else:
use_cython = True
cmdclass = {}
ext_modules = []
if use_cython:
ext_modules += [
Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.pyx"],
extra_compile_args=['-g'], # -g for debugging
define_macros=[('CYTHON_TRACE', '1')]),
]
cmdclass.update({'build_ext': build_ext})
else:
ext_modules += [
Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.c"],
define_macros=[('CYTHON_TRACE', '1')]), # compiled C files are stored in /home/pdiracdelta/.pyxbld/
]
setup(
name='cpc_naive',
author=censored,
author_email=censored,
license=censored,
packages=find_packages(),
cmdclass=cmdclass,
ext_modules=ext_modules,
install_requires=['Cython', 'numpy'],
include_dirs=[numpy.get_include()],
setup_requires=['pytest-runner'],
tests_require=['pytest']
)
UPDATE 3 (partial fix):
As suggested by #hoefling I downgraded pytest-runner to a version <4 (in fact 3.0.1) and this resolves the error in update 1, but now I get the same Exception as with the pytest-cython solution:
E ImportError: No module named 'calculateScore'
It just doesn't seem to recognize the module. Perhaps this is due to some absolute/relative import mojo I don't understand.
How can I use pytest with Cython? How can I discover why these methods aren't working and then fix it?
FINAL UPDATE:
After taking both the original problem and the question Updates into consideration (thanks #hoefling for solving these issues!), this question is now reduced to the question of:
why can pytest no import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine?

Answer 1

As #hoefling suggested, one should use pytest-runner version <0.4 to avoid the
AttributeError: type object 'test' has no attribute 'install_dists'
To then answer the actual and final question (in addition to partial, off-topic, user-specific fixes added to the question post itself) of why pytest cannot import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine:
that remaining issue is solved here.