I know there are lots of similar questions on the website but I couldn't find an answer to my problem.
I'm wrapping C++ classes with Cython in order to use them with Python3. After building the external module with a setup.py, when I run the python program I got the following error:
from "name of.pyx file" import "name of the class to import"
Import error: /home/.../filename.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E.
I'm on Ubuntu 16.04, I build the extensions from the terminal with the command line python3 setup.py build_ext --inplace, and then run the .py from the terminal or from Spyder in Anaconda (I got the error in both cases.)
From what I read the error might come from the cython compilation because i'm not linking some libraries. Is this true? If it is, could someone explain me how to do it?
I let you here my setup.py, in comments all the different setups I tried.
setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
#setup(ext_modules = cythonize(
#"pycoralv1.pyx", # our Cython source
#sources=["coralv1cpp.cpp"], # additional source file(s)
#language="c++", # generate C++ code
#))
#setup(ext_modules = cythonize(Extension(
# "pyCoralv1", # the extension name
# sources=["pyCoralv1.pyx", "Coralv1cpp.cpp"], # the Cython source and
# additional C++ source files
# language="c++", # generate and compile C++ code
# )))
#setup(
# name = "testcoral",
# ext_modules = cythonize('*.pyx'),
#)
ext_modules = [
Extension(
"pyCoralv1",
sources=["pyCoralv1.pyx", "Coralv1cpp.cpp"],
extra_compile_args=['-fopenmp',"-fPIC"],
extra_link_args=['-fopenmp',"-I", "/usr/include/glib-2.0", "-l", "glib-2.0", "-I", "/usr/lib/x86_64-linux-gnu/glib-2.0/include"],
language="c++",
)
]
for e in ext_modules:
e.pyrex_directives = {"boundscheck": False}
setup(
name='Coral library',
ext_modules=cythonize(ext_modules),
include_dirs = [numpy.get_include()]
)
The problem was solved after installing libgcc in anaconda: conda install libgcc, there was a missing library.
Related
I am currently developing a python package that uses cython and numpy and I want the package to be installable using the pip install command from a clean python installation. All dependencies should be installed automatically. I am using setuptools with the following setup.py:
import setuptools
my_c_lib_ext = setuptools.Extension(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
setuptools.setup(
name="my_lib",
version="0.0.1",
author="Me",
author_email="me#myself.com",
description="Some python library",
packages=["my_lib"],
ext_modules=[my_c_lib_ext],
setup_requires=["cython >= 0.29"],
install_requires=["numpy >= 1.15"],
classifiers=[
"Programming Language :: Python :: 3",
"Operating System :: OS Independent"
]
)
This has worked great so far. The pip install command downloads cython for the build and is able to build my package and install it together with numpy.
Now I want to improve the performance of my cython code, which leads to some changes in my setup.py. I need to add include_dirs=[numpy.get_include()] to either the call of setuptools.Extension(...) or setuptools.setup(...) which means that I also need to import numpy. (See http://docs.cython.org/en/latest/src/tutorial/numpy.html and Make distutils look for numpy header files in the correct place for rationals.)
This is bad. Now the user cannot call pip install from a clean environment, because import numpy will fail. The user needs to pip install numpy before installing my library. Even if I move "numpy >= 1.15" from install_requires to setup_requires the installation fails, because the import numpy is evaluated earlier.
Is there a way to evaluate the include_dirs at a later point of the installation, for example, after the dependencies from setup_requires or install_requires have been resolved? I really like to have all dependencies resolved automatically and I dont want the user to type multiple pip install commands.
The following snippet works, but it is not officially supported because it uses an undocumented (and private) method:
class NumpyExtension(setuptools.Extension):
# setuptools calls this function after installing dependencies
def _convert_pyx_sources_to_lang(self):
import numpy
self.include_dirs.append(numpy.get_include())
super()._convert_pyx_sources_to_lang()
my_c_lib_ext = NumpyExtension(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
The article How to Bootstrap numpy installation in setup.py proposes using a cmdclass with custom build_ext class. Unfortunately, this breaks the build of the cython extension because cython also customizes build_ext.
First question, when is numpy needed? It is needed during the setup (i.e. when build_ext-funcionality is called) and in the installation, when the module is used. That means numpy should be in setup_requires and in install_requires.
There are following alternatives to solve the issue for the setup:
using PEP 517/518 (which is more straight forward IMO)
using setup_requires-argument of setup and postponing import of numpy until setup's requirements are satisfied (which is not the case at the start of setup.py's execution)
PEP 517/518-solution:
Put next to setup.py a pyproject.toml-file , with the following content:
[build-system]
requires = ["setuptools", "wheel", "Cython>=0.29", "numpy >= 1.15"]
which defines packages needed for building, and then install using pip install . in the folder with setup.py. A disadvantage of this method is that python setup.py install no longer works, as it is pip that reads pyproject.toml. However, I would use this approach whenever possible.
Postponing import
This approach is more complicated and somewhat hacky, but works also without pip.
First, let's take a look at unsuccessful tries so far:
pybind11-trick
#chrisb's "pybind11"-trick, which can be found here: With help of an indirection, one delays the call to import numpy until numpy is present during the setup-phase, i.e.:
class get_numpy_include(object):
def __str__(self):
import numpy
return numpy.get_include()
...
my_c_lib_ext = setuptools.Extension(
...
include_dirs=[get_numpy_include()]
)
Clever! The problem: it doesn't work with the Cython-compiler: somewhere down the line, Cython passes the get_numpy_include-object to os.path.join(...,...) which checks whether the argument is really a string, which it obviously isn't.
This could be fixed by inheriting from str, but the above shows the dangers of the approach in the long run - it doesn't use the designed mechanics, is brittle and may easily fail in the future.
the classical build_ext-solution
Which looks as following:
...
from setuptools.command.build_ext import build_ext as _build_ext
class build_ext(_build_ext):
def finalize_options(self):
_build_ext.finalize_options(self)
# Prevent numpy from thinking it is still in its setup process:
__builtins__.__NUMPY_SETUP__ = False
import numpy
self.include_dirs.append(numpy.get_include())
setupttools.setup(
...
cmdclass={'build_ext':build_ext},
...
)
Yet also this solution doesn't work with cython-extensions, because pyx-files don't get recognized.
The real question is, how did pyx-files get recognized in the first place? The answer is this part of setuptools.command.build_ext:
...
try:
# Attempt to use Cython for building extensions, if available
from Cython.Distutils.build_ext import build_ext as _build_ext
# Additionally, assert that the compiler module will load
# also. Ref #1229.
__import__('Cython.Compiler.Main')
except ImportError:
_build_ext = _du_build_ext
...
That means setuptools tries to use the Cython's build_ext if possible, and because the import of the module is delayed until build_ext is called, it founds Cython present.
The situation is different when setuptools.command.build_ext is imported at the beginning of the setup.py - the Cython isn't yet present and a fall back without cython-functionality is used.
mixing up pybind11-trick and classical solution
So let's add an indirection, so we don't have to import setuptools.command.build_ext directly at the beginning of setup.py:
....
# factory function
def my_build_ext(pars):
# import delayed:
from setuptools.command.build_ext import build_ext as _build_ext#
# include_dirs adjusted:
class build_ext(_build_ext):
def finalize_options(self):
_build_ext.finalize_options(self)
# Prevent numpy from thinking it is still in its setup process:
__builtins__.__NUMPY_SETUP__ = False
import numpy
self.include_dirs.append(numpy.get_include())
#object returned:
return build_ext(pars)
...
setuptools.setup(
...
cmdclass={'build_ext' : my_build_ext},
...
)
One (hacky) suggestion would be using the fact that extension.include_dirs is first requested in build_ext, which is called after the setup dependencies are downloaded.
class MyExt(setuptools.Extension):
def __init__(self, *args, **kwargs):
self.__include_dirs = []
super().__init__(*args, **kwargs)
#property
def include_dirs(self):
import numpy
return self.__include_dirs + [numpy.get_include()]
#include_dirs.setter
def include_dirs(self, dirs):
self.__include_dirs = dirs
my_c_lib_ext = MyExt(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
setup(
...,
setup_requires=['cython', 'numpy'],
)
Update
Another (less, but I guess still pretty hacky) solution would be overriding build instead of build_ext, since we know that build_ext is a subcommand of build and will always be invoked by build on installation. This way, we don't have to touch build_ext and leave it to Cython. This will also work when invoking build_ext directly (e.g., via python setup.py build_ext to rebuild the extensions inplace while developing) because build_ext ensures all options of build are initialized, and by coincidence, Command.set_undefined_options first ensures the command has finalized (I know, distutils is a mess).
Of course, now we're misusing build - it runs code that belongs to build_ext finalization. However, I'd still probably go with this solution rather than with the first one, ensuring the relevant piece of code is properly documented.
import setuptools
from distutils.command.build import build as build_orig
class build(build_orig):
def finalize_options(self):
super().finalize_options()
# I stole this line from ead's answer:
__builtins__.__NUMPY_SETUP__ = False
import numpy
# or just modify my_c_lib_ext directly here, ext_modules should contain a reference anyway
extension = next(m for m in self.distribution.ext_modules if m == my_c_lib_ext)
extension.include_dirs.append(numpy.get_include())
my_c_lib_ext = setuptools.Extension(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
setuptools.setup(
...,
ext_modules=[my_c_lib_ext],
cmdclass={'build': build},
...
)
I found a very easy solution in this post:
Or you can stick to https://github.com/pypa/pip/issues/5761. Here you install cython and numpy using setuptools.dist before actual setup:
from setuptools import dist
dist.Distribution().fetch_build_eggs(['Cython>=0.15.1', 'numpy>=1.10'])
Works well for me!
The goal is to use the pytest unit test framework for a Python3 project that uses Cython. This is not a plug-and-play thing, because pytest by default is not able to import the Cython modules.
One unsuccessful solution would be to use the pytest-cython plugin, but it simply does not work for me:
> py.test --doctest-cython
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --doctest-cython
inifile: None
rootdir: /censored/path/to/my/project/dir
To verify that I have the package installed:
> pip freeze | grep pytest-cython
pytest-cython==0.1.0
UPDATE:
I'm using PyCharm and it seems that it is not using my pip-installed packages but rather uses a custom(?) pycharm repository for packages used by my project. Once I added pytest-cython to that repository, the command runs but strange enough it doesn't recognize the Cython module anyway, although the package/add-on is specifically designed for that purpose:
> pytest --doctest-cython
Traceback:
tests/test_prism.py:2: in <module>
from cpc_naive.prism import readSequence, processInput
cpc_naive/prism.py:5: in <module>
from calculateScore import calculateScore, filterSortAlphas,
calculateAlphaMatrix_c#, incrementOverlapRanges # cython code
E ImportError: No module named 'calculateScore'
Another unsuccessful solution I got here is to use pytest-runner, but this yields:
> python3 setup.py pytest
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'pytest'
UPDATE:
I first had forgotten to add setup_requires=['pytest-runner', ...] and tests_require=['pytest', ...] to the setup script. Once i did that, I got another error:
> python3 setup.py pytest
Traceback (most recent call last):
File "setup.py", line 42, in <module>
tests_require=['pytest']
(...)
AttributeError: type object 'test' has no attribute 'install_dists'
UPDATE 2 (setup.py):
from distutils.core import setup
from distutils.extension import Extension
from setuptools import find_packages
from Cython.Build import cythonize
import numpy
try: # try to build the .c file
from Cython.Distutils import build_ext
except ImportError: # if the end-user doesn't have Cython that's OK; you should have shipped the .c files anyway.
use_cython = False
else:
use_cython = True
cmdclass = {}
ext_modules = []
if use_cython:
ext_modules += [
Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.pyx"],
extra_compile_args=['-g'], # -g for debugging
define_macros=[('CYTHON_TRACE', '1')]),
]
cmdclass.update({'build_ext': build_ext})
else:
ext_modules += [
Extension("cpc_naive.calculateScore", ["cpc_naive/calculateScore.c"],
define_macros=[('CYTHON_TRACE', '1')]), # compiled C files are stored in /home/pdiracdelta/.pyxbld/
]
setup(
name='cpc_naive',
author=censored,
author_email=censored,
license=censored,
packages=find_packages(),
cmdclass=cmdclass,
ext_modules=ext_modules,
install_requires=['Cython', 'numpy'],
include_dirs=[numpy.get_include()],
setup_requires=['pytest-runner'],
tests_require=['pytest']
)
UPDATE 3 (partial fix):
As suggested by #hoefling I downgraded pytest-runner to a version <4 (in fact 3.0.1) and this resolves the error in update 1, but now I get the same Exception as with the pytest-cython solution:
E ImportError: No module named 'calculateScore'
It just doesn't seem to recognize the module. Perhaps this is due to some absolute/relative import mojo I don't understand.
How can I use pytest with Cython? How can I discover why these methods aren't working and then fix it?
FINAL UPDATE:
After taking both the original problem and the question Updates into consideration (thanks #hoefling for solving these issues!), this question is now reduced to the question of:
why can pytest no import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine?
As #hoefling suggested, one should use pytest-runner version <0.4 to avoid the
AttributeError: type object 'test' has no attribute 'install_dists'
To then answer the actual and final question (in addition to partial, off-topic, user-specific fixes added to the question post itself) of why pytest cannot import the Cython module calculateScore, even though running the code just with python (no pytest) works just fine:
that remaining issue is solved here.
I just came across a very strange error while using nose and cython inside my virtualenv with python3. For some reason nosetests started giving me an ImportError even though python -m unittest basic_test.py was working. I made a new directory to reproduce the error to make sure there wasn't something weird in that directory.
Here are the three files: fileA.pyx, setup.py, and basic_test.py
file1.pyx
class FileA:
def __init__(self):
self.temp = {}
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('fileA', ['fileA.pyx'],)
]
setup(
name='test',
ext_modules=ext_modules,
cmdclass={'build_ext': build_ext},
)
basic_test.py:
def test():
from fileA import FileA
FileA()
assert True
Fresh directory. I run python setup.py build_ext --inplace. It compiles. I run nosetests the single test passes.
Then I do touch __init__.py and then run nosetests again and it fails with this error:
ImportError: No module named 'fileA'
Is this a bug or do I not understand how init affects imports?
Update:
I found this post about import traps and read something that might explain how adding init breaks it. I still don't get exactly how it fits in though.
This is an all new trap added in Python 3.3 as a consequence of fixing
the previous trap: if a subdirectory encountered on sys.path as part
of a package import contains an init.py file, then the Python
interpreter will create a single directory package containing only
modules from that directory, rather than finding all appropriately
named subdirectories as described in the previous section.
I tried to run a python code, say myfile.py (also tried to rename it as myfile.pyx) as follows:
import pyximport
pyximport.install(setup_args={"script_args":["--compiler=mingw32"]},
reload_support=True)
import myfile
myfile.mycode()
I am using PyCharm. The code seems to have run fine without any error and even gave me correct results on the Python Console within PyCharm.
However no pyd (or pxd) files were generated. How can I know if my code (myfile.mycode()) ran via Cython or via regular Python?
I am using Python 3.4, Cython 0.21.2.
Thanks
pyximport generates a temporary pyd file that is not in the working directory. You probably want to build a setup.py that looks something like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension('myfile',
sources=['myfile.pyx'],
language='c++',
)]
setup(
name = 'myfile',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
which you can compile using:
python setup.py build_ext -i clean
I have a little problem with cx_freeze and hoping one of you can help me. I have searched trough this wonderfull forum but I can't find the answer.
I have used cx_freeze before with python 3.3 and ktinker and that worked flawless.
Now I made a little tool with a bit more complex gui and tried Glade.
Building the gui with Glade works perfect for me and on Linux and Windows 7 the application I have made works fine (in python interpreter).
When I run python setup.py bdist_msi I don't see any faults but when I try to run the exe in windows I get this error window:
(I can't post images jet)
The last 4 lines are:
_load_backward_compatible
File "ExtentionLoader_gi_gi.py", line 22, in <module
File "ExtentionLoader_gi_gi.py", line 14, in_bootstrap_
ImportError: DLL load failed: The specified module could not be found
I don't use any plugins, exotic imports so the dll's I have to load are only the dll's for Gobject. The setup file I have made from an example on this forum. For my ktinker app I did not have to import any dll.
Finally the question: Is there a list of dll's somewhere that tells me what dll's I have to add?
And is there something wrong with my setup.py?
The code is nothing special but if you want to check it: https://github.com/EddenBeer/CodeGenerator
The imports in Python:
import csv import sys import datetime
from gi.repository import Gtk
Installed on Windows 7:
Python-3.4.2
cx_Freeze-4.3.3.win32-py3.4
pygi-aio-3.14.0_rev6-setup
Setup.py:
import os, site, sys
from cx_Freeze import setup, Executable
## Get the site-package folder, not everybody will install
## Python into C:\PythonXX
site_dir = site.getsitepackages()[1]
include_dll_path = os.path.join(site_dir, "gnome")
## Collect the list of missing dll when cx_freeze builds the app
missing_dll = ['libgtk-3-0.dll',
'libgdk-3-0.dll',
'libatk-1.0-0.dll',
'libcairo-gobject-2.dll',
'libgdk_pixbuf-2.0-0.dll',
'libjpeg-8.dll',
'libpango-1.0-0.dll',
'libpangocairo-1.0-0.dll',
'libpangoft2-1.0-0.dll',
'libpangowin32-1.0-0.dll',
'libgnutls-26.dll',
'libgcrypt-11.dll',
'libp11-kit-0.dll'
]
## We also need to add the glade folder, cx_freeze will walk
## into it and copy all the necessary files
glade_folder = 'glade'
## We need to add all the libraries too (for themes, etc..)
gtk_libs = ['etc', 'lib', 'share']
## Create the list of includes as cx_freeze likes
include_files = []
for dll in missing_dll:
include_files.append((os.path.join(include_dll_path, dll), dll))
## Let's add glade folder and files
include_files.append((glade_folder, glade_folder))
## Let's add gtk libraries folders and files
for lib in gtk_libs:
include_files.append((os.path.join(include_dll_path, lib), lib))
base = None
## Lets not open the console while running the app
if sys.platform == "win32":
base = "Win32GUI"
executables = [
Executable("CodeGenerator.py",
base=base
)
]
buildOptions = dict(
compressed = False,
includes = ["gi", "csv", "datetime",],
packages = ["gi"],
include_files = include_files
)
setup(
name = "Code Generator",
author = "Ed den Beer",
version = "1.0",
description = "Generating copy instructions for RsLogix5000 out of a list with tags in a CSV file",
options = dict(build_exe = buildOptions),
executables = executables
)
My problem is solved.
Looking for an answer if found a utility called ListDlls.exe.
In this link is explaned how to use it:
https://bitbucket.org/anthony_tuininga/cx_freeze/issue/92/pygi-and-cx_freeze-error