as of today, I have been able to compile my codes with Cython and speed up my programs. Now, I would like to learn parallel computing but since I am new to this I am trying simple examples first and I was trying to implement the one given here. When I compile my setup.py using the following code:
from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension('test_multi',
['test_multi.pyx'],
libraries=['m'],
extra_compile_args = ['-O3', '-ffast-math', '-march=native', '-fopenmp' ],
extra_link_args=['-fopenmp']
)
]
setup(cmdclass={"build_ext": build_ext}, ext_modules=ext_modules)
the compiler is warning me that it does not recognize '-O3', '-ffast-math', '-march=native', '-fopenmp' and the compile command is failing. I also tried to modify ext_modules as defined here:
ext_modules = [
Extension(
'test_multi',
['test_multi.pyx'],
extra_compile_args=['/openmp'],
extra_link_args=['/openmp'],
)
]
but also in this case, '/openmp' is not recognized.
What am I missing? I tried to have a look around but I was not able to find an answer (or at least if there was one, I did not get it). Could you please help me? If you need some extra info, please ask me. Thanks a lot!
EDIT: I am using Windows 10 and Visual Studio Build Tools 2019.
EDIT 2: the warning I get is LNK4044:unrecognized option '/openmp'; ignored.
Related
I am currently developing a python package that uses cython and numpy and I want the package to be installable using the pip install command from a clean python installation. All dependencies should be installed automatically. I am using setuptools with the following setup.py:
import setuptools
my_c_lib_ext = setuptools.Extension(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
setuptools.setup(
name="my_lib",
version="0.0.1",
author="Me",
author_email="me#myself.com",
description="Some python library",
packages=["my_lib"],
ext_modules=[my_c_lib_ext],
setup_requires=["cython >= 0.29"],
install_requires=["numpy >= 1.15"],
classifiers=[
"Programming Language :: Python :: 3",
"Operating System :: OS Independent"
]
)
This has worked great so far. The pip install command downloads cython for the build and is able to build my package and install it together with numpy.
Now I want to improve the performance of my cython code, which leads to some changes in my setup.py. I need to add include_dirs=[numpy.get_include()] to either the call of setuptools.Extension(...) or setuptools.setup(...) which means that I also need to import numpy. (See http://docs.cython.org/en/latest/src/tutorial/numpy.html and Make distutils look for numpy header files in the correct place for rationals.)
This is bad. Now the user cannot call pip install from a clean environment, because import numpy will fail. The user needs to pip install numpy before installing my library. Even if I move "numpy >= 1.15" from install_requires to setup_requires the installation fails, because the import numpy is evaluated earlier.
Is there a way to evaluate the include_dirs at a later point of the installation, for example, after the dependencies from setup_requires or install_requires have been resolved? I really like to have all dependencies resolved automatically and I dont want the user to type multiple pip install commands.
The following snippet works, but it is not officially supported because it uses an undocumented (and private) method:
class NumpyExtension(setuptools.Extension):
# setuptools calls this function after installing dependencies
def _convert_pyx_sources_to_lang(self):
import numpy
self.include_dirs.append(numpy.get_include())
super()._convert_pyx_sources_to_lang()
my_c_lib_ext = NumpyExtension(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
The article How to Bootstrap numpy installation in setup.py proposes using a cmdclass with custom build_ext class. Unfortunately, this breaks the build of the cython extension because cython also customizes build_ext.
First question, when is numpy needed? It is needed during the setup (i.e. when build_ext-funcionality is called) and in the installation, when the module is used. That means numpy should be in setup_requires and in install_requires.
There are following alternatives to solve the issue for the setup:
using PEP 517/518 (which is more straight forward IMO)
using setup_requires-argument of setup and postponing import of numpy until setup's requirements are satisfied (which is not the case at the start of setup.py's execution)
PEP 517/518-solution:
Put next to setup.py a pyproject.toml-file , with the following content:
[build-system]
requires = ["setuptools", "wheel", "Cython>=0.29", "numpy >= 1.15"]
which defines packages needed for building, and then install using pip install . in the folder with setup.py. A disadvantage of this method is that python setup.py install no longer works, as it is pip that reads pyproject.toml. However, I would use this approach whenever possible.
Postponing import
This approach is more complicated and somewhat hacky, but works also without pip.
First, let's take a look at unsuccessful tries so far:
pybind11-trick
#chrisb's "pybind11"-trick, which can be found here: With help of an indirection, one delays the call to import numpy until numpy is present during the setup-phase, i.e.:
class get_numpy_include(object):
def __str__(self):
import numpy
return numpy.get_include()
...
my_c_lib_ext = setuptools.Extension(
...
include_dirs=[get_numpy_include()]
)
Clever! The problem: it doesn't work with the Cython-compiler: somewhere down the line, Cython passes the get_numpy_include-object to os.path.join(...,...) which checks whether the argument is really a string, which it obviously isn't.
This could be fixed by inheriting from str, but the above shows the dangers of the approach in the long run - it doesn't use the designed mechanics, is brittle and may easily fail in the future.
the classical build_ext-solution
Which looks as following:
...
from setuptools.command.build_ext import build_ext as _build_ext
class build_ext(_build_ext):
def finalize_options(self):
_build_ext.finalize_options(self)
# Prevent numpy from thinking it is still in its setup process:
__builtins__.__NUMPY_SETUP__ = False
import numpy
self.include_dirs.append(numpy.get_include())
setupttools.setup(
...
cmdclass={'build_ext':build_ext},
...
)
Yet also this solution doesn't work with cython-extensions, because pyx-files don't get recognized.
The real question is, how did pyx-files get recognized in the first place? The answer is this part of setuptools.command.build_ext:
...
try:
# Attempt to use Cython for building extensions, if available
from Cython.Distutils.build_ext import build_ext as _build_ext
# Additionally, assert that the compiler module will load
# also. Ref #1229.
__import__('Cython.Compiler.Main')
except ImportError:
_build_ext = _du_build_ext
...
That means setuptools tries to use the Cython's build_ext if possible, and because the import of the module is delayed until build_ext is called, it founds Cython present.
The situation is different when setuptools.command.build_ext is imported at the beginning of the setup.py - the Cython isn't yet present and a fall back without cython-functionality is used.
mixing up pybind11-trick and classical solution
So let's add an indirection, so we don't have to import setuptools.command.build_ext directly at the beginning of setup.py:
....
# factory function
def my_build_ext(pars):
# import delayed:
from setuptools.command.build_ext import build_ext as _build_ext#
# include_dirs adjusted:
class build_ext(_build_ext):
def finalize_options(self):
_build_ext.finalize_options(self)
# Prevent numpy from thinking it is still in its setup process:
__builtins__.__NUMPY_SETUP__ = False
import numpy
self.include_dirs.append(numpy.get_include())
#object returned:
return build_ext(pars)
...
setuptools.setup(
...
cmdclass={'build_ext' : my_build_ext},
...
)
One (hacky) suggestion would be using the fact that extension.include_dirs is first requested in build_ext, which is called after the setup dependencies are downloaded.
class MyExt(setuptools.Extension):
def __init__(self, *args, **kwargs):
self.__include_dirs = []
super().__init__(*args, **kwargs)
#property
def include_dirs(self):
import numpy
return self.__include_dirs + [numpy.get_include()]
#include_dirs.setter
def include_dirs(self, dirs):
self.__include_dirs = dirs
my_c_lib_ext = MyExt(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
setup(
...,
setup_requires=['cython', 'numpy'],
)
Update
Another (less, but I guess still pretty hacky) solution would be overriding build instead of build_ext, since we know that build_ext is a subcommand of build and will always be invoked by build on installation. This way, we don't have to touch build_ext and leave it to Cython. This will also work when invoking build_ext directly (e.g., via python setup.py build_ext to rebuild the extensions inplace while developing) because build_ext ensures all options of build are initialized, and by coincidence, Command.set_undefined_options first ensures the command has finalized (I know, distutils is a mess).
Of course, now we're misusing build - it runs code that belongs to build_ext finalization. However, I'd still probably go with this solution rather than with the first one, ensuring the relevant piece of code is properly documented.
import setuptools
from distutils.command.build import build as build_orig
class build(build_orig):
def finalize_options(self):
super().finalize_options()
# I stole this line from ead's answer:
__builtins__.__NUMPY_SETUP__ = False
import numpy
# or just modify my_c_lib_ext directly here, ext_modules should contain a reference anyway
extension = next(m for m in self.distribution.ext_modules if m == my_c_lib_ext)
extension.include_dirs.append(numpy.get_include())
my_c_lib_ext = setuptools.Extension(
name="my_c_lib",
sources=["my_c_lib/some_file.pyx"]
)
setuptools.setup(
...,
ext_modules=[my_c_lib_ext],
cmdclass={'build': build},
...
)
I found a very easy solution in this post:
Or you can stick to https://github.com/pypa/pip/issues/5761. Here you install cython and numpy using setuptools.dist before actual setup:
from setuptools import dist
dist.Distribution().fetch_build_eggs(['Cython>=0.15.1', 'numpy>=1.10'])
Works well for me!
I know there are lots of similar questions on the website but I couldn't find an answer to my problem.
I'm wrapping C++ classes with Cython in order to use them with Python3. After building the external module with a setup.py, when I run the python program I got the following error:
from "name of.pyx file" import "name of the class to import"
Import error: /home/.../filename.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E.
I'm on Ubuntu 16.04, I build the extensions from the terminal with the command line python3 setup.py build_ext --inplace, and then run the .py from the terminal or from Spyder in Anaconda (I got the error in both cases.)
From what I read the error might come from the cython compilation because i'm not linking some libraries. Is this true? If it is, could someone explain me how to do it?
I let you here my setup.py, in comments all the different setups I tried.
setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
#setup(ext_modules = cythonize(
#"pycoralv1.pyx", # our Cython source
#sources=["coralv1cpp.cpp"], # additional source file(s)
#language="c++", # generate C++ code
#))
#setup(ext_modules = cythonize(Extension(
# "pyCoralv1", # the extension name
# sources=["pyCoralv1.pyx", "Coralv1cpp.cpp"], # the Cython source and
# additional C++ source files
# language="c++", # generate and compile C++ code
# )))
#setup(
# name = "testcoral",
# ext_modules = cythonize('*.pyx'),
#)
ext_modules = [
Extension(
"pyCoralv1",
sources=["pyCoralv1.pyx", "Coralv1cpp.cpp"],
extra_compile_args=['-fopenmp',"-fPIC"],
extra_link_args=['-fopenmp',"-I", "/usr/include/glib-2.0", "-l", "glib-2.0", "-I", "/usr/lib/x86_64-linux-gnu/glib-2.0/include"],
language="c++",
)
]
for e in ext_modules:
e.pyrex_directives = {"boundscheck": False}
setup(
name='Coral library',
ext_modules=cythonize(ext_modules),
include_dirs = [numpy.get_include()]
)
The problem was solved after installing libgcc in anaconda: conda install libgcc, there was a missing library.
I'm trying to use cx_Freeze to generate a .app from a python project. Generally I have it working, but some of my modules which depend on scipy have an import error when executed:
No module named '_csr'
under the build folder I see a file:
scipy.sparse.sparsetools._csr.so
and watching the output of the build command seems to suggest that it's copying csr:
$ python3 setup.py bdist_mac | grep csr
m scipy.sparse.csr /usr/local/lib/python3.3/site-packages/scipy/sparse/csr.py
m scipy.sparse.sparsetools._csr /usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csr.so
m scipy.sparse.sparsetools.csr /usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/csr.py
? _csr imported from scipy.sparse.sparsetools.csr
? os.path imported from NIF_WRF.util.StopPow, distutils.file_util, matplotlib.backends.backend_tkagg, matplotlib.cbook, numpy.core.memmap, numpy.distutils.command.scons, os, pkg_resources, pkgutil, scipy.lib.blas.scons_support, scipy.lib.blas.setup, scipy.lib.lapack.scons_support, scipy.linalg.setup, scipy.sparse.csgraph.setup, scipy.sparse.linalg.dsolve.setup, scipy.sparse.linalg.eigen.arpack.setup, scipy.sparse.linalg.isolve.setup, scipy.sparse.sparsetools.bsr, scipy.sparse.sparsetools.coo, scipy.sparse.sparsetools.csc, scipy.sparse.sparsetools.csgraph, scipy.sparse.sparsetools.csr, scipy.sparse.sparsetools.dia, scipy.special.setup, shutil, sysconfig
? scipy.lib.six.moves imported from scipy.integrate.quadrature, scipy.interpolate.interpolate, scipy.interpolate.polyint, scipy.linalg.special_matrices, scipy.misc.common, scipy.optimize.anneal, scipy.optimize.linesearch, scipy.optimize.nonlin, scipy.sparse.base, scipy.sparse.compressed, scipy.sparse.coo, scipy.sparse.csc, scipy.sparse.csr, scipy.sparse.dok, scipy.sparse.lil, scipy.sparse.linalg.eigen.lobpcg.lobpcg, scipy.sparse.linalg.isolve.lgmres, scipy.spatial.distance, scipy.special.basic, scipy.stats.stats
copying /usr/local/lib/python3.3/site-packages/scipy/sparse/sparsetools/_csr.so -> build/exe.macosx-10.8-x86_64-3.3/scipy.sparse.sparsetools._csr.so
The problem seems to be related to this other question but that user seemed to solve it by building again, which hasn't helped here. Any ideas?
UPDATE
I mucked around in the .app package contents and found that renaming scipy.sparse.sparsetools._csr.so to _csr.so solves that error (though generates another similar one for another scipy component). It seems like the cx_Freeze script is not properly naming scipy inputs.
Also, here are the versions I'm using:
cx_Freeze: 4.3.2
scipy: 0.13.0
python: 3.3.2
i am trying to create an executable from my python script. My script runs fine, but after freezing it, starting the .exe gives me the following error:
http://www.bild.me/bild.php?file=4663406scipyerror.png
I am using Python 3.2.3, Scipy 0.12.0b1, Numpy 1.7.0 and Matplotlib 1.2.0 (all 32bit).
Any ideas/hints on how to solve this? My guess is i have to include something manually in my freezing script, but i am running out of guesses :-(
I got it finally to work, but I am very unsatisfied with my solution:
1) copy _odepack.pyd and odepack.py from the SciPy package to my program folder
2) in odepack.py change from . import _odepack to import _odepack (otherwise ValueError: Attempted relative import in non-package is raised)
3) in my main change from scipy.integrate import odeint to from odepack import odeint
Now it is working as expected and after using cx_freeze it is still working.
Still got no idea why it would not work before :-(
Thanks ThomasK for pushing me in the right direction though :-)
I finally got around this vode-problem by specifying "scipy.integrate.vode" as an include in the cx setup-file. This resulted in a file "scipy.integrate.vode.pyd" to end up in the build folder. I am using SciPy 0.11, Python 3.2.3 and the latest cx on Windows.
But adding such a "scipy.integrate.vode" file manually to the build folder would not fix the problem for me either, even though such manual-include-fixes were needed for many other .pyd files cx could not find either (and whereby the above setup.py include-solution would not work instead)...
Thanks for sharing your distress and wisdom, would not have managed to freeze my program otherwize...
Trying to learn cx-freeze. I have a python program that I am trying freeze to exe.
I use PySerial and no matter how I try to include win32 nothing seems to help. I use Python 3.2 and win7.
I have searched the web thin, and others have had the same problem, but no solution seems to be appearing. But I doubt that no one have succeeded in cx_freezing something that uses PySerial.
I am completely stuck. Any help would be much appreciated
Error:
Traceback (most recent call last):
File "C:\Python32\lib\site-packages\
7, in <module>
exec(code, m.__dict__)
File "snapper.py", line 8, in
File "C:\Python32\lib\site-packages\
from serial.serialwin32 import *
File "C:\Python32\lib\site-packages\
e>
from serial import win32
ImportError: cannot import name win32
Setup.py:
from cx_Freeze import setup,Executable
includefiles = ['caml.pkl', 'seql.pkl']
includes = ['DataBase', 'serial.win32']
excludes = ['Tkinter']
packages = []
setup(
name = 'Setup',
version = '0.1',
description = 'Snapper configuration utility',
author = 'LST',
author_email = 'info#-.com',
options = {'build_exe': {'excludes':excludes,'packages':packages,'include_files':includefiles}},
executables = [Executable('snapper.py')]
)
Any idea where to go from here?
Thanks in advance
I tried to do a blind import:
if False:
import serial.win32
no luck...
Maybe i am looking at this the wrong way....
Okay, problem solved.
You need to use packages to force cx_Freeze to include serial.win32 (not "include")
Following line works:
packages = ['serial.win32']
Memo to my self and others: Be sure to check the dist folder for actually included packages. I have no idea why all packages didn't get included by cx_Freeze in the first place, but this works for me.
If you can use a different tool to freeze your program, PyInstaller says it supports PySerial.