Link f2py generated *.so file in a python package using setuptools - python-3.x

Question

I wish to deploy a package to PyPi using setuptools. However, the core part of the package is actually written in Fortran, and I am using f2py to wrap it in python. Basically the project's structure looks like this:
my_project
license.txt
README.md
setup.py
my_project
init.py
myfunc.py
hello.so
The module myfunc.py imports hello.so (import my_project.hello) which can then be used by functions inside myfunc.py. This works perfectly on my machine.
Then I tried standard setuptools installation: sudo python3 setup.py install on my Ubuntu, and it gets installed perfectly. But unfortunately, while importing, it throws ModuleNotFoundError: No module named 'hello'.
Now, from what I understand, on Linux based systems, for python, the shared libraries *.so are stored in /usr/lib/python3/dist-packages/. So I manually copied this hello.so there, and I got a working package! But of course this works only locally. What I would like to do is to tell setuptools to include hello.so inside the python-egg and automatically do the copying etc so that when a user uses pip3 install my_package, they will have access to this shared library automatically. I can see that numpy has somehow achieved that but even after looking at their code, I haven't been able to decode how they did it. Can someone help me with this? Thanks in advance.

Answer 1

You can achieve this with a setup.py file like this (simplified version, keep only the relevant parts for building external modules)
import os
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
class f2py_Extension(Extension):
def __init__(self, name, sourcedirs):
Extension.__init__(self, name, sources=[])
self.sourcedirs = [os.path.abspath(sourcedir) for sourcedir in sourcedirs]
self.dirs = sourcedirs
class f2py_Build(build_ext):
def run(self):
for ext in self.extensions:
self.build_extension(ext)
def build_extension(self, ext):
# compile
for ind,to_compile in enumerate(ext.sourcedirs):
module_loc = os.path.split(ext.dirs[ind])[0]
module_name = os.path.split(to_compile)[1].split('.')[0]
os.system('cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name))
setup(
name="foo",
ext_modules=[f2py_Extension('fortran_external',['foo/one.F90','foo/bar/two.F90'])],
cmdclass=dict(build_ext=f2py_Build),
)
The essential parts for building an external module are ext_modules and cmdclass in setup(...). ext_modules is just a list of Extension instances, each of which describes a set of extension modules. In the setup.py above, I tell ext_modules I want to create two external modules with two source files foo/test.F90 and foo/bar/two.F90. Based on ext_modules, cmdclass is responsible for compiling the two modules, in our case, the command for compiling the module is
'cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name)
Project structure before installation
├── foo
│   ├── __init__.py
│   ├── bar
│   │   └── two.F90
│   └── one.F90
└── setup.py
Project structure after python setup.py install
├── build
│   └── bdist.linux-x86_64
├── dist
│   └── foo-0.0.0-py3.7-linux-x86_64.egg
├── foo
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-37.pyc
│   ├── bar
│   │   ├── two.F90
│   │   └── two.cpython-37m-x86_64-linux-gnu.so
│   ├── one.F90
│   └── one.cpython-37m-x86_64-linux-gnu.so
├── foo.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
└── setup.py
The two source files one.F90 and two.F90 are very simple
one.F90
module test
implicit none
contains
subroutine add(a)
implicit none
integer :: a
integer :: b
b = a + 1
print *, 'one',b
end subroutine add
end module test
two.F90
module test
implicit none
contains
subroutine add(a)
implicit none
integer :: a
integer :: b
b = a + 2
print *, 'two',b
end subroutine add
end module test
After I installed the package, I can successfully run
>>> from foo.bar.two import test
>>> test.add(5)
two 7
and
>>> from foo.one import test
>>> test.add(5)
one 6

Answer 2

Here is an approach based on F2PY's documentation (the example there covers building multiple F2PY modules, and multiple source files per module), making use of numpy.distutils, that supports Fortran source files.
The structure of a minimal example with multiple F2PY extension modules is based on a src directory layout. It is not necessary/required, but has the advantage that the test routine cannot run unless the package has been installed successfully.
Source layout
my_project
|
+-- src
| |
| +-- my_project
| |
| +-- __init__.py
| +-- mod1.py
| +-- funcs_m.f90
| +-- two
| |
| +-- pluss2.f90
| +-- times2.f90
|
+-- test_my_project.py
+-- setup.py
setup.py
from setuptools import find_packages
from numpy.distutils.core import setup, Extension
ext1 = Extension(name='my_project.modf90',
sources=['src/my_project/funcs_m.f90'],
f2py_options=['--quiet'],
)
ext2 = Extension(name='my_project.oldf90',
sources=['src/my_project/two/plus2.f90', 'src/my_project/two/times2.f90'],
f2py_options=['--quiet'],
)
setup(name="my_project",
version="0.0.1",
package_dir={"": "src"},
packages=find_packages(where="src"),
ext_modules=[ext1, ext2])
__init__.py
The __init__.py file is empty. (Can e.g. import the F2PY modules here if desired)
mod1.py
def add(a, b):
""" add inputs a and b, and return """
return a + b
funcs_m.f90
module funcs_m
implicit none
contains
subroutine add(a, b, c)
integer, intent(in) :: a
integer, intent(in) :: b
integer, intent(out) :: c
c = a + b
end subroutine add
end module funcs_m
plus2.f90
subroutine plus2(x, y)
integer, intent(in) :: x
integer, intent(out) :: y
y = x + 2
end subroutine plus2
times2.f90
subroutine times2(x, y)
integer, intent(in) :: x
integer, intent(out) :: y
y = x * 2
end subroutine times2
test_my_project.py
import my_project.mod1
import my_project.oldf90
import my_project.modf90
print("mod1.add: 1 + 2 = ", my_project.mod1.add(1, 2))
print("modf90.funcs_m.add: 1 + 2 = ", my_project.modf90.funcs_m.add(1, 2))
x = 1
x = my_project.oldf90.plus2(x)
print("oldf90.plus2: 1 + 2 = ", x)
x = my_project.oldf90.times2(x)
print("oldf90.plus2: 3 * 2 = ", x)
Installing
Now, one can use pip to install the package. There are several advantages to using pip (including ease of upgrading, or uninstalling) as opposed to setup.py install (but this can still be used for building the package for distribution!). From the directory containing setup.py:
> python -m pip install .
Testing
And then, to test the just installed package
> python test_my_project.py
mod1.add: 1 + 2 = 3
modf90.funcs_m.add: 1 + 2 = 3
oldf90.plus2: 1 + 2 = 3
oldf90.plus2: 3 * 2 = 6
This setup has been tested with success on Windows 10 (with ifort), on Ubuntu 18.04 (with gfortran) and on MacOS High Sierra (with gfortran), all with Python 3.6.3.