Why do include paths in python2 and python3 differ? - python-3.x

Question

While trying to make my scons based build system as platform independent as possible I was wondering about the following:
Why does python2 return the include path /usr/local/include/python2.7? This path does not contain Python.h and building fails if I rely on that path.
python2
Use sysconfig inside python2:
$ /usr/bin/python2
Python 2.7.13 (default, Nov 23 2017, 15:37:09)
[GCC 6.3.0 20170406] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/local/include/python2.7'
Gives /usr/local/include/python2.7. This is an empty folder.
Call python2-config from shell:
$ /usr/bin/python2-config --includes
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7
This gives a different path. I was able to find Python.h in /usr/include/python2.7.
python3
Use sysconfig inside python3:
$ /usr/bin/python3
Python 3.5.3 (default, Nov 23 2017, 11:34:05)
[GCC 6.3.0 20170406] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/include/python3.5m'
Call python3-config from shell:
/usr/bin/python3-config --includes
-I/usr/include/python3.5m -I/usr/include/python3.5m
The resulting path /usr/include/python3.5m is the same for boath approaches. Python.h is located here.
anaconda
If I use anaconda python (2 or 3) the paths are also consistent (as with python3).
I already found some workarounds such as placing a softlink to usr/include in usr/local/include or just throwing away the local/ in the path, but both do not really look like a nice solution.
edit:
Currently the incorrect include path in python2 makes my build system not as platform independend as I would like. Adding an (optional) environment variable PYTHON_INCLUDE_PATH gives me the possibility to define the correct path, if python2 is used. But a solution where the correct path is always returned would help me a lot (either on python side or using scons-specific functionality).
As my build system is based on scons a

Answer 1

Python uses an installation scheme that differs depending on the
platform and on the installation options.
https://docs.python.org/2/library/sysconfig.html#installation-paths
There are various schemes including posix_local and posix_prefix which determine the location of various installation directories. It seems that sysconfig does not actually record which scheme was used to install the particular build of Python -- it only has the build information.
So when you call sysconfig.get_path() it guesses the scheme based on the default for the current platform [1]. The Python2.7 sysconfig guesses posix_local while the Python3 sysconfig guesses posix_prefix [2].
It looks like both versions of Python are installed using the posix_prefix scheme, so you can specify that when you call sysconfig.get_path:
$ python -c "import sysconfig; print(sysconfig.get_path('include', 'posix_prefix'))"
/usr/include/python2.7
$ python3 -c "import sysconfig; print(sysconfig.get_path('include', 'posix_prefix'))"
/usr/include/python3.5m
[1] https://github.com/python/cpython/blob/2.7/Lib/sysconfig.py#L169
[2] Run sysconfig as a script:
$ python -m sysconfig | head
Platform: "linux-x86_64"
Python version: "2.7"
Current installation scheme: "posix_local"
$ python3 -m sysconfig | head
Platform: "linux-x86_64"
Python version: "3.5"
Current installation scheme: "posix_prefix"
I can't actually find posix_local in the sysconfig source so not sure where it comes from.
EDIT
I looked into this a bit more and learned that it specific to the Debian/Ubuntu versions of Python; upstream Python doesn't use /usr/local/ or have a posix_local scheme. The Debian packages use a kind of hybrid approach which is the same as posix_prefix with the addition of /usr/local/ for modules.
I haven't found a link the source online, but my local system has this in its Python2.7 sysconfig.py (notice the FIXME):
def _get_default_scheme():
if os.name == 'posix':
# the default scheme for posix on Debian/Ubuntu is posix_local
# FIXME: return dist-packages/posix_prefix only for
# is_default_prefix and 'PYTHONUSERBASE' not in os.environ and 'real_prefix' not in sys.__dict__
# is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
return 'posix_local'
return os.name
The Debian python3 sysconfig.py does away with posix_local and uses the same default as upstream python:
def _get_default_scheme():
if os.name == 'posix':
# the default scheme for posix is posix_prefix
return 'posix_prefix'
return os.name
You might want to replicate that to be compatible with Mac or Windows:
sysconfig.get_path('include', 'posix_prefix' if os.name == 'posix' else os.name)
https://wiki.debian.org/Python#Deviations_from_upstream
https://www.debian.org/doc/packaging-manuals/python-policy/python.html#paths