I'm trying to switch from python2 to python3. In the process I'm also switching from anaconda to miniconda3 as my primary package management tool. There are some other packages that I clone from github. I found that I can no longer import any modules from packages downloaded from github rather than conda. For example, import linetools.utils used to work and no longer works. import linetools.linetools.utils works but this is not enough, since various modules inside the package reference each other. This package is not written by me so changing all occurrences of linetools.utils to linetools.linetools.utils is not desirable.
Here are additional information for you to help solve my problem.
My $PYTHONPATH environment variable is set to /Users/lwymarie/python/. I also tried putting this same path to my $path variable as well. Didn't solve my problem.
Here are the packages I have under PYTHONPATH. Just showing a few.
flemish.local> ls $PYTHONPATH
sdsspy Barak desisim linetools desiutil pymc3 specutils PypeIt pyqtgraph statsmodels
RemoteObserving fitsio Ska.Numpy ginga redrock
Here is the expected behavior, when I used python2 and anaconda2. I'm using the package linetools as an example.
flemish.local> python
Python 2.7.12 |Anaconda custom (x86_64)| (default, Jul 2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import linetools.utils
Here's the sys.path when the above worked. Just showing a few of the elements.
>>> import sys
>>> sys.path
['', '/Users/lwymarie/python', '/Users/lwymarie/anaconda/lib/python27.zip', '/Users/lwymarie/anaconda/lib/python2.7', '/Users/lwymarie/anaconda/lib/python2.7/plat-darwin', '/Users/lwymarie/anaconda/lib/python2.7/plat-mac', '/Users/lwymarie/anaconda/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/lwymarie/anaconda/lib/python2.7/lib-tk', '/Users/lwymarie/anaconda/lib/python2.7/lib-old', '/Users/lwymarie/anaconda/lib/python2.7/lib-dynload', '/Users/lwymarie/anaconda/lib/python2.7/site-packages', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/PIL', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/aeosa', '/Users/lwymarie/python/specutils', '/Users/lwymarie/python/ginga', '/Users/lwymarie/python/statsmodels', '/Users/lwymarie/python/Ska.Numpy', '/Users/lwymarie/python/linetools', '/Users/lwymarie/python/pymc3', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/numpydoc-0.6.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/nbsphinx-0.2.9-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/recommonmark-0.4.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/CommonMark-0.5.4-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/joblib-0.10.2-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/Theano-0.8.2-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/ginga-2.7.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/QtPy-1.3.1-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/desiutil-1.9.9.dev596-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/desisim-0.24.0.dev1308-py2.7.egg', '/Users/lwymarie/python/redrock/py', '/Users/lwymarie/python/PypeIt', '/Users/lwymarie/anaconda/lib/python2.7/site-packages']
Exiting Python, here's the $path variable when the above worked.
flemish.local> echo $path
/Users/lwymarie/anaconda/bin . /usr/local/bin /usr/local/etc /opt/local/bin /opt/local/sbin /usr/sbin /sbin /usr/bin /bin /usr/lang /etc /usr/etc /usr/X11/bin /usr/local/scisoft/bin /usr/local/texlive/2020/bin/x86_64-darwin /Library/Ruby/Gems/2.0.0 ./py/ ./
Here is the unwanted behavior, when I used python3 and miniconda3.
flemish.local> python
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import linetools.utils
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'linetools.utils'
Here's the sys.path when the above error happened. I also tried the sys.path.append("/Users/lwymarie/python/linetools/") trick and it still couldn't find the module.
>>> import sys
>>> sys.path
['', '/Users/lwymarie/python', '/Users/lwymarie/miniconda3/lib/python37.zip', '/Users/lwymarie/miniconda3/lib/python3.7', '/Users/lwymarie/miniconda3/lib/python3.7/lib-dynload', '/Users/lwymarie/miniconda3/lib/python3.7/site-packages']
Exiting Python, here's the $path variable when the above error happened.
flemish.local> echo $path
/Users/lwymarie/miniconda3/bin /Users/lwymarie/miniconda3/bin . /usr/local/bin /usr/local/etc /opt/local/bin /opt/local/sbin /usr/sbin /sbin /usr/bin /bin /usr/lang /etc /usr/etc /usr/X11/bin /usr/local/scisoft/bin /usr/local/texlive/2020/bin/x86_64-darwin /Library/Ruby/Gems/2.0.0 ./py/ ./
Here's an image of the directory tree of the linetools package.
If you successfully help me get python3 and linetools working, your good deed will be rewarded with eternal life and happiness.
There is '/Users/lwymarie/python/linetools' in your python2 sys.path but not in your python3 sys.path. Did you re-setup linetools using python3?
cd ~/python/linetools
python setup.py develop
I think what Steven Lau suggested should work.
But keep in mind that you have multiple index paths. You have /Users/lwymarie/python and /Users/lwymarie/python/linetools (I see this for your py2 setup).
Related
I have two versions of python in my mac os, the first python2.7 which is the default that came with the system. Later I installed python3.7 that I use most of the time.
I have recently installed openCV using homebrew.
When I'm using openCV with python2.7, it's working normally.
But the problem is when I try to use it with python3. Importing cv2 in python3 gives error: ModuleNotFoundError: No module named 'cv2'
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "copyright", "credits" or "license()" for more information.
>>> WARNING: The version of Tcl/Tk (8.5.9) in use may be unstable.
Visit http://www.python.org/download/mac/tcltk/ for current
information.
>>> import cv2
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
import cv2
ModuleNotFoundError: No module named 'cv2'
>>>
Is there anything I can do so that I can link the installed openCV with python3 ??
Thanks for the help
Sorry I cannot give you specifics because your setup is not identical to mine, but I am sure we can get you sorted out.
Firstly, when you install packages, such as OpenCV, they tend to create a directory somewhere called lib which contains the C/C++ functions you can call from that package. Inside that directory, you normally find "shared object libraries" which traditionally end in "XXX.so" on macOS. More interestingly, they also contain a sub-directory called site-packages which contains the Python bindings (links). So, on my system, which is likely different from yours, I can find all those site-packages directories with:
find / -type d -name site-packages 2>/dev/null
Sample Output
/usr/local/lib/python3.7/site-packages
/usr/local/lib/python2.7/site-packages
...
...
/usr/local/Cellar/tbb/2018_U5/lib/python2.7/site-packages
/usr/local/Cellar/vips/8.6.5/lib/python3.7/site-packages
Hopefully, you can see that /usr/local/lib/python3.7/site-packages is looking a very likely candidate for where all the Python v3.7 bindings for OpenCV should be.
Good, so now we know how to find the Python bindings, we need to tell Python that information. How? Well, not unreasonably, Python looks at an environment variable called PYTHONPATH to find its stuff. So, using our skill and judgement we need to marry up what we found in the first step with what we now know from the second step. So we do:
export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.7/site-packages
And everything should work. All we need to do is put that in our login profile (probably $HOME/.profile) and we will be ready to go every time we log in.
I have a problem trying to import cv2 in python3 when I run it from my home folder...
luis#luis-Ubuntu:~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so: undefined symbol: PyCObject_Type
>>>
but when I run python3 inside /usr/lib/python3/dist-packages or /usr/local/lib/python3.5/dist-packages path it works fine...
luis#luis-Ubuntu:~$ cd /usr/lib/python3/dist-packages
luis#luis-Ubuntu:/usr/lib/python3/dist-packages$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>>
I know this is related to include a path on the sys library,
I added the export PYTHONPATH on my ~/.bashrc but didn't solve the problem...
export PYTHONPATH="/usr/local/lib/python3.5/dist-packages:$PYTHONPATH"
I also found that if I insert the path before importing cv2 it works but I need to do this on all scripts or every time I run python3 from a terminal...
luis#luis-Ubuntu:~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so: undefined symbol: PyCObject_Type
>>> import sys
>>> sys.path.insert(0,'/usr/lib/python3/dist-packages')
>>> import cv2
>>>
but I want to fix this permanently, does anyone know how to solve this...
btw... runs fine on python2...
From the print(sys.path) that you have provided,
...'/opt/ros/kinetic/lib/python2.7/dist-packages'...
I think the problem lies in this item. Although the python3 path also resides in sys.path, the python2.7 path precedes it. Python will catch the one in python2.7 first. When you are running in directly inside /python3/dist-packages, the current directory is placed first, and that precedes python2.7.
I think there are two ways:
Remove python2.7/dist-packages from your PYTHONPATH
Call sys.path.remove('/opt/ros/kinetic/lib/python2.7/dist-packages') before you import cv2. See this question.
The first approach should be the "proper" way but I am not sure how it is deal with if you want to use both python2/3 at the same time. You might be interested in this post.
You can add the following two lines before you import cv2 in your python code. It works for me without changing any source file:
import sys
sys.path.remove('/opt/ros/kinetic/lib/python2.7/dist-packages')
Then import cv2:
import cv2
The root of this problem are the ROS commands like source/opt/ros/kinetic/setup.bash in the bashrc file /home/username/.bashrc , which force changes in the Python path. Even if you are not using ROS, the commands are still executed and thus you are directed to find cv2 in /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so in the ROS folder, which is not compatible with Python 3. More discussion on the same issue can be seen at:
After install ROS Kinetic, cannot import OpenCV
. Here I propose another solution which is not mentioned by any answer at that post.
The idea is to run source/opt/ros/kinetic/setup.bash(and any other bash related to ROS) only when using ROS. This way you do not have to modify .bashrc file (or editing the PYTHONPATH like to what you do) every time.
First, remove ROS-related commands like source/opt/ros/kinetic/setup.bash from the basrch file and make sure that you can import cv2 in Python 3.x with no error now.
Then, create a environment and install all the ROS related packages here. By doing so, we will have to activate this ros_environment when running ROS. For creating environment, see
https://conda.io/docs/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands
Next, activate your newly-created environment and follow
https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux
to create files in the suggested path. Namely,
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh
Edit the ./etc/conda/activate.d/env_vars.sh as follows
source/opt/ros/kinetic/setup.bash
Also add in any other ROS-related bash in the file.
As for ./etc/conda/deactivate.d/env_vars.sh, what I do is exporting the PYTHONPATH back to Python 3. For example, it could be
export PYTHONPATH="/usr/lib/python3/dist-packages"
Note that this is not really deactivating the source/opt/ros/kinetic/setup.bash command. I just found doing this way would direct my PYTHONPATH back to the default python 3, and the correct cv path can be found even after activation and deactivation of the ROS environment .
While trying to make my scons based build system as platform independent as possible I was wondering about the following:
Why does python2 return the include path /usr/local/include/python2.7? This path does not contain Python.h and building fails if I rely on that path.
python2
Use sysconfig inside python2:
$ /usr/bin/python2
Python 2.7.13 (default, Nov 23 2017, 15:37:09)
[GCC 6.3.0 20170406] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/local/include/python2.7'
Gives /usr/local/include/python2.7. This is an empty folder.
Call python2-config from shell:
$ /usr/bin/python2-config --includes
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7
This gives a different path. I was able to find Python.h in /usr/include/python2.7.
python3
Use sysconfig inside python3:
$ /usr/bin/python3
Python 3.5.3 (default, Nov 23 2017, 11:34:05)
[GCC 6.3.0 20170406] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_path('include')
'/usr/include/python3.5m'
Call python3-config from shell:
/usr/bin/python3-config --includes
-I/usr/include/python3.5m -I/usr/include/python3.5m
The resulting path /usr/include/python3.5m is the same for boath approaches. Python.h is located here.
anaconda
If I use anaconda python (2 or 3) the paths are also consistent (as with python3).
I already found some workarounds such as placing a softlink to usr/include in usr/local/include or just throwing away the local/ in the path, but both do not really look like a nice solution.
edit:
Currently the incorrect include path in python2 makes my build system not as platform independend as I would like. Adding an (optional) environment variable PYTHON_INCLUDE_PATH gives me the possibility to define the correct path, if python2 is used. But a solution where the correct path is always returned would help me a lot (either on python side or using scons-specific functionality).
As my build system is based on scons a
Python uses an installation scheme that differs depending on the
platform and on the installation options.
https://docs.python.org/2/library/sysconfig.html#installation-paths
There are various schemes including posix_local and posix_prefix which determine the location of various installation directories. It seems that sysconfig does not actually record which scheme was used to install the particular build of Python -- it only has the build information.
So when you call sysconfig.get_path() it guesses the scheme based on the default for the current platform [1]. The Python2.7 sysconfig guesses posix_local while the Python3 sysconfig guesses posix_prefix [2].
It looks like both versions of Python are installed using the posix_prefix scheme, so you can specify that when you call sysconfig.get_path:
$ python -c "import sysconfig; print(sysconfig.get_path('include', 'posix_prefix'))"
/usr/include/python2.7
$ python3 -c "import sysconfig; print(sysconfig.get_path('include', 'posix_prefix'))"
/usr/include/python3.5m
[1] https://github.com/python/cpython/blob/2.7/Lib/sysconfig.py#L169
[2] Run sysconfig as a script:
$ python -m sysconfig | head
Platform: "linux-x86_64"
Python version: "2.7"
Current installation scheme: "posix_local"
$ python3 -m sysconfig | head
Platform: "linux-x86_64"
Python version: "3.5"
Current installation scheme: "posix_prefix"
I can't actually find posix_local in the sysconfig source so not sure where it comes from.
EDIT
I looked into this a bit more and learned that it specific to the Debian/Ubuntu versions of Python; upstream Python doesn't use /usr/local/ or have a posix_local scheme. The Debian packages use a kind of hybrid approach which is the same as posix_prefix with the addition of /usr/local/ for modules.
I haven't found a link the source online, but my local system has this in its Python2.7 sysconfig.py (notice the FIXME):
def _get_default_scheme():
if os.name == 'posix':
# the default scheme for posix on Debian/Ubuntu is posix_local
# FIXME: return dist-packages/posix_prefix only for
# is_default_prefix and 'PYTHONUSERBASE' not in os.environ and 'real_prefix' not in sys.__dict__
# is_default_prefix = not prefix or os.path.normpath(prefix) in ('/usr', '/usr/local')
return 'posix_local'
return os.name
The Debian python3 sysconfig.py does away with posix_local and uses the same default as upstream python:
def _get_default_scheme():
if os.name == 'posix':
# the default scheme for posix is posix_prefix
return 'posix_prefix'
return os.name
You might want to replicate that to be compatible with Mac or Windows:
sysconfig.get_path('include', 'posix_prefix' if os.name == 'posix' else os.name)
https://wiki.debian.org/Python#Deviations_from_upstream
https://www.debian.org/doc/packaging-manuals/python-policy/python.html#paths
I am running Ubuntu 16 and have both python 2 and 3. I have downloaded wxpython and it works with the python2 interpreter but not 3. I get
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
>>>
and
Python 3.5.2 (default, Sep 14 2017, 22:51:06)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'wx'
What do I need to do to get installed for python 3?
A similar situation arises on Fedora 25, on which I was able to solve this. Your mileage may vary on Ubuntu.
First note that wxPython is available in two major versions, let's call them wx3 and wx4. You can identify your running version through import wx; print(wx.version()). The version string on Fedora 25 reads '3.0.2.0 gtk3 (classic)', i.e. a brand of wx3. On sourceforge these versions are known as 'wxPython' and 'wxPython4', and wxpython.org calls wx4 'phoenix'.
Inspecting the source code of wx3 you will note that wx3's syntax is incompatible with python3. Conversely, wx4 is compatible both with python2.7 and python3.
wx4 doesn't seem to be available on Fedora 25, therefore python3 can't run any wx out-of-the-box. Ubuntu may or may not have the same issue.
The following worked for me to install wx4 in a python3 virtual environment:
pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/fedora-26 wxPython
I presume the answer to your question would be
pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-16.04 wxPython
In case you need to port a (py2, wx3) application to python3, you would be wise creating an intermediate step: (py2, wx3) -> (py2, wx4) -> (py3, wx4), noting that (py3, wx3) is an impossibility.
To create the (py2, wx4) environment was more cumbersome for me, because the above pip install command fails to find header files when run using pip2.
What ended up working for me was to download the 4.0.0b2 source https://pypi.python.org/packages/bc/6f/f7bb525517557e1c596bf22ef3f242b87afaeab57c9ad460cb94b3b0714e/wxPython-4.0.0b2.tar.gz#md5=2e3716205da8f52d8039095d14534bf7
and follow the build instructions https://github.com/wxWidgets/Phoenix/blob/master/README.rst , from which I only used the build command python build.py dox etg --nodoc sip build .
After building, you need to tell your python2 where to find the wx4 library. I ended up doing that by creating a virtualenv, and creating a symbolic link like so:
/home/user/venv/lib/python2.7/site-packages/wx -> /home/user/downloads/wxPython/wxPython-4.0.0b2/wx
That latter directory cointaining the result of the build.
I noticed my jupyter notebooks started having problems; I discovered there was a problem with duplicate IPython in my path.
Doing some research, it looks like up until today my dist-packages was:
/usr/local/lib/python3.5/dist-packages/
now, it's:
/usr/lib/python3/dist-packages/
a little more research shows that my path got messed up, and there's an egg file on the top:
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print("\n".join(sys.path))
/usr/local/lib/python3.5/dist-packages/buildozer-0.34.dev0-py3.5.egg
/usr/lib/python3/dist-packages
/usr/lib/python35.zip
/usr/lib/python3.5
/usr/lib/python3.5/plat-x86_64-linux-gnu
/usr/lib/python3.5/lib-dynload
/home/veggiebenz/.local/lib/python3.5/site-packages
/usr/local/lib/python3.5/dist-packages
I'd like to find out where this is getting added. There's no PYTHONPATH environment variable.
Also, the final entry (/usr/local/lib/python3.5/dist-packages) I would like to get it back to being the first entry.
Well I figured it out, so I'm posting this here in case someone else has the same issue.
Turns out there was a .pth file that was prepending these items to sys.path.
From python docs:
A path configuration file is a file whose name has the form name.pth
and exists in one of the four directories mentioned above; its
contents are additional items (one per line) to be added to sys.path
In any event, here's what I did to fix it.
sudo locate easy-install.pth
Found it here:
/usr/local/lib/python3.5/dist-packages/
I opened this file in text editor and commented out the lines referring to the egg file and to the incorrect dist-packages folder. Then a reboot to give everything a fresh start. Now my python environment is happy and my jupyter notebooks are working correctly.