ImportError: No module named numpy in Debian OS - python-3.x

In my virtual environment on my debian machine, I am able to import numpy at the prompt But when I call it in a program it errors out.
I am looking over the net and they say to uninstall and re-install. I tried but novail.
Also, I am able to import it in python prompt but not when I call the script? Please help me out of this.
I am able to call successfully at the prompt:-
(venv) root#c3-redsuren-vm01:~/my-project# python
Python 3.8.2 (default, Apr 13 2020, 08:44:45)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> import numpy
>>> exit()
Here when I try to execute my program it errors out:-
(venv) root#c3-redsuren-vm01:~/my-project# ./test_nump.py
Traceback (most recent call last):
File "./test_nump.py", line 2, in <module>
import numpy
ImportError: No module named numpy
(venv) root#c3-redsuren-vm01:~/my-project# cat test_nump.py
#!/usr/bin/python
import numpy
print("test")
(venv) root#c3-redsuren-vm01:~/my-project#

As per request from the OP.
For any executable file, bash reads the first line with the shebang or #!... to extract the interpreter for the script. If it does not find the shebang, it will try to service the file type using the magic numbers and then use the relevant loading method. That's why running an ELF binary does not require you to specify the executor.
Now, virtual environments works by overriding your bash search path, i.e. when you type a command, where should it look. For example, if you have a binary name ls where should it find it to load from. It refers to your search path or the $PATH variable and scans the directories sequentially and stops at the first match. Virtual environment will prepend it's path to the $PATH when you do a source .venv. The source command simply tells your bash session to apply the bash directives from the .venv file.
In your source file, you have supplied the interpreter using the shebang directive to the global installation of python which does not have numpy in it's module list so it fails.
If you run it using python script.py, because the $PATH is overridden by virtual env, the shell will find the python version of virtual env first than the global one which has your numpy module.
Why does /usr/bin/env python work ? Because it's going to refer to your $PATH in your environment variable (which is modified by virtual env) and will find the correct python installation.

Related

python3 cannot find a module that I could import with python2

I'm trying to switch from python2 to python3. In the process I'm also switching from anaconda to miniconda3 as my primary package management tool. There are some other packages that I clone from github. I found that I can no longer import any modules from packages downloaded from github rather than conda. For example, import linetools.utils used to work and no longer works. import linetools.linetools.utils works but this is not enough, since various modules inside the package reference each other. This package is not written by me so changing all occurrences of linetools.utils to linetools.linetools.utils is not desirable.
Here are additional information for you to help solve my problem.
My $PYTHONPATH environment variable is set to /Users/lwymarie/python/. I also tried putting this same path to my $path variable as well. Didn't solve my problem.
Here are the packages I have under PYTHONPATH. Just showing a few.
flemish.local> ls $PYTHONPATH
sdsspy Barak desisim linetools desiutil pymc3 specutils PypeIt pyqtgraph statsmodels
RemoteObserving fitsio Ska.Numpy ginga redrock
Here is the expected behavior, when I used python2 and anaconda2. I'm using the package linetools as an example.
flemish.local> python
Python 2.7.12 |Anaconda custom (x86_64)| (default, Jul 2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import linetools.utils
Here's the sys.path when the above worked. Just showing a few of the elements.
>>> import sys
>>> sys.path
['', '/Users/lwymarie/python', '/Users/lwymarie/anaconda/lib/python27.zip', '/Users/lwymarie/anaconda/lib/python2.7', '/Users/lwymarie/anaconda/lib/python2.7/plat-darwin', '/Users/lwymarie/anaconda/lib/python2.7/plat-mac', '/Users/lwymarie/anaconda/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/lwymarie/anaconda/lib/python2.7/lib-tk', '/Users/lwymarie/anaconda/lib/python2.7/lib-old', '/Users/lwymarie/anaconda/lib/python2.7/lib-dynload', '/Users/lwymarie/anaconda/lib/python2.7/site-packages', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/PIL', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/aeosa', '/Users/lwymarie/python/specutils', '/Users/lwymarie/python/ginga', '/Users/lwymarie/python/statsmodels', '/Users/lwymarie/python/Ska.Numpy', '/Users/lwymarie/python/linetools', '/Users/lwymarie/python/pymc3', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/numpydoc-0.6.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/nbsphinx-0.2.9-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/recommonmark-0.4.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/CommonMark-0.5.4-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/joblib-0.10.2-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/Theano-0.8.2-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/ginga-2.7.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/QtPy-1.3.1-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/desiutil-1.9.9.dev596-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/desisim-0.24.0.dev1308-py2.7.egg', '/Users/lwymarie/python/redrock/py', '/Users/lwymarie/python/PypeIt', '/Users/lwymarie/anaconda/lib/python2.7/site-packages']
Exiting Python, here's the $path variable when the above worked.
flemish.local> echo $path
/Users/lwymarie/anaconda/bin . /usr/local/bin /usr/local/etc /opt/local/bin /opt/local/sbin /usr/sbin /sbin /usr/bin /bin /usr/lang /etc /usr/etc /usr/X11/bin /usr/local/scisoft/bin /usr/local/texlive/2020/bin/x86_64-darwin /Library/Ruby/Gems/2.0.0 ./py/ ./
Here is the unwanted behavior, when I used python3 and miniconda3.
flemish.local> python
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import linetools.utils
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'linetools.utils'
Here's the sys.path when the above error happened. I also tried the sys.path.append("/Users/lwymarie/python/linetools/") trick and it still couldn't find the module.
>>> import sys
>>> sys.path
['', '/Users/lwymarie/python', '/Users/lwymarie/miniconda3/lib/python37.zip', '/Users/lwymarie/miniconda3/lib/python3.7', '/Users/lwymarie/miniconda3/lib/python3.7/lib-dynload', '/Users/lwymarie/miniconda3/lib/python3.7/site-packages']
Exiting Python, here's the $path variable when the above error happened.
flemish.local> echo $path
/Users/lwymarie/miniconda3/bin /Users/lwymarie/miniconda3/bin . /usr/local/bin /usr/local/etc /opt/local/bin /opt/local/sbin /usr/sbin /sbin /usr/bin /bin /usr/lang /etc /usr/etc /usr/X11/bin /usr/local/scisoft/bin /usr/local/texlive/2020/bin/x86_64-darwin /Library/Ruby/Gems/2.0.0 ./py/ ./
Here's an image of the directory tree of the linetools package.
If you successfully help me get python3 and linetools working, your good deed will be rewarded with eternal life and happiness.
There is '/Users/lwymarie/python/linetools' in your python2 sys.path but not in your python3 sys.path. Did you re-setup linetools using python3?
cd ~/python/linetools
python setup.py develop
I think what Steven Lau suggested should work.
But keep in mind that you have multiple index paths. You have /Users/lwymarie/python and /Users/lwymarie/python/linetools (I see this for your py2 setup).

How to link openCv with Python3 in mac Os

I have two versions of python in my mac os, the first python2.7 which is the default that came with the system. Later I installed python3.7 that I use most of the time.
I have recently installed openCV using homebrew.
When I'm using openCV with python2.7, it's working normally.
But the problem is when I try to use it with python3. Importing cv2 in python3 gives error: ModuleNotFoundError: No module named 'cv2'
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "copyright", "credits" or "license()" for more information.
>>> WARNING: The version of Tcl/Tk (8.5.9) in use may be unstable.
Visit http://www.python.org/download/mac/tcltk/ for current
information.
>>> import cv2
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
import cv2
ModuleNotFoundError: No module named 'cv2'
>>>
Is there anything I can do so that I can link the installed openCV with python3 ??
Thanks for the help
Sorry I cannot give you specifics because your setup is not identical to mine, but I am sure we can get you sorted out.
Firstly, when you install packages, such as OpenCV, they tend to create a directory somewhere called lib which contains the C/C++ functions you can call from that package. Inside that directory, you normally find "shared object libraries" which traditionally end in "XXX.so" on macOS. More interestingly, they also contain a sub-directory called site-packages which contains the Python bindings (links). So, on my system, which is likely different from yours, I can find all those site-packages directories with:
find / -type d -name site-packages 2>/dev/null
Sample Output
/usr/local/lib/python3.7/site-packages
/usr/local/lib/python2.7/site-packages
...
...
/usr/local/Cellar/tbb/2018_U5/lib/python2.7/site-packages
/usr/local/Cellar/vips/8.6.5/lib/python3.7/site-packages
Hopefully, you can see that /usr/local/lib/python3.7/site-packages is looking a very likely candidate for where all the Python v3.7 bindings for OpenCV should be.
Good, so now we know how to find the Python bindings, we need to tell Python that information. How? Well, not unreasonably, Python looks at an environment variable called PYTHONPATH to find its stuff. So, using our skill and judgement we need to marry up what we found in the first step with what we now know from the second step. So we do:
export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.7/site-packages
And everything should work. All we need to do is put that in our login profile (probably $HOME/.profile) and we will be ready to go every time we log in.

how do I correctly use Path (PyCharm)

I'm using PyCharm 2018.1.4
If I write
from pathlib import Path
p = Path('.')
this runs fine.
On the other hand, if I write
import pathlib
p = Path('.')
I get
NameError: name 'Path' is not defined
I thought by using import pathlib I'm importing the complete library, including Path.
Compared to a terminal session:
$ bpython
bpython version 0.17.1 on top of Python 3.6.4 /Users/fanta4/anaconda3/bin/python
>>> import pathlib
>>> p = Path('.')
>>>
no problem.
And just python:
Nick-iMac:~ fanta4$ which python
/Users/fanta4/anaconda3/bin/python
Nick-iMac:~ fanta4$ python
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
>>> import pathlib
>>> p = Path('.')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'Path' is not defined
Where is the problem in PyCharm?
In PyCharm I see python 3.6 (File > Default Settings > Project Interpreter)
Thanks!
If you do a naked import (i.e. import pathlib), the Path class is not in the local namespace of your script. It is an attribute of the module object pathlib. To successfully access Path in this case, you must explicitly refer to it via its parent object. I.e.: pathlib.Path.
Also, I'm not familiar with bpython, but what you've described in your terminal session does not occur in IPython. In fact, I consider it extremely poor design that importing a library in any interactive environment implicitly imports all of its children objects. Number one, it risks polluting the namespace. Number two, it causes confusion in both new and old users of a language.

import cv2 on python3

I have a problem trying to import cv2 in python3 when I run it from my home folder...
luis#luis-Ubuntu:~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so: undefined symbol: PyCObject_Type
>>>
but when I run python3 inside /usr/lib/python3/dist-packages or /usr/local/lib/python3.5/dist-packages path it works fine...
luis#luis-Ubuntu:~$ cd /usr/lib/python3/dist-packages
luis#luis-Ubuntu:/usr/lib/python3/dist-packages$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>>
I know this is related to include a path on the sys library,
I added the export PYTHONPATH on my ~/.bashrc but didn't solve the problem...
export PYTHONPATH="/usr/local/lib/python3.5/dist-package‌​s:$PYTHONPATH"
I also found that if I insert the path before importing cv2 it works but I need to do this on all scripts or every time I run python3 from a terminal...
luis#luis-Ubuntu:~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so: undefined symbol: PyCObject_Type
>>> import sys
>>> sys.path.insert(0,'/usr/lib/python3/dist-packages')
>>> import cv2
>>>
but I want to fix this permanently, does anyone know how to solve this...
btw... runs fine on python2...
From the print(sys.path) that you have provided,
...'/opt/ros/kinetic/lib/python2.7/dist-packages'...
I think the problem lies in this item. Although the python3 path also resides in sys.path, the python2.7 path precedes it. Python will catch the one in python2.7 first. When you are running in directly inside /python3/dist-packages, the current directory is placed first, and that precedes python2.7.
I think there are two ways:
Remove python2.7/dist-packages from your PYTHONPATH
Call sys.path.remove('/opt/ros/kinetic/lib/python2.7/dist-packages') before you import cv2. See this question.
The first approach should be the "proper" way but I am not sure how it is deal with if you want to use both python2/3 at the same time. You might be interested in this post.
You can add the following two lines before you import cv2 in your python code. It works for me without changing any source file:
import sys
sys.path.remove('/opt/ros/kinetic/lib/python2.7/dist-packages')
Then import cv2:
import cv2
The root of this problem are the ROS commands like source/opt/ros/kinetic/setup.bash in the bashrc file /home/username/.bashrc , which force changes in the Python path. Even if you are not using ROS, the commands are still executed and thus you are directed to find cv2 in /opt/ros/kinetic/lib/python2.7/dist-packages/cv2.so in the ROS folder, which is not compatible with Python 3. More discussion on the same issue can be seen at:
After install ROS Kinetic, cannot import OpenCV
. Here I propose another solution which is not mentioned by any answer at that post.
The idea is to run source/opt/ros/kinetic/setup.bash(and any other bash related to ROS) only when using ROS. This way you do not have to modify .bashrc file (or editing the PYTHONPATH like to what you do) every time.
First, remove ROS-related commands like source/opt/ros/kinetic/setup.bash from the basrch file and make sure that you can import cv2 in Python 3.x with no error now.
Then, create a environment and install all the ROS related packages here. By doing so, we will have to activate this ros_environment when running ROS. For creating environment, see
https://conda.io/docs/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands
Next, activate your newly-created environment and follow
https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux
to create files in the suggested path. Namely,
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh
Edit the ./etc/conda/activate.d/env_vars.sh as follows
source/opt/ros/kinetic/setup.bash
Also add in any other ROS-related bash in the file.
As for ./etc/conda/deactivate.d/env_vars.sh, what I do is exporting the PYTHONPATH back to Python 3. For example, it could be
export PYTHONPATH="/usr/lib/python3/dist-packages"
Note that this is not really deactivating the source/opt/ros/kinetic/setup.bash command. I just found doing this way would direct my PYTHONPATH back to the default python 3, and the correct cv path can be found even after activation and deactivation of the ROS environment .

Command line application shoogle requires python3 but Ubuntu 16 defaults to python 2.7

The command line application shoogle (https://github.com/tokland/shoogle) which exposes google api services at a terminal command line requires python3 but the ubuntu 16 default python is 2.7.
I have tried alias and calling the shoogle app from subprocess in a python3 shell but (of course) the os still provides the default. I have been reluctant to make system wide changes to .bashrc or PYTHONPATH e.g. as so many other resource expect 2.7. But I am currently using this on a virtual machine so if it does break I can recover. That seems the only option but impractical in a production environment.
I've found very little shoogle help online (the author suggests SO etc. for support) so if any one has any experience with shoogle or suggestions to get the requiured python version I'd be happy to hear.
Running shoogle from a python3 interpreter finds the 2.7 files:
3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609]
Python Type "help", "copyright", "credits" or "license" for more information.
from subprocess import call
call (['shoogle', 'show'])
Traceback (most recent call last):
File "/usr/local/bin/shoogle", line 11, in <module>
import shoogle
File "/usr/local/lib/python2.7/dist-packages/shoogle/__init__.py", line 5,
in <module> from .shoogle import *
File "/usr/local/lib/python2.7/dist-packages/shoogle/shoogle.py", line 14, in <module>
from . import commands
File "/usr/local/lib/python2.7/dist-packages/shoogle/commands/__init__.py", line 2, in <module>
from . import execute
Using #Surest Texans suggestions I uninstalled the shoogle app and used pip3 install to reinstall. Now the application works as expected when called from the command line.

Resources