I am trying to use a local instance of readthedocs to build documentation for a project written in python 3. (I already have successfully built documentation for one python 2 project).
When I select the option to set The Python interpreter used to create the virtual environment to CPython3.X in the advanced settings, then the build fails, with the command
python3.5 -mvirtualenv --no-site-packages --no-download /home/user/rtd/checkouts/readthedocs.org/user_builds/project/envs/latest
producing the error /usr/bin/python3.5: No module named virtualenv.
However pip3 install virtualenv yeilds Requirement already satisfied: virtualenv in /usr/local/lib/python2.7/dist-packages. I've tried switching to using conda but am having problems with that. Is there anyway to use virtualenv with the global python 3 installation?
I know from the installation instructions that
you’ll need to install Python 3 with virtualenv in your system as well.
However it is not clear to me what they are saying there. Could someone clarify?
Related
I have a 2-part question about conda vs. pip virtual environments. I found great information on the answers What is the difference between pip and conda? and Does Conda replace the need for virtualenv? but still have something unclear.
I have a given python project (say PR) that I need to install and further develop on a linux server (say S) where python is installed with anaconda. Now, the usage/installation instructions of PR tell me to use python to create virtual environment and pip to install all packages. That is,
python3 -m venv PR
pip install --editable . (the dot included at the end)
According to "pip install --editable ./" vs "python setup.py develop" the latter reads the file setup.py (included in PR) which contains a function setup(...) with option install_requires listing all the required packages and installs them automatically. I have tested this on my own computer (which does not have conda) and it works fine. At least no error messages.
Now I need to further develop PR on S. My question Part 1: can I use conda instead of pip to create and update virtual environment? If yes, what would be the conda command replacing pip install --editable . ? I'm positive I will later need to install other packages as well. I'm worried about conflicts between conda/pip.
On S, I have Spyder and no other python IDEs. I have never used Spyder but I'm very familiar with PyCharm (Windows) and VS Code (Linux) so I assume debugging with Spyder will be similar to those. My question Part 2 (tied to Part 1): if I have to use pip to install packages, does Spyder see those? Or can it only see conda-installed packages?
(Edit/update): Thank you Carlos for comments. I continue my question:
I created and activated the virtual environment (VE) with conda
conda create PR_venv
conda activate PR_venv
Installed pip with
conda install pip
(this upgraded pip and installed several other packages too, including newer version of python). Installed PR and its required packages with pip
pip install -e .
Now, if I run the PR package inside this active VE interactively from the terminal, everything works fine. I would like to do the same from within spyder, to get the IDE debugging abilities in my hand.
When I start spyder, open a python file to be run, click "Run" button, it crashes in the import statements.
Spyder cannot see the installed packages. It can see only the local package PR but none of the packages installed by pip for this VE.
I am not sure what is the correct question here; I'm confused how are conda VEs related to spyder/jupyter/ipython ? I cannot find information in the conda documents about this.
I cannot find from spyder documents anything about VEs. Do I have to somehow re-install the packages (how?) inside Spyder? It seems pointless because the packages are installed already.
(Edit/Update 2): The information on https://docs.spyder-ide.org/current/installation.html makes me even more confused: Spyder is presented as both a stand-alone program and as a python package. So do I have to re-install Spyder inside the VE(?!) with
conda activate PR_venv
conda install spyder
Any clarification would be appreciated. I have always thought that the IDEs are stand-alone programs and that's it. This Spyder setup twists my brains into pretzel.
(Spyder maintainer here) About your questions:
can I use conda instead of pip to create and update virtual environment?
Yes, you can. Please see here to learn about the functionality offered by conda for managing environments.
If yes, what would be the conda command replacing pip install --editable . ?
Conda doesn't offer a good replacement for that command. However, you can still use it in a conda environment, as long as all you've installed all your package dependencies with conda before running it. That would avoid mixing conda and pip packages, which usually leads to really bad results.
if I have to use pip to install packages, does Spyder see those? Or can it only see conda-installed packages?
Spyder can work with pip and conda packages without problems. Just make sure of not mixing them (as I said above) and you'll be fine. In addition, please read our documentation to learn how to connect a local Spyder instance to a remote server.
Part 1: yes I can use conda to create VE and pip to install packages
conda create PR_venv
conda activate PR_venv
conda install pip
pip install --editable .
conda list
The last line shows which packages are installed by conda and which by pip (shown as pypi)
Part 2: spyder by default cannot see the packages. Need to do two things:
conda install spyder-kernels
Open Spyder and Tools > Preferences > Python Interpreter > Use the following interpreter > [full path to VE python command]
Restart Spyder. Now it can see the packages.
(Edit:) this link is great: https://github.com/spyder-ide/spyder/wiki/Working-with-packages-and-environments-in-Spyder
EDIT:
In addition to the behaviour outlined below, the Python3.10 based environment seems to be ignoring packages installed with the pip -e option (development mode).
Specifically, a package installed in development mode is not listed in pip freeze (which, it does in the Python3.9 based virtual environment) and a simple import <package_name> fails. If the package is installed normally (i.e. not in development mode) everything works as expected.
I am using virtualenv to create a virtual environment based around Python 3.10. While virtualenv finishes without errors and does seem to activate, it still fails to pick up its own Python unless the PYTHONPATH environment variable has been set manually.
I am not sure if the situation I am faced with is due to Ubuntu's way of incorporating Python or the way virtualenv is setting up the environment so that it picks up a local interpreter. Here is what I have gathered this far:
My base system is an Ubuntu 21.04. It has its own Python 3 (Python3.9.5) installation which I have not touched at all except installing the python3-virtualenv package using apt.
With Python 3.10, I installed the python3.10-dev package and proceeded to create a virtual environment in the usual way:
> virtualenv -p python3.10 the_env/
> source the_env/bin/activate
Although this looks OK so far, this environment does not have any information about its own site-packages directory. Not even the one that virtualenv is supposed to be creating which includes pip. In this installation, if you try to > pip --version you simply get an error that
the pip package does not exist (the pip "executable" location is picked up correctly, but because the interpreter does not know anything about its site-packages it fails to start pip properly).
Long story short, I created two environments, one based on Python3.9 (which works perfectly) and one based on Python3.10 (which does not work) and did a very simple test in each environment:
> python -m site
On the Python3.9 environment, sys.path includes a path that leads all the way to this particular environment's site-packages
On the Python3.10 environment, sys.path does not include that particular path but still includes the typical paths you expect to find (e.g. those pointing to the interpreter itself and the top environment directory but not the specific path that points to the site-packages location.
Following this, I defined a PYTHONPATH manually, before activating the environment which points exactly to the site-packages for that particular environment and everything worked as expected.
I suspect that this might be something to do with the fact that my system's Python is 3.9 which means that the USER_SITE variable is valid while in the case of Python3.10, it is not (because, I do not have a use for it, this is just a virtual environment I am creating). So, I suspect that this might be throwing off the way the site module determines where things are.
As I am not sure, I would like to ask the following:
Could this be something to do with the way Ubuntu handles the Python installation that might just be creating this small problem with virtual environments?
Could the problem be with virtualenv that does not explicitly specify a PYTHONPATH?
Could this behaviour be something of a corner case of the site module?
What worked for me is an installation from source. After unpacking the source code, as a summary:
$ ./configure --enable-optimizations --with-ensurepip=install --prefix=/path/to/install/to/
$ make -j
$ make test
$ make install
$ /path/to/install/to/bin/python3.10 -m venv /path/to/test
$ source /path/to/test/bin/activate
$ pip list
Package Version
---------- -------
pip 21.2.4
setuptools 58.1.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/path/to/test/bin/python3.10 -m pip install --upgrade pip' command.
$ python -m pip install --upgrade pip
[...]
$ pip list
Package Version
---------- -------
pip 21.3.1
setuptools 58.1.0
Edit
As mentioned below, I believe the reason why the procedure above just works is simply that when building and installing Python from source, the installation simply will be correct, which I suspect the 3.10 installation in Ubuntu 21.04 is not.
I made my Python installation from source somewhere under my home directory, in order to not risk messing things up, and I also did not permanently modify the PATH.
Setting the path to $PATH:/path/to/install/to/bin should be fine however, either permanently, or just when running mkvirtualenv.
Doing that, my new installation even seems to integrate seamlessly with my system's virtualenvwrapper.
I don't really need to be on the Python leading edge myself, but if I did, I would definitely make myself independent of Ubuntu's latest development by using the procedure described above.
By the way, if I update pip directly in the Python installation built from source, I will no longer get the message about the older pip (see above) when I install new virtual environments.
Edit 2
Bug reported to Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1955742
Edit 3
Also, since Ubuntu 21.04 has end of life in a month or so, upgrading to 21.10 could really make sense. I have just tried that, and it seems that Python 3.10 virtual environments work just fine in that release.
With the advent of Ubuntu 22.04 (which caused a few minor issues with some specific python virtualenv setups I had) and having already spent some time figureing this out, I ended up switching to using pyenv.
Pyenv made it very easy to install any python version and any number of virtual environments within it which can be further customised in complete isolation.
The only thing to be careful of is installing all necessary python prerequisites before installing a specific version to avoid missing functionality from some packages (e.g. not including the lzma library will generate an ominous warning from pandas ("your python installation is incomplete..."). This can be ignored if you are not using that functionality or otherwise, easily fixed.
I think that this is a better option overall if you have to manage different versions and specific configurations for python, even across distros.
This solution is very close to the one suggested before ("install from source") so I will be accepting that one and leave my contribution as additional information about this problem.
For a project I made a virtual environment (venv) using Python3. I installed all the necessary dependencies using a simple bash script (see picture below) after I activated my venv. (I verified the installed packages using: pip3 list and concluded that every dependency was installed succesfully.)
My project uses snakemake, so I ran this snakemake commando:
snakemake --snakefile Snakefile.py all
I get this error:
I know it has to do something with the venv, because without the venv snakemake runs perfectly. I have read the Snakemake installation documents and it says I have to install conda and make & activate a conda venv. But, I do not have the sudo privileges to download and install conda (I work on a protected server).
What is happening and does someone know a fix?
One possible reason could be the difference in Python versions. What version of Python does the pip3 prepare environment for?
As I can see from the picture provided, the invalid syntax may be because of the version of Python doesn't support f-strings.
Imagine the following two scenarios: when you run Snakemake manually, you use the latest Python3 (e.g. 3.9). But if the pip3 is configured for an older version (e.g. 3.5), you can configure a very different environment for Python3.5 that doesn't support f-strings.
Question
I am not sure why when virtualenv active, pip freeze stills showing all modules when it suppose NOT to. What am I doing wrong?
On the terminal
I tried two ways to create a virtual env:
virtualenv my-virtualenv
virtualenv --no-site-packages my-virtualenv
then activate it
source my-virtualenv/bin/activate
Results
Both show all packages (when they suppose not to). Running command pip freeze I get:
(my-virtualenv)$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
actionlib==1.11.9
angles==1.9.11
bondpy==1.7.19
camera-calibration==1.12.20
camera-calibration-parsers==1.11.12
catkin==0.7.6
cv-bridge==1.12.4
diagnostic-analysis==1.9.2
diagnostic-common-diagnostics==1.9.2
diagnostic-updater==1.9.2
dynamic-reconfigure==1.5.48
gazebo-plugins==2.5.13
gazebo-ros==2.5.13
...
rosnode==1.12.7
rosparam==1.12.7
rospy==1.12.7
rosservice==1.12.7
rostest==1.12.7
rostopic==1.12.7
rosunit==1.13.5
roswtf==1.12.7
tf2-ros==0.5.16
topic-tools==1.12.7
xacro==1.11.2
Running
(my-virtualenv)$ which python
/home/user/.../my-virtualenv/bin/python
(my-virtualenv)$ python -V
Python 2.7.12
and
(my-virtualenv)$ which python3
/usr/bin/python3
(my-virtualenv)$ python3 -V
Python 3.5.2
Hence, I even can use python3 when virtualenv is activated but it uses the pc installed module. Moreover, new installed packages are install in the pc and not the virtualenv and I can't install python3 in the vitualenv because it "exists" already (but in the pc).
I was having the same issue and it was because, somehow, Python configuration of ROS was generating it.
I solved it removing the source commands of ROS from the ~/.bashrc. E.g.:
source /opt/ros/melodic/setup.bash
source $HOME/ROS/aslam_ws/devel/setup.bash
By the way, I realised that, before changing anything, creating the environment from PyCharm works properly too (when in the terminal I still had the issue).
There must be a better solution but I haven't found it yet.
Some things you should know before I ask my question:
I am utterly new to both Linux & Python, and have a hard time understanding official documentation and technical answers (but have a burning desire to deeply understand both)
I am running elementary OS 0.4.1 Loki
My Python 3 version is 3.5.2. When I search the online documentation on the venv module for python 3.5.2, I get the documentation for the 3.5.6 version. I don't understand why there is no documentation for the .2 version.
So, here's my problem. I was trying to create a virtual environment using venv and proceeded thusly:
According to Python's 3.5.6 venv module documentation, a virtual environment is created using the command pyvenv /path/to/new/virtual/environment. I tried that command and got:
The program 'pyvenv' is currently not installed. You can install it by typing: sudo apt install python3-venv
I then searched documentation for newer Python versions and tried the new venv command python3 -m venv /path/to/new/virtual/environment and got the following result:
The virtual environment was not created successfully because ensurepip is not available. On Debian/Ubuntu systems, you need to install the python3-venv package using the following command. apt-get install python3-venv
In both cases, the solution seems to be to install python3-venv. My question is: What exactly am I installing by installing python3-venv: Isn't venv already part of the Standard Library? Furthermore, why do I have to install it via apt-get if it is a Python module? It is my understanding that standard library modules are imported, not installed; and that modules external to the standard library are installed via pip. Related to this, why is ensurepip not available?
Second part of my question: if installing python3-venv is the way to go, what is the proper way to create a virtual environment using venv in Python 3.5.2: pyvenv my_virtual_environment or python3 -m venv my_virtual_environment?
Don't worry about the documentation not matching the micro version number – increments in that place are only for bugfixes, so the documentation stays the same.
Your question is interesting since venv is indeed not an optional module. My guess is that the Python version shipped with your OS (or that you installed yourself) seems to come with a stripped down or no standard library. For instance, the python3.5-minimal package doesn't appear to have it. Does your Python have the other modules in the standard library?
Edit: See also this question.
Installation can be described as "putting files onto your computer, in the right place". Importing a module, however, means that you tell Python to make available some functionality. To import a module, it must be installed (e.g. in /usr/lib/python3.5 for Python 3 on my computer), and one method for installing additional modules is via apt.
The python3 -m venv my_virtual_environment method should work in 3.5 as well and is the future-proof version, so you should probably go with that.