I needed python3.7 for a project and it wasn't available as a ubuntu package, so I bootstrapped a python3.7 environment using conda. From that 'grandparent' environment, I created a 'parent' python3 virtual environment for the specific project.
From the parent environment in my project, I'm running a command line tool from the google-cloud-sdk which creates another test environment. That script creates a 'grandchild' environment by calling something equivalent to (parent) $ python3 -m venv /tmp/grandchild.
That grandchild environment doesn't have the pip binary installed, for some reason or another. And this is the problem. This missing pip causes the google script to fail to install dev-test dependencies. The parent, and the child do have pip installed, however, but pip doesn't get passed down.
When I take conda out of the picture, and rely only on the python that ships with my ubuntu package system (under /usr/lib), I can nest my virtual environments ad nauseam, and pip always seems to get inherited properly. I think this is something specific to python/pip conda environments that I'm tripping on.
I think this is a separate cause problem to this: pip missing from Python venv
(I don't have that file ~/.pydistutils.cfg anywhere on my box)
UPDATE:
I've found a way to reliably repro this, and without the need for conda. This happens when the parent is created with virtualenv, i.e. virtualenv parent, and the child gets created from that parent using -m venv, i.e. (parent) $ python3 -m venv child. The child then doesn't get pip copied into it.
Nesting environments created with virtualenv works fine, and nesting environments created with venv works well too, but not when venv is used within a virtualenv-created environnment. They don't mix.
Note: environments don't really "nest" per-se, they are independent copies. I mean that one is created from the other.
One workaround is to create all the environments with the same method.
e.g. use python3 -m venv for all environments along the chain.
python3 -m venv parent; source parent/bin/activate
(parent) $ python3 -m venv child
If that seems to tangle you into a web of symlinks, you may also provide the --copies flag: python3 -m venv --copies …, which can avoid that.
Related
EDIT:
In addition to the behaviour outlined below, the Python3.10 based environment seems to be ignoring packages installed with the pip -e option (development mode).
Specifically, a package installed in development mode is not listed in pip freeze (which, it does in the Python3.9 based virtual environment) and a simple import <package_name> fails. If the package is installed normally (i.e. not in development mode) everything works as expected.
I am using virtualenv to create a virtual environment based around Python 3.10. While virtualenv finishes without errors and does seem to activate, it still fails to pick up its own Python unless the PYTHONPATH environment variable has been set manually.
I am not sure if the situation I am faced with is due to Ubuntu's way of incorporating Python or the way virtualenv is setting up the environment so that it picks up a local interpreter. Here is what I have gathered this far:
My base system is an Ubuntu 21.04. It has its own Python 3 (Python3.9.5) installation which I have not touched at all except installing the python3-virtualenv package using apt.
With Python 3.10, I installed the python3.10-dev package and proceeded to create a virtual environment in the usual way:
> virtualenv -p python3.10 the_env/
> source the_env/bin/activate
Although this looks OK so far, this environment does not have any information about its own site-packages directory. Not even the one that virtualenv is supposed to be creating which includes pip. In this installation, if you try to > pip --version you simply get an error that
the pip package does not exist (the pip "executable" location is picked up correctly, but because the interpreter does not know anything about its site-packages it fails to start pip properly).
Long story short, I created two environments, one based on Python3.9 (which works perfectly) and one based on Python3.10 (which does not work) and did a very simple test in each environment:
> python -m site
On the Python3.9 environment, sys.path includes a path that leads all the way to this particular environment's site-packages
On the Python3.10 environment, sys.path does not include that particular path but still includes the typical paths you expect to find (e.g. those pointing to the interpreter itself and the top environment directory but not the specific path that points to the site-packages location.
Following this, I defined a PYTHONPATH manually, before activating the environment which points exactly to the site-packages for that particular environment and everything worked as expected.
I suspect that this might be something to do with the fact that my system's Python is 3.9 which means that the USER_SITE variable is valid while in the case of Python3.10, it is not (because, I do not have a use for it, this is just a virtual environment I am creating). So, I suspect that this might be throwing off the way the site module determines where things are.
As I am not sure, I would like to ask the following:
Could this be something to do with the way Ubuntu handles the Python installation that might just be creating this small problem with virtual environments?
Could the problem be with virtualenv that does not explicitly specify a PYTHONPATH?
Could this behaviour be something of a corner case of the site module?
What worked for me is an installation from source. After unpacking the source code, as a summary:
$ ./configure --enable-optimizations --with-ensurepip=install --prefix=/path/to/install/to/
$ make -j
$ make test
$ make install
$ /path/to/install/to/bin/python3.10 -m venv /path/to/test
$ source /path/to/test/bin/activate
$ pip list
Package Version
---------- -------
pip 21.2.4
setuptools 58.1.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/path/to/test/bin/python3.10 -m pip install --upgrade pip' command.
$ python -m pip install --upgrade pip
[...]
$ pip list
Package Version
---------- -------
pip 21.3.1
setuptools 58.1.0
Edit
As mentioned below, I believe the reason why the procedure above just works is simply that when building and installing Python from source, the installation simply will be correct, which I suspect the 3.10 installation in Ubuntu 21.04 is not.
I made my Python installation from source somewhere under my home directory, in order to not risk messing things up, and I also did not permanently modify the PATH.
Setting the path to $PATH:/path/to/install/to/bin should be fine however, either permanently, or just when running mkvirtualenv.
Doing that, my new installation even seems to integrate seamlessly with my system's virtualenvwrapper.
I don't really need to be on the Python leading edge myself, but if I did, I would definitely make myself independent of Ubuntu's latest development by using the procedure described above.
By the way, if I update pip directly in the Python installation built from source, I will no longer get the message about the older pip (see above) when I install new virtual environments.
Edit 2
Bug reported to Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1955742
Edit 3
Also, since Ubuntu 21.04 has end of life in a month or so, upgrading to 21.10 could really make sense. I have just tried that, and it seems that Python 3.10 virtual environments work just fine in that release.
With the advent of Ubuntu 22.04 (which caused a few minor issues with some specific python virtualenv setups I had) and having already spent some time figureing this out, I ended up switching to using pyenv.
Pyenv made it very easy to install any python version and any number of virtual environments within it which can be further customised in complete isolation.
The only thing to be careful of is installing all necessary python prerequisites before installing a specific version to avoid missing functionality from some packages (e.g. not including the lzma library will generate an ominous warning from pandas ("your python installation is incomplete..."). This can be ignored if you are not using that functionality or otherwise, easily fixed.
I think that this is a better option overall if you have to manage different versions and specific configurations for python, even across distros.
This solution is very close to the one suggested before ("install from source") so I will be accepting that one and leave my contribution as additional information about this problem.
Question: How can I fix the Dockerfile to properly freeze requirements.txt and successfully build?
I am working through deploying a Dockerfile of a Python script utilizing Pyenv and Pipenv for local development.
On the build step where the Piplock file is frozen to requirements.txt, I receive the following error:
Error: Invalid value for "--python": Expected Python at path
/Users/jz/.local/share/virtualenvs/quanter-TP0oWHoL/bin/python3 does
not exist
My Dockerfile is:
FROM python:3.7
RUN pip install pipenv
COPY Pipfile* /tmp/
RUN cd /tmp && pipenv --python /Users/x/.local/share/virtualenvs/quanter-TP0oWHoL/bin/python3 lock --requirements > requirements.txt
ENV RDKAFKA_INSTALL=system
RUN pip install -r /tmp/requirements.txt
COPY . /tmp/app/
RUN pip install /tmp/app/
CMD ["python", "./tmp/app/main.py"]
Creation of the local Pipenv environment provided this information about the interpreter, (which was used in the Dockerfile):
Using /usr/local/opt/pyenv/versions/3.8.0/bin/python3 (3.8.0) to
create virtualenv… ⠙ Creating virtual environment...Using base prefix
'/usr/local/opt/pyenv/versions/3.8.0' New python executable in
/Users/x/.local/share/virtualenvs/quanter-TP0oWHoL/bin/python3 Also
creating executable in
/Users/x/.local/share/virtualenvs/quanter-TP0oWHoL/bin/python
Pyenv is using 3.8.0 locally:
pyenv versions
system
3.7.5
* 3.8.0 (set by /Users/x/Projects/quanter/.python-version)
Any help getting this working would be greatly appreciated! Thank you.
The original error came from the idea that pipenv can not find python executable on the given path. It is hardcoded, but on every fully new build, pipenv will create env with a different name. Take a look at this interesting discussion: How does pipenv know the virtualenv for the current project?, it seems like it could be predictable, but there should be a better solution.
Firstly, you do not need to specify a path to python executable directly because python image comes only with one system python installation, e.g. it will be available by default via python (of course you can create different environments with different versions of python by hand).
Secondly, you can handle pipenv in docker env in a more nicer and preferable way, instead of converting pipenv into pip like flow. Take a look at the example below:
FROM python:3.7
COPY Pipfile /
COPY Pipfile.lock /
RUN pip3 install pipenv \
&& pipenv install --system --deploy --ignore-pipfile
Several notes worthing to consider:
Commit Pipfile.lock into VCS. In this case, you can be sure that in every environment you have exactly the same versions of packages and its dependencies.
You may be interested in --system option. That says to install dependencies into a system instead of virtualenv (this is what you expect). But according to this answer: https://stackoverflow.com/a/55610857/9926721 It is not officially recommended. But still, a quite popular project wemake-django-template uses it as well.
Consider splitting up environments. There is useful flag --dev for it. No more several requirement files. Handle it more efficient way.
I am very new at command line usage. I am using python 3.7.2, Bash and VSCode Integrated Terminal. I am trying to create a virtual environment using venv and following python documentation:
https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments
The command to use is this one:
$ python3 -m venv test-env
and I get:
bash: python3: command not found
Later I have found a similar answer in an stackoverflow post:
How to create and activate virtual environment in windows 10 using bash command
And I use the command:
py -m virtualenv test-env
and I get this:
No module named virtualenv
I am very new using the command line so i don´t really know what is going on and how to work it around.
Hi i can see that you are using two different tools to create your environment.
Those are "venv" and "virtualenv".
Venv is a library that already comes with your python installation.
Virtualenv is an external one.
I had the same problem before and the solution is very simple.
I recommend you to stick with venv because it works pretty ok and you don´t need to do extra job installing external libraries.
So for solving your problem the Bash Shell is telling you that the command Python3 has not been found.
So try instead just:
python -m venv test-env
Sometimes Python documentation is not accurate enough and I know when you start using commands, accuracy in the sintax is extremely important.
Try this steps,it'll helped you:
First, make a directory :
mkdir testing
Then, moved to this directory named testing :
cd testing
When you type following command in this directory:
python3 -m venv env (OR, python -m venv env)
You got error like :
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.8-venv
Type the following command but before that keep an eye on the version of python you installed on the machine; in my case its python3.8
sudo apt install python3.8-venv
Now, we can create a virtual environment and store its tools in the "bhandari" folder .
python3 -m venv bhandari
Note: you can named this "bhandari" folder; anyname you like( Standard practice is to name it "env" ...)
Now to activate your virtual environment, from the directory of your folder, type the following command this will activate our virtual environment in the “bhandari” folder
source bhandari/bin/activate
If you have successfully activated your virtual environment, you should see the (bhandari) word indicating that we are working in a virtual environment.
After this, we can install anything that will be isolated from the rest of the system....
On Mac, if I simply open a new terminal window and run:
python --version
I get:
3.6
but if I do this:
virtualenv venv && source venv/bin/activate
and then, in that environment, I run :
python --version
I get:
2.7
I need virtualenv to run 3.6. How do I do that?
This :
/usr/bin/python
is 2.7 but this:
/usr/local/bin/python
is 3.6. The path for my normal user has /usr/local/bin come up before /usr/bin/. Is virtualenv running as someone else? How do I control its path?
I ran this:
virtualenv -p /usr/local//Cellar/python/3.6.5/bin/python3 venv
but then I do this:
virtualenv venv && source venv/bin/activate
and I'm running in an environment with 2.7.
On Python 3 you don't need virtualenv script anymore, you should just use the venv module included with standard lib:
python3 -m venv myvenv
But if you really want to keep using the old virtualenv script, you can - specify the interpreter explicitly with the -p option:
virtualenv -p /path/to/python3 myvenv
The easiest way is to change python globally to Python3 as I think you're using it more often than Python 2.7 (or hopefully always). To achieve this, add the following line of code at the end of your .bash_profile:
alias python='python3'
virtuanenv is using /usr/bin/python, hence it should work now.
If you don't want to change it globally, than you should use the following command to create your Python3.6 virtual environment:
python3 -m venv venv
or the explicit Python version if you have multiple Python3 versions installed:
python3.6 -m venv venv
On more suggestion at the end: I recommend you to read something about pipenv as it's the new recommended way to handle virtual environments and your whole package management at once. It's super easy and fixes a lot of common issues. Here's a nice article from realpython.com on that topic.
Hope I could help you. Have a nice day.
I am able to install python module in virtualenv but it is accessible outside the virtualenv as well. How to restrict its usage in virtualenv?
I went to the virtualenv path and then typed activate. It got activated as I could see root at the beginning.
And then I used the command pip install module_name
Activated the virutal env and then deactivated it.
When I activated the virtual env I was able to import the module.
When I deactivated it still the module was easily imported.
I assume the module was installed globally. I want it to be installed only in virtual env and should not be accessible outside.
The virtualenv tool is basically used to isolate the dependencies required for multiple projects. The python version installed within virtualenv won't be visible in the global directory.Try following the below steps properly in the virtualenv.
pip install virtualenv
cd my_project_folder
virtualenv venv
virtualenv -p /usr/bin/python3.5 venv
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python2.7
source venv/bin/activate
[Vitualenv Link:][1]
http://docs.python-guide.org/en/latest/dev/virtualenvs/
virtualenv -p /usr/bin/python3 name_of_venv #to create venv
source name_of_venv/bin/activate #to activate venv
pip install module_name #to install module
It is not visible from the outside. As soon as you type deactivate it is no longer importable.