Pyenv choose virtualenv directory - python-3.x

I just began to use pyenv to manage my python versions, and began to use the pyenv virtualenv plugin to manage my virtualenvs, and so far, I have loved it. One thing I miss however, is that with virtualenv, you could actually place virtual environments in repository directories so that your repository was a completely reproducible environment. Does anyone know of a way to choose the directory of your virtualenv in pyenv?

Short answer: You can’t, as far as I know.
It wouldn’t really work either, right? If you use pyenv virtualenv to install a virtualenv into a repo, and you clone that repo to another machine… how would pyenv on the new machine know to take control of the virtualenv in the repository?
Also, “you probably shouldn’t do that”. Virtualenvs are not 100% decoupled from the underlying Python installation, and aren’t really all that portable. And do you really want to litter your repositories with a bunch of easily replicated junk? The “right” way to go about things is probably to to maintain a requirements.txt for pip — that way you can easily reproduce your development environment wherever you clone your repo.
That all said, there’s nothing stopping you from using plain old virtualenv to create a virtualenv anywhere you like, even if you installed virtualenv into a Python interpreter under pyenv control. That virtualenv itself will of course not be administered by pyenv, but you can still use it as you always did…

Here is a GitHub issue on the project tracker asking about this: https://github.com/pyenv/pyenv/issues/802
The reply by a project collaborator was:
You can just create virtualenv in any location. To use it from pyenv, create symlink to the environment in ~/.pyenv/versions
although I must say I'm not very satisfied with this solution.

There's a workaround
In your project directory, create a .python-version file. Assuming you already installed some virtual environment using pyenv-virtualenv, then append this line to it(you may want to change the version number):
3.7.1/envs/your-project-name#3.7.1
Whenever you enter into the directory, the correct python version will be used, and it works pretty well.

How about:
pyenv install -s 3.8.3 # install preferred version if needed
PYENV_VERSION=3.8.3 python -m venv /tmp/your/prefered/dir ; source /tmp/your/prefered/dir/activate
Note that this method uses venv, not virtualenv, which is slightly different, but a part of a standard library.

Slightly left-field answer, but I ended up writing a hacky equivalent of pyenv that looks for a virtualenv in a directory and its parents because pyenv doesn't really seem to support this as layed out in the answer about.
I called this script enpy, put it on my path and then called e.g. enpy -c import sys; print(sys.executable)
#!/usr/bin/python3
import os
import sys
from pathlib import Path
NAMES = ("venv", "env", ".env", ".venv")
def find_python():
cwd = Path(os.getcwd())
for direc in [cwd] + list(cwd.parents):
for p in NAMES:
venv = direc / p
if venv.exists():
return venv / "bin" / "python"
else:
raise Exception("Could not find python")
python = find_python()
os.execvp(str(python), [str(python)] + sys.argv[1:])

To build on-top of a previous comment, you can add this to your .zshrc:
pyCreate () {
if [ -d .venv ];then
rm -rf .venv; fi &&
if [ -f .python-version ];then
pyVersion="local"
else
pyVersion="global"; fi &&
location=${1:-requirements.txt} &&
if [ -f $location ];then
action=(pip install -r $location)
else
action=(echo "\n No file: $location \n"); fi &&
PYENV_VERSION=`pyenv $pyVersion` python -m venv .venv &&
source .venv/bin/activate &&
pip install --upgrade pip && $action
}

I use the simple recipe of
python -m venv virt to create the local venv
Link the venv to pyenv with ln -s <local virt> ~/.pyenv/versions/<venv name>
Now you can do pyenv local <venv name> and all other normal operations from pyenv with this venv.
For the interested I have made two CLIs
pyenv-here to create a new venv and set local dir in one go
pyenv-link to link a local virt to pyenv

The top comment is wrong; you certainly can.
Once you create your virtual environment with pyenv virtualenv 3.9.15 your-environment-name, you can specify the directories in which this environment will be used automatically with the following command. In your desired directory, run:
pyenv local your-environment-name
The command will create the necessary .python-version file that includes the name of your environment. From now on, every time you enter the folder, the specified environment will be used.

If you want to just use pyenv specific python version and create virtual env from it in wanted directory, you can for example do this:
~/.pyenv/versions/3.10.8/bin/python -m venv my/path/to/venv

Related

Creating virtual environment Python

I need to provide my version of Python and packages for a project.
How can I do that?
I tried:
sudo apt-get install python3-venv
virtualenv my-env -p python3
source tutorial-env/bin/activate
This should show installed packages, but it shows:
pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
pkg-resources (0.0.0)
setuptools (39.0.1)
This is also suspicious:
(tutorial-env) linux#LINUXMINT:~$ pip freeze > requirements.txt
(tutorial-env) linux#LINUXMINT:~$ cat requirements.txt
pkg-resources==0.0.0
(tutorial-env) linux#LINUXMINT:~$ python -m pip install -r requirements.txt
Requirement already satisfied: pkg-resources==0.0.0 in ./tutorial-env/lib/python3.6/site-packages (from -r requirements.txt (line 1))
And I cannot find requirements.txt in my directory.
Pass the version of python when creating your virtualenv like this:
virtualenv my-env -p python3
I think you should not use any external module for creating a virtual environment.
You can create a Virtual Environment by using venv attribute of Python in Command Line.
What is the syntax?
The syntax is pretty simple.
C:\>python -m venv path\to\where\you\want\to\create\it
It can be easily done, and you can get a reference from the example below:
C:\>python -m venv "C:\Users\Bhavyadeep\Desktop\Discord Bots\Bot" 1\Bot-1-env
Here the name Bot-1-env is the name of the folder which will be created on execution of the command, and it doesn't have to exist.
What if I am using an IDE (like VS Code), then how will I create a virtual environment?
Creating a Virtual Environment in an IDE is much easier than creating it using CMD. In CMD you need to specify the full path of the directory where the Environment has to be created whereas in an IDE you can create one using its own terminal and also there won't be any need of adding and full path to the directory.
Syntax for IDEs with their Terminals is:
C:\>python -m venv My-Env
This would simply create a Virtual Environment in the folder of the project you are working on in the IDE. If you still want to create it using full path you can do the same as above in the Terminal of IDE.
Example with Images and Code in One Step:
My target directory would be the Desktop for now to explain.
Write and Execute the command in Command Line.
I entered the line in the image and pressed Enter.
Here the name of the Folder would be Example-Venv and it doesn't exist. This command created a folder with that name and created that a Virtual Environment.
This command created a folder, and it can be seen in the picture below.
Now you can use it anywhere you want by simply making this folder as the Interpreter.
How to set interpreter?
The following links would explain:
Pycharm: https://www.jetbrains.com/help/pycharm/configuring-python-interpreter.html
VS Code: https://code.visualstudio.com/docs/python/environments#:~:text=To%20select%20a%20specific%20environment,or%20library%20versions%20as%20needed.
Spyder: https://www.spyder-ide.org/blog/release-spyder-330/#:~:text=Just%20set%20the%20path%20under,start%20in%20the%20selected%20environment.
Sublime: https://medium.com/#hariyanto.tan95/set-up-sublime-text-3-to-use-python-3-c845b742c720
I was glad to help! If you still get any problem, please feel free to ask in the comments and I would gladly help you! :)
Thank You! :)

Virtualenv python not recognizing .pth for custom site-packages

I installed a new virtual env with python3.6. After checking its site packages (python3.6 -m site --user-site) I was pointed to /Users/username/.local/lib/python3.6/site-packages.
I added a file named custom.pth with the contents:
/Users/username/Python Files/Packages
but for some reason, it still fails to recognize our inhouse packages. However, I have the exact same .pth file for the system's python3.6 and it works without a hitch. Is there something else I can try?
It finally worked when I placed the .pth file into the <venv folder>/lib/python3.6/site-packages/ directory. Perhaps it's an issue with the activated venv overriding the python's site-packages
Python virtual environments are intended to isolate packages from both global and user site-packages. User site-packages are not available to Python in a virtual environment. Compare python -c "import sys; print(sys.path)" before and after activating a venv. Hence their .pth files are not processed.
Outside of a venv .pth files from the user site-packages are processed.

Set the PYTHONPATH for a older version of Python to a directory where I want to install it though I have a default version of Python3

I am using ubuntu 18.04. I have Python-3.6.6 installed as a default version. But for practice I have installed Python-3.6.5 from source code to a directory where I wanted to install it and changed the .bashrc file accordingly. But it is showing me the default python3 path when I call "which" command.
How to use the python version which one I have installed recently?
If you only want to run some practice scripts you can select directly which python you want to use for your script. In the command line, e.g. type
PYTHONPATH=/foo/bar/python365 python somescript.py
where /foo/bar/python365 points to your python-3.6.5
You can also change your PYTHONPATH variable to look first in the python-3.6.5 folder before it looks to the default python (3.6.6) you had installed before. However, I do not recommend this, since it will make all python applications choose the python-3.6.5 as default. For extended practice with python-3.6.5 consider a virtual environment or an anaconda environment.
You should use virtualenv or something similar, like pipenv, I myself use virtualenv.
Here's a guide on how to do that: https://docs.python-guide.org/dev/virtualenvs/
virtualenv is a bit lower level and so not that intuitive to use, but since I used virtualenv before I found pipenv, I just stuck with it, while adding a few aliases to my .bashrc to make it easier.
I used two different python versions because discord.py would break on 3.7 (that doesn't seem to be the case anymore, but I still have it setup with 3.6 and 3.7), so here's what my .bashrc for two different versions looked like, this is on windows, but if you are on unix system or macOS (I want to make this answer as general as possible, it works the same on all operating systems really), just change the paths to lead to your interpreters (whichever you have) and don't use winpty:
alias python36="winpty C:/Users/Hevaesi/AppData/Local/Programs/Python/Python36-32/python.exe"
alias python37="winpty C:/Users/Hevaesi/AppData/Local/Programs/Python/Python37-32/python.exe"
alias pip36="C:/Users/Hevaesi/AppData/Local/Programs/Python/Python36-32/python.exe -m pip"
alias pip37="C:/Users/Hevaesi/AppData/Local/Programs/Python/Python37-32/python.exe -m pip"
alias venv36="python36 -m virtualenv"
alias venv37="python37 -m virtualenv"
alias python="winpty python"
Now, to use Python 3.6, I use venv36 venv in my project folder, and it setups Python 3.6 environment, using venv37 venv would create Python 3.7 environment.
I also wrote a function to make activating less of a hassle (again, this is in bashrc)
activate() {
if [ "$#" == 1 ]; then
source "$1/Scripts/activate"
else
echo "Usage: activate <env_name>"
fi
}
So, after I created my env once, I go to project folder, type activate venv and it starts up whatever version I wanted the project to have all by itself, to run my project I just use python myfile.py, when virtual environment is activated, it will use the interpreter that's in active virtual environment, regardless of what you have in your PATH variable. Also, using pip will automatically install to your virtual environment, not your actual installation, meaning global package list will be clean, or in best case, completely empty.
You should only install what you need and keep it separate environments, normally you use a single environment per project. Well, unless your project is client/server at once (for whatever reason) and server uses different libraries than client, for example, more of them, client environment could have less modules since that's what the server is for, to provide something without having to do it yourself, so they could be two projects in one, hence two different environments, separating dependencies based on what the "unit" in the project really needs.
Note: you have to install virtualenv (or pipenv or whatever you want to use really) into all of your python installations, if you want to create virtual environments from them, that's what pip36 and pip37 here is for - for installing packages directly into actual python installations, for now, I only had to do pip36 install virtualenv and pip37 install virtualenv.

Pip freeze doesnt show freshly installed packages with Pycharm

I use Pycharm to create and manage my virtualenvs in my projects.
The problem is that after adding a library with pycharm, when I type the command (pip3 freeze --user), the library does not appear in the command result.
I have to manually type the pip install command each time so that the library is visible.
What manipulation should I do in PyCharm to solve this problem?
For what you are saying, the first thing that comes to mind is that you should use:
pip freeze
And not
pip3 freeze
Because the command mapped to the pip version when you have virtualenv activated is the first. Note that for installing you seem to use pip, and not pip3
Moreover, the --user option afaik is related to the packages installed in the user folder:
--user Install to the Python user install directory for your platform. Typically
~/.local/, or %APPDATA%\Python on
Windows. (See the Python documentation for site.USER_BASE for full details.)
If your packages are installed in the virtualenv folder, I would tell you to not use that option.
Also please make sure you have your virtualenv activated. In linux you can do so by source path/to/virtualenv/activate
Edit
I understand that the reason you are using pip3 is because you may have different versions of Python in your machine. Let me explain you a bit further how it works, because version management is usually a headache for many programmers and it is common to find problems when doing so.
If you install different versions of Python in your linux machine, and you do that as root, then the installation will proceed for the whole system. Usually Python2 installation folder for Linux machines is /usr/bin/python. However, I am uncertain of which directory is used for Python3 installations. You can check that easily by doing whereis python3. You can serach the path to binary of any command by doing whereis command. Note that this works also for whereis python as far as you don't have virtualenv activated.
Aditionally, the link to the binary of a command (or the set of instructions to be exectued, more broadly) is defined in certain folders in Linux, depending on whether you created the command as root or as a user, and possibly also on the distro. This works differently in Windows, that uses the Registry Edit utility to handle command mappings. When you enable your virtualenv, what you are doing is creating an environment that enables mapping system commands such as python to the Python installation in your virtualenv folder.
When you disable the virtualenv, the command points again to the default installation path. Same happens with pip, so incorrect usage of this tool may result in different packages being installed in different locations, and therefore not appearing available for the right Python version at any given circumstance.
In Linux, environment variables are shell dependent, though you can write them out with echo $variable and set them with variable=value (from bash). The search path is simply called PATH and you can get yours by typing echo $PATH.
Source: https://askubuntu.com/a/262073/426469
I encourage you to check other questions in SE network such as this: https://unix.stackexchange.com/a/42211/96121, to learn more about this.
Addendum
Quick tip: it is common to use the pip freeze command as follows:
pip freeze > requirements.txt
It is a standard that leads to understanding that modules in such file are required for the correct functioning of your application. That lets you easily exclude the virtualenv folder when you install the program in another computer, since you can readily know the requriments for a fresh installation. However, you can use the command as you want.

Python 3.6 (ubuntu 16.04) venv installing symlink not binary in environment's /bin

In Ubuntu 16.04, when I do python3.6 -m venv myenvironment, venv does not install the python 3.6 binaries in ../myenvironment/bin but only symlinks to the system binaries. The official docs for 3.6 suggests otherwise:
28.3.1 . . . (venv) also creates a bin subdirectory containing a copy of the python binary.
Unless by "copy" they mean a symlink, venv is not doing what the doc says. The pip binaries are there, but not python 3.6 or any other version binaries.
Maybe it doesn't matter since within that environment 3.6 will be the version used. But then which modules/packages will it use? venv did create the lib/python3.6/site-packages subdirectory as expected. Should I assume that unless I put a different version of a module/package there that the system-wide library will be used within this virtual environment?
Edit: In partial answer to my own question, it seems the default on Ubuntu 16.04 is for venv to not install the python binaries. Adding the --copies option forces venv to place the binaries into the env's ../bin directory. venv creates a subdirectory for the env's site-packages as expected. It does not, however, put the standard library modules/packages into the env's ../lib/python3.6 sub-directory.
According to the output of print(sys.path) (when run from within the env), the env's python copy still looks in the system's /usr/lib/python3.6 directory for the standard library. I guess that's okay if that's how it should work, but does one ever want to use a particular version of a module or package in the standard library, or is that just not ever done and only none-standard-library modules/packages are placed in the env's site-packages subdirectory?
2nd Edit: Here's an SO Q&A on the venv's behavior regarding standard library modules and packages:
Where is the standard library in python virtual environment?
The gist of the answers is that venv does not create a copy of the standard library in the env's directories. The system-wide standard library is always used. At least one of the participants in that Q&A commented it was odd that venv behaves that way. Seems odd to me as well. Does anyone know why?

Resources