I'm in the middle of switching from VS Code to Atom and I'm trying to set up a virtual environment for my python project.
It was pretty easy to do in VS Code, I'd run the following script and it would automagically start using the new env (with all the required packages) when I'd run the script:
python3 -m venv my_env
source my_env/bin/activate
pip3 install -r requirements.txt
Now I'm trying to set up Hydrogen to work the same way. When I run lines of code inline with Hydrogen, I want them to be run in a virtual environment that has the imported modules I need from a requirements.txt file.
I was able to install the python3 kernel with the following commands:
python3 -m venv my_environment_name # create a virtual environment
source my_environment_name/bin/activate # activate the virtual environment
python -m pip install ipykernel # install the python kernel (ipykernel) into the virtual environment
python -m ipykernel install
And Atom is able to see it:
Screenshot
However, I'm still puzzled as how to install my dependencies into the kernel. And if I do install my dependencies there, I don't want my next python projects to have all those modules in there. I'd love to have the fresh-slate that virtual environments promise.
Any help here would be appreciated. Has anyone had experience setting up a virtual environment that can be used by the Hydrogen package?
Ok, after some more experimentation, I was able to connect to a kernel that I had installed my requirements.txt into.
Here are the steps I took:
python3 -m venv env
source env/bin/activate
# make sure requirements.txt has ipykernel in it
pip3 install -r requirements.txt
python3 -m ipykernel install --user --name=env
Then in Atom, press cmd-shift-p and find Hydrogen: Update Kernels.
Or manually Packages->Hydrogen->Update Kernels
After, I was able to use the kernel by doing cmd-shift-p again and selecting Hydrogen: Start Local Kernel and selecting env.
When I would run import statements via Hydrogen (selecting them and pressing cmd-enter), they would now know what to import! Horray!
Windows Version.
This is the windows version of the excellent answer by Tim Estes. Without the use of requirements.txt in case some folks such as myself would like to install ipykernel separately. (Thanks so much Tim, it saved me a ton of time)
python -m venv env
env/bin/activate
pip install ipykernel
python -m ipykernel install --user --name=env
Similar to Tim's answer, go ahead and update kernels at Atom.
Packages->Hydrogen->Update Kernels
Related
I have a 2-part question about conda vs. pip virtual environments. I found great information on the answers What is the difference between pip and conda? and Does Conda replace the need for virtualenv? but still have something unclear.
I have a given python project (say PR) that I need to install and further develop on a linux server (say S) where python is installed with anaconda. Now, the usage/installation instructions of PR tell me to use python to create virtual environment and pip to install all packages. That is,
python3 -m venv PR
pip install --editable . (the dot included at the end)
According to "pip install --editable ./" vs "python setup.py develop" the latter reads the file setup.py (included in PR) which contains a function setup(...) with option install_requires listing all the required packages and installs them automatically. I have tested this on my own computer (which does not have conda) and it works fine. At least no error messages.
Now I need to further develop PR on S. My question Part 1: can I use conda instead of pip to create and update virtual environment? If yes, what would be the conda command replacing pip install --editable . ? I'm positive I will later need to install other packages as well. I'm worried about conflicts between conda/pip.
On S, I have Spyder and no other python IDEs. I have never used Spyder but I'm very familiar with PyCharm (Windows) and VS Code (Linux) so I assume debugging with Spyder will be similar to those. My question Part 2 (tied to Part 1): if I have to use pip to install packages, does Spyder see those? Or can it only see conda-installed packages?
(Edit/update): Thank you Carlos for comments. I continue my question:
I created and activated the virtual environment (VE) with conda
conda create PR_venv
conda activate PR_venv
Installed pip with
conda install pip
(this upgraded pip and installed several other packages too, including newer version of python). Installed PR and its required packages with pip
pip install -e .
Now, if I run the PR package inside this active VE interactively from the terminal, everything works fine. I would like to do the same from within spyder, to get the IDE debugging abilities in my hand.
When I start spyder, open a python file to be run, click "Run" button, it crashes in the import statements.
Spyder cannot see the installed packages. It can see only the local package PR but none of the packages installed by pip for this VE.
I am not sure what is the correct question here; I'm confused how are conda VEs related to spyder/jupyter/ipython ? I cannot find information in the conda documents about this.
I cannot find from spyder documents anything about VEs. Do I have to somehow re-install the packages (how?) inside Spyder? It seems pointless because the packages are installed already.
(Edit/Update 2): The information on https://docs.spyder-ide.org/current/installation.html makes me even more confused: Spyder is presented as both a stand-alone program and as a python package. So do I have to re-install Spyder inside the VE(?!) with
conda activate PR_venv
conda install spyder
Any clarification would be appreciated. I have always thought that the IDEs are stand-alone programs and that's it. This Spyder setup twists my brains into pretzel.
(Spyder maintainer here) About your questions:
can I use conda instead of pip to create and update virtual environment?
Yes, you can. Please see here to learn about the functionality offered by conda for managing environments.
If yes, what would be the conda command replacing pip install --editable . ?
Conda doesn't offer a good replacement for that command. However, you can still use it in a conda environment, as long as all you've installed all your package dependencies with conda before running it. That would avoid mixing conda and pip packages, which usually leads to really bad results.
if I have to use pip to install packages, does Spyder see those? Or can it only see conda-installed packages?
Spyder can work with pip and conda packages without problems. Just make sure of not mixing them (as I said above) and you'll be fine. In addition, please read our documentation to learn how to connect a local Spyder instance to a remote server.
Part 1: yes I can use conda to create VE and pip to install packages
conda create PR_venv
conda activate PR_venv
conda install pip
pip install --editable .
conda list
The last line shows which packages are installed by conda and which by pip (shown as pypi)
Part 2: spyder by default cannot see the packages. Need to do two things:
conda install spyder-kernels
Open Spyder and Tools > Preferences > Python Interpreter > Use the following interpreter > [full path to VE python command]
Restart Spyder. Now it can see the packages.
(Edit:) this link is great: https://github.com/spyder-ide/spyder/wiki/Working-with-packages-and-environments-in-Spyder
I have different venvs in my machine in which I have python 3.10.
Now for a specific project, I realised that python 3.10 is not suitable as some libraries are still not compatible. Therefore when creating a new venv for a new project, I would like to downgrade python, say to 3.8, only for this specific venv.
How can I do that?
What should I type onto the terminal to do this?
PS: I use VS and its terminal to create venv
The recommended way by python.org
The recommended way of managing virtual environments since Python 3.5 is with the venv module within the Python Standard Library itself.
Source: https://docs.python.org/3/library/venv.html#creating-virtual-environments
That is not the same as virtualenv, which is a third party package, outside the Python Standard Library.
Source: https://pypi.org/project/virtualenv/
Dangerous to downgrade (and to upgrade)
Depending on if your system itself uses Python, it could be dangerous for system stability to change the system Python version. Your system might need exactly that version of Python. That is true with Ubuntu.
Install another version of Python
Safer than downgrading or upgrading is installing other versions of Python on the same system.
For example, in Ubuntu 20.04, to install Python 3.9:
# Update package lists
me#mydevice:~$ sudo apt update
# Add the deadsnakes repository
me#mydevice:~$ sudo add-apt-repository ppa:deadsnakes/ppa
# Install Python 3.9
me#mydevice:~$ sudo apt install python3.9
Install the venv package and create a venv virtual environment
# Install the venv package for Python 3.9
me#mydevice:~$ sudo apt install python3.9-venv
# Make a folder for venv virtual environments
me#mydevice:~$ mkdir ~/.venvs
# Create a new venv virtual environment with Python 3.9 in it
me#mydevice:~$ python3.9 -m venv ~/.venvs/my-venv-name
# Activate the new venv
me#mydevice:~$ source ~/.venvs/my-venv-name/bin/activate
(my-venv-name) me#mydevice:~$
Check versions within the venv virtual environment
# Check the Python version inside the venv:
(my-venv-name) me#mydevice:~$ python -V
Python 3.9.9
# Check the Pip version inside the venv:
(my-venv-name) me#mydevice:~$ pip3 --version
pip 21.2.4 from /home/me/.venvs/my-venv-name/lib/python3.9/site-packages/pip (python 3.9)
Deactivate the venv virtual environment
(my-venv-name) me#mydevice:~$ deactivate
me#mydevice:~$
Check versions outside any virtual environments
# Check Python version:
me#mydevice:~$ python -V
Python 3.8.10
# Check the Pip version:
me#mydevice:~$ pip3 --version
pip 20.0.2 from /home/me/.venvs/my-venv-name/lib/python3.8/site-packages/pip (python 3.8)
Install more Python versions
To install more Python versions, just change the version number from 3.9 to which ever version you choose, that is available from the deadsnakes repository.
I believe the best way to work with different python versions in isolation is pyenv, managing virtual environments can be done with pyenv-virtualenv.
I think this article from Real Python does a good job at explaining how to manage different python versions as well as different virtual environments.
For posterity, with the tools mentioned above you can do the following (once the proper python versions are installed)
pyenv virtualenv <python_version> <environment_name>
# Then activate it
pyenv local <environment_name>
Now that you've created a virtual environment in the folder, it should be picked up any time you enter the folder. VSCode should also pick it up, as per its documentation.
P.S: The reason I think it's a good approach it's because it allows you to manage python versions as well as environments with a single tool. Each version is installed only once, in one place, which should help because it reduces complexity.
Simple and recent
Supposed that you have a different version of Python installed in your system. To check use the following command to check:
> py --list
-3.10-64 *
-3.7-64
And you want to create a new virtual environment for python 3.7 on a 'test_env' directory. Run the following command:
> py -3.7 -m venv test_env
Then activate the test_env by running the following command on Windows PowerShell:
> .\test_env\Scripts\Activate.ps1
Or Linux:
$ source test_env/bin/activate
Check:
python --version
Python 3.7.0
You can have multiple python versions installed at the same time and you can create virtual environments with the needed version. Make sure you have installed the python version you need and then specify its location when you create the virtual environment:
virtualenv -p <path-to-new-python-installation> <new-venv-name>
Example:
virtualenv -p C:\Users\ssharma\AppData\Local\Programs\Python\Python38\python.exe venv38
This will create a virtual environment called venv38 with Python 3.8.
you can do it by using "virtualenv" library. It can be installed with command pip install virtualenv
then followed by command
virtualenv "name_of_your_environment" #no quotes
and then use the following code to activate your venv
"name_of_your_environment"\Scripts\activate #NOTE YOU MUST BE at your directory where you created your env.
its for VS CODE but I prefer installing conda and then creating env on conda prompt using conda which later you can access to vs code to and its easy to activate that env from anywhere just type conda activate 'name_of_your_env' on vs terminal
You can use anaconda:
conda create -n mypython3.8 python=3.8
For merely the sake of enough porridge in your stew.
So I am a python newb like first project ever newb. I jumped right in disregarding virtualenv and installed everything globally. Now I need to be able to share my project with other team members.
Can I create a virtualenv after making the mistake of installing all project packages globally?
I am using python 3. I've read these links:
pip installing in global site-packages instead of virtualenv
How to import a globally installed package to virtualenv folder
But I don't think thats what Im looking for. I want to go the requirements.txt route I think.
Any guidance or suggestions?
Yes, you can create a virtual env.
You can create a requirements.txt file for the packages you installed globally.
pip3 freeze > requirements.txt
and then you can use this requirements file for installing all the packages in the virtual env which will be isolated from your global environment.
First you need to install virtualenv:
pip3 install virtualenv
Create a new virtual env using the following command:
virtualenv -p python3 envname
You can activate the virtualenv by:
source /path/to/new/virtual/environment/bin/activate
To deactivate the environment and return to your local environment just run:
deactivate
Install the requirements from the file.
cat requirements.txt | xargs -n 1 pip3 install
This should install all your packages in the virtual environment.
To check which python you are using use which python command and to check installed packages use pip3 list
I hope this will clear your doubt.
I usually use conda to manage my environments, but now I am on a project that needs a little more horsepower than my laptop. So I am trying to use my university's workstations which have new Intel Xeons. But I don't have admin rights and the workstation does not have conda so I am forced to work with virtualenv and pip3.
How do I generate a requirements.txt from conda that will work with pip3 and venv?
conda list -e > requirements.txt
does not generate a compatible file:
= is not a valid operator. Did you mean == ?
The conda output is:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-64
certifi=2016.2.28=py36_0
cycler=0.10.0=py36_0
freetype=2.5.5=2
icu=54.1=0
libpng=1.6.30=1
matplotlib=2.0.2=np113py36_0
mkl=2017.0.3=0
numpy=1.13.1=py36_0
openssl=1.0.2l=0
pip=9.0.1=py36_1
pyparsing=2.2.0=py36_0
pyqt=5.6.0=py36_2
python=3.6.2=0
python-dateutil=2.6.1=py36_0
pytz=2017.2=py36_0
qt=5.6.2=2
readline=6.2=2
scikit-learn=0.19.0=np113py36_0
scipy=0.19.1=np113py36_0
setuptools=36.4.0=py36_1
sip=4.18=py36_0
six=1.10.0=py36_0
sqlite=3.13.0=0
tk=8.5.18=0
wheel=0.29.0=py36_0
xz=5.2.3=0
zlib=1.2.11=0
I thought I would just manually change all = to == but the there are two = in the conda output. Which one to change? Surely there is an easier way?
EDIT: pip freeze > requirements.txt gives:
certifi==2016.2.28
cycler==0.10.0
matplotlib==2.0.2
matplotlib-venn==0.11.5
numpy==1.13.1
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.2
scikit-learn==0.19.0
scipy==0.19.1
six==1.10.0
As the comment at the top indicates, the output of
conda list -e > requirements.txt
can be used to create a conda virtual environment with
conda create --name <env> --file requirements.txt
but this output isn't in the right format for pip.
If you want a file which you can use to create a pip virtual environment (i.e. a requirements.txt in the right format)
you can install pip within the conda environment, then use pip to create requirements.txt.
conda activate <env>
conda install pip
pip freeze > requirements.txt
Then use the resulting requirements.txt to create a pip virtual environment:
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
When I tested this, the packages weren't identical across the outputs (pip included fewer packages) but it was sufficient to set up a functional environment.
For those getting odd path references in requirements.txt, use:
pip list --format=freeze > requirements.txt
Problem
In a conda environment with simply calling
pip freeze
Output:
ipykernel # file:///C:/ci/ipykernel_1607454116140/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython # file:///D:/bld/ipython_1612487184680/work
...
Wanted format:
ipykernel==5.3.4
ipython==7.20.0
...
Solution
In an activated conda environment I had to use
pip list --format=freeze
to get the correct format for generating a requirements file for people who prefer to use pip with virtual environments.
Save to file:
pip list --format=freeze > requirements.txt
Following the discussion, I'd like to mention that you can actually see some separation of pip and conda roles.
pip is a standard package manager, it does one thing and does it well. requirements.txt can be generated in one environment and installed by pip in a new environment.
Now there is conda output: you rightfully capture their comment which says 'we generated this list of libraries to work with conda'. Note that python itself is in the conda list and (properly) not in requirements.txt. conda replicates own installation, that is why its list of libraries is longer, and has python itself.
pip produces a list of packages that were installed on top of standard library to make the package you wrote work. Hope it helps to distinguish between the two.
Also pipenv is a newer tool, that can do both virtual environment and package management for you.
Just in case someone is looking to generate requirements.txt from an existing project in conda, use following
Go to your project environment conda activate <env_name>
conda list gives you list of packages used for the environment
conda list -e > requirements.txt save all the info about packages to your folder
conda env export > <env_name>.yml
pip freeze
As mentioned in the comments above, the correct full answer is:
pip list --format=freeze > requirements.txt
activate the conda env
conda activate flask-test
get the path of the conda env and copy it
conda list
append the copied path with lib\site-packages and use it in pip with --path option
pip freeze --path C:\Users\username\Miniconda3\envs\flask-test\lib\site-packages > requirements.txt
on Linux the path is like /home/username/miniconda3/envs/flask-app/lib/python3.8/site-packages/
I made some simple python script to convert conda requirements.txt for pip3.
Just copy below code.
f = open(r"requirements.txt", "r").read()
f1 = f.split("\n")
for line in f1:
if "=" in line:
a = line.split("=")[0:-1]
print(a[0]+"==" + a[1])
then just copy the printed output;
Sample output:
_libgcc_mutex==0.1
_openmp_mutex==4.5
blas==1.0
brotli==1.0.9
bzip2==1.0.8
On Mac, if I simply open a new terminal window and run:
python --version
I get:
3.6
but if I do this:
virtualenv venv && source venv/bin/activate
and then, in that environment, I run :
python --version
I get:
2.7
I need virtualenv to run 3.6. How do I do that?
This :
/usr/bin/python
is 2.7 but this:
/usr/local/bin/python
is 3.6. The path for my normal user has /usr/local/bin come up before /usr/bin/. Is virtualenv running as someone else? How do I control its path?
I ran this:
virtualenv -p /usr/local//Cellar/python/3.6.5/bin/python3 venv
but then I do this:
virtualenv venv && source venv/bin/activate
and I'm running in an environment with 2.7.
On Python 3 you don't need virtualenv script anymore, you should just use the venv module included with standard lib:
python3 -m venv myvenv
But if you really want to keep using the old virtualenv script, you can - specify the interpreter explicitly with the -p option:
virtualenv -p /path/to/python3 myvenv
The easiest way is to change python globally to Python3 as I think you're using it more often than Python 2.7 (or hopefully always). To achieve this, add the following line of code at the end of your .bash_profile:
alias python='python3'
virtuanenv is using /usr/bin/python, hence it should work now.
If you don't want to change it globally, than you should use the following command to create your Python3.6 virtual environment:
python3 -m venv venv
or the explicit Python version if you have multiple Python3 versions installed:
python3.6 -m venv venv
On more suggestion at the end: I recommend you to read something about pipenv as it's the new recommended way to handle virtual environments and your whole package management at once. It's super easy and fixes a lot of common issues. Here's a nice article from realpython.com on that topic.
Hope I could help you. Have a nice day.