This may be a noob question, I am new to using Conda environments. I am looking for some advice on how to tackle the following workflow in the best way.
I have both a work desktop and a desktop at home. I want to be able to, at the end of the day, take my work environment home.
Note: I work in Ubuntu on subsystems for windows
Say I start a project from scratch. I currently use the following workflow:
I create a conda environment.
conda create --name my_new_project python=3.10
activate my workspace.
conda activate my_new_project
I install python packages I need:
conda install -c conda-forge opencv
etc...
At the end of the day I want to copy that environment and take it to another PC, so I do the following:
conda env export --f my_new_project.yml
Finally on my home PC I do
conda env create --file my_new_project.yml
This works but requires me to make a new environment every time I switch PC. Is there a way to load the differences between the two conda environments and only add the new packages? Or is there another better way to tackle this?
There's no need to create a new environment every time. You only do that once and then update the existing environment, i.e. use the following as step 5:
conda env update -f dependencies.yml
I also suggest you to put your code, including the dependencies file, into version control if you aren't doing that already. Then, getting up to speed with your project on a different computer will only require two steps.
Related
I installed latest version of Anaconda3-2020-02.
I was trying to follow this instruction in order to create environment for running with python==3.6 instead of python==3.7., because I have python 3.6. installed.
So, running the
conda create --name snakes python=3.6
and then activating my environment with conda activate snakes, it enters environment (snakes). However there are no anaconda packages inside like jupyter notebook or others, no anaconda-navigator... So whats the purpose of it and how can I run these programs from environment?
Also, for some reason (when I am not in the environment, just regular bash shell) $PATH is not set up to the /bin directory in anaconda, just to /condabin. Can you explain this also, because I am not able to run nothing except conda command from shell after recommended installation.
If by default after installation environment is not activated, you should activate it by sourcing a file anaconda3/bin/activate. You will see indicator (base) on the left of your bash prompt.
Good thing about anaconda3-2020-02 is that it is not messing with system python, as newest python is being launched inside environment and proper$PATH is set up just inside environment. If only, for some reason, specific version of python is needed, then it makes sense to set it up with this instruction. I would be just using default one with python 3.7 probably from (base) environment.
Keep in mind that by default anaconda components are not being set up inside new environment being created. In order to bring them, for example jupyter, you should run the command like this:
conda create --name snakes python=3.6 jupyter
I just started using conda to install packages two months ago. At that moment, following instruction online, I silly created individual environment for each package. I have to switch between environments. I did not give a name for each environment, so I have to input whole path. I am wondering how to add a name for an existing environment?
If I run "conda env list", I could get below output which is saved in ~/.conda/environments.txt
base * /cm/shared/apps/anaconda2
py27 /cm/shared/apps/anaconda2/envs/py27
/cm/shared/apps/anaconda3
/cm/shared/apps/anaconda3/envs/gsas2_env
/cm/shared/apps/anaconda3/envs/pmg_env
/cm/shared/apps/anaconda3/envs/py3.8
/cm/shared/apps/anaconda3/envs/r_env
/cm/shared/apps/anaconda3/envs/scipy_env
I hope to create a name for each environment, just like "py27" is for "/cm/shared/apps/anaconda2/envs/py27".
I highly appreciate it if you could directly merge all environments into one.
Thanks for your help.
BTW: I created these environments using the root account, and then copy the path to regular user account "~/.conda/environments.txt"
Given the pending retirement of python 2 support in January 2020, we should be thinking about migrating from Miniconda2, which is python 2 based, to Miniconda3, which is python 3 based.
As of the date of asking this question, the Miniconda web pages don't address this topic.
Is there a good procedure for converting from Miniconda2 to Miniconda3 that will preserve the existing environments created under Miniconda2?
You could try to upgrade your python version as suggested in this answer:
conda install python=3.7
But I'm not sure how safe that would be. (unsafe)
A safe approach is to simply install miniconda3 into a new path on your machine reproduce identically your current environments on the new miniconda installation.
To do that, you'll have to create a spec list for each of your environments in miniconda2 by:
conda activate myenv
conda list --explicit > myenv-spec-file.txt
Then under your miniconda3 installation, do:
conda create --name myenv --file myenv-spec-file.txt
The conda docs have detailed instructions on this process.
Keep in mind that when you install miniconda3, it will add an entry into your .*rc file (e.g. .bashrc, if using bash) and the new conda based on python 3 will be used when running any conda command. So you should create your spec files prior to installing miniconda3.
Edit: As pointed out by merv and nekomatic, upgrading conda in-place is not safe.
I am using conda version 4.5.11, python 3.6.6, and Windows 10.
I create a virtual environment using conda
conda create --name venv
When I check for installed packages
conda list
it is (as expected), empty.
But
pip list
is quite long.
Question #1: Why? - when I create a virtual environment using
python -m venv venv
the pip list is empty.
When I am not in an activated virtual environment, then
conda list
is also quite long, but it isn't the same as the pip list (* see follow up below)
In general, the pip list is a subset of the conda list. There is at least one exception ('tables' in the pip list, not in conda list) but I haven't analysed too closely. The conda list changes/displays some (all?) hyphens to underscores (or pip does the reverse). There also a few instances of versions being different.
Question #2: Why? (and follow up questions - can they be? and should I care?)
I was hoping to have a baseline conda 'environment' (that may not be the right word) -ie, the packages I have installed/updated into Ananconda/conda and then all virtual environments would be pulled from that. If I needed to install something new, it would be first installed into the baseline. Only when I need to create an application using different versions of packages from the baseline (which I don't envision in the foreseeable future) would I need to update the virtual environments differently.
Question #3: Am I overthinking this? I am looking for consistency and hoping for understanding.
-- Thanks.
Craig
Follow Up #1: After installing some packages to my empty conda venv, the results of conda list and pip list are still different, but the pip list is much shorter than it was, but is a subset of the conda list (it does not include two packages I don't use, so I don't care)
Follow Up #2: In the empty environment, I ran some code
python my-app.py
and was only mildly surprised that it ran without errors. As expected, when I installed a package (pytest), it failed to run due to the missing dependencies. So ... empty is not empty.
1. conda list vs pip list
If all you did was create the environment (conda create -n venv), then nothing is installed in there, including pip. Nevertheless, the shell is still going to try to resolve pip on using the PATH environment variable, and is possibly finding the pip in the Anaconda/Miniconda base environment.
2. pip list is subset of conda list outside env
This could simply be a matter of conda installing things other than Python packages, which pip doesn't have the option to install. Conda is a more generic package manager and brings in all the dependencies (e.g., shared libraries) necessary to run each package - by very definition this is a broader range than what is available from the PyPI.
3. Overthinking
I think this is more of a workflow style question, and generally outside the scope of StackOverflow because it's going to get opinionated answers. Try searching around for best practice recommendations and pick a style suited to your goals.
Personally, I would never try to install everything into my base/root Conda environment simply because the more one installs, the more one has dependency requirements pulling in different directions. In the end, Conda will centralize all packages anyway (anaconda/pkgs or miniconda3/pkgs), so I focus on making modular environments that serve specific purposes.
I've got anaconda installed and was able to create a Python 3.3 environment. I can switch to it and conda info -e shows that I've switched.
However, I'm confused about what to set my PATH variable to. If I hard code it to the exact env then it works, but I thought that the purpose of conda was to be able to switch easily, as well as update and maintain various environments separately.
Perhaps I misunderstood and there's no way around setting my PATH everytime...
In the Windows cmd shell, use the activate and deactivate commands to change the PATH automatically. For instance, if your environment is called python3, run python3 to "activate" (i.e., add to the PATH) the python3 environment. Use deactivate to remove it.