I am using conda on a cluster. There is a space limit on the /home directory.
This has reached its limits and I would like to move the ~/.conda directory to another one keeping already installed packages. What is the best way to proceed?
conda version : 4.7.5
Linux: centos
I tried:
conda config --set pkgs_dirs /mnt/lustre/users/bdiallo/.conda/pkgs/
and got the following error message:
CondaValueError: Key 'pkgs_dirs' is not a known primitive parameter.
Windows:
conda config --add pkgs_dirs \\server\share\conda\pkgs
Linux:
conda config --add pkgs_dirs /mnt/lustre/users/bdiallo/.conda/pkgs/
Or manually edit .condarc in your home directory (ref):
pkgs_dirs:
- /opt/anaconda/pkgs
Watch for typos, both pkg and dir need to be plural: pkgs_dirs.
Thanks to #darthbith and #naeem-khoshnevis in comments.
Related
I followed the conda doc to update the conda on a google could server with this:
conda update -n base -c defaults conda
after this, it shows:
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.11
latest version: 4.12.0
Please update conda by running
$ conda update -n base -c defaults conda
# All requested packages already installed.
then i try this:
conda update --all
the same output plus some packages and progress:
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.11
latest version: 4.12.0
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: <path>
The following packages will be downloaded:
<packages and progress>
then i run conda info
active environment : None
shell level : 0
user config file : <path>/.condarc
populated config files :
conda version : 4.5.11
conda-build version : 2.0.2
python version : 3.5.6.final.0
base environment : <path>/anaconda3 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : <path>/anaconda3/pkgs
<path>/.conda/pkgs
envs directories : <path>//anaconda3/envs
<path>//.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.25.1 CPython/3.5.6 Linux/5.13.0-1019-gcp ubuntu/20.04 glibc/2.31
UID:GID : <the ID>
netrc file : None
offline mode : False
There was no error during the last installation. but it seems it is still not updated, i did it not in any environment, i downloaded and installed the conda version 4.2.0 on this google cloud server, I want to know is my conda updated to 4.12 or not, if not how could i update it properly?
thanks
While normally not recommend to update Python in-place, that is quite outdated and is likely what is preventing the conda package from being updated.
Try:
conda install -n base --dry-run python=3.9 conda=4.12
to see if updating is possible. If so, try again without the --dry-run flag.
⚠️ Note this is a risky update - if the conda package does not upgrade correctly with python, the installation could fail. I'd recommend a backup first.
I want to create a bash script to install a new virtual environment "ABC" in conda. But before I go ahead and run a command to create this env, I want to check if conda is already installed on the machine. If not installed, I want to install miniconda and then create the env "ABC". If conda is already installed then I would just go ahead and create the environment. (All this should happen within the same script)
I just want to know if it is possible to check the existence of conda within a bash script and then proceed with the rest of the installations?
'''
#!/bin/bash
<code_to_check_existence_of_conda_env_here ?>
//If it does not exist, I will run the below code
mkdir -p miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O
miniconda3/miniconda.sh
bash miniconda3/miniconda.sh -b -u -p ~/miniconda3
conda env update -f my_env.yml
'''
The main issue here is that Conda has multiple components. Which components are loaded depends on the how Conda is installed and what user the BASH script is executing under. I'll try describing the components and hopefully you can decide what is suitable to verify.
Conda Components
1: Conda Package
The first is a Python package conda, which is installed in the Anaconda/Miniconda/Miniforge base environment. If the base environment is activated, one could test
python -m conda
which will give something like
/path/to/python: No module named conda
if it isn't there. Otherwise, it outputs the conda entrypoint's documentation.
2: Conda Entrypoint
The entrypoint conda, which acts as a CLI, is located under the condabin directory of the Anaconda/Miniconda/Miniforge installation. When a user runs conda init, a managed section is added to their shell initialization file (.bashrc for Linux BASH), that includes code to add the condabin to PATH. This is most likely what OP wants to identify, however, running with the shebang /bin/bash will not load the .bashrc file. Instead, one should probably be using
#!/bin/bash -l
or
#!/usr/bin/env bash -l
Then the entrypoint can be located with
which conda
3: Conda Activate
Finally, Conda also includes some shell-only functions, which are defined in the aforementioned shell initialization code. This sets up a middleman shell function, also called conda (essentially an alias), which can be viewed with
type conda
This function serves to determine whether the conda (de)?activate commands are being requested, which are pure shell functions, or something that needs to be forwarded to the entrypoint.
Recommendation
Were I designing this, I would write an interactive script that checks for #2 (which conda) and if that comes up blank then prompt the user to either provide the PATH to the Conda entrypoint (maybe they installed it in a weird place or didn't run conda init) or install Miniforge.1
I also would not use the base environment to install stuff - that is a bad idea for an end-user, let alone a third-party - and instead create a new environment. I would prompt the user with a specific default environment name, but also provide an option for them to customize.
[1] Yes, Miniforge, not Miniconda. Commercial use of the Anaconda defaults channels now requires a paid license, so better to use the free Miniforge.
I'm not really good at bash scripting but I would execute a command:
pip3 freeze | grep conda
and based on the output (if exit status was an error or not) install it or directly create the environment.
run the command conda list on your machine, the output will of be different when its installed vs not installed, then run if statement on the output.
In bash, if we want to check if a software is installed or not, the following check is used.
if dpkg -l $SOFTWARE; then
<Do stuff>
ensureconda
This sounds like the intended problem that the experimental tool ensureconda was designed to solve. This, however, requires a Python installation with pip:
pip install ensureconda
Here are the command options:
$ ensureconda --help
Usage: ensureconda [OPTIONS]
Ensures that a conda/mamba is installed.
Options:
--mamba / --no-mamba search for mamba
--micromamba / --no-micromamba search for micromamba, install if not
present
--conda / --no-conda search for conda
--conda-exe / --no-conda-exe search for conda.exe / conda-standalone,
install if not present
--no-install don't install conda/mamba if no version can
be discovered
--min-conda-version VERSIONNUMBER
minimum version of conda to accept (defaults
to 4.8.2)
--min-mamba-version VERSIONNUMBER
minimum version of mamba/micromamba to
accept (defaults to 0.7.3)
--help Show this message and exit.
Type conda activate
or conda activate env_name
The first command will activate the base environment if it is already installed. The second command will do activate the user created environment if it is installed.
I run through the following steps to attempt to start up an app for production:
-Setup a virtualenv for the python dependencies: virtualenv -p /usr/bin/python3.8 ~/app_env
-Install pip dependencies: . ~/app_env/bin/activate && pip install -r ~/app/requirements.txt
-Un-comment the lines for privilege dropping in uwsgi.ini and change the uid and gid to your account name
-Login to root with sudo -s and re-source the env with source /home/usr/app_env/bin/activate
-Set the courthouse to production mode by setting the environment variable with export PRODUCTION=1
-Start the app: cd /home/usr/app && ./start_script.sh
And I get the following error:
(app_env) root#usr-Spin-SP314-53N:/home/usr/Desktop/app# ./start.sh
uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
I tried a few things such as installing a newer libpcre version like mentioned here, tried also the steps mentioned here but that didn't work. Also the environment I'm setting up doesn't use anaconda but regular python. I even tried pip install uwsgi in my virtual env but it said the requirement was already satisfied. I'm not much of an expert when it comes to somewhat complex package management like this, help with how to solve this would be greatly appreciated. Thanks. I'm on Ubuntu 20.04, using python 3.8.
What solved it for me was apparently just reinstalling UWSGI, like in this thread, in my virtual env while forcing it to ignore the cache so it could know to use the pcre library I installed.
In order, doing this
uwsgi --version
Was giving me this
uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
So I made sure I had the latest libpcre installed
sudo apt-get install libpcre3-dev
And then what linked it all together was this
pip install uwsgi -I --no-cache-dir
I tried to solve this error but it did not work no matter what I did and then reinstalled uwsgi, but the following 2 lines solved my problem
sudo find / -name libpcre.so.*
#change the path of the /home/anaconda3/lib/libpcre.so.1 with the one appears after above one.
sudo ln -s /home/anaconda3/lib/libpcre.so.1 /lib
which python
I have a conda environment at home that I'm using on my Ph.D., but now that I'm needing more computational power I have to transfer (or install a perfect copy) of my environment on one of the University's computers. The computers have no internet connection, all I have is SSH.
My attempt to copy the entire /anaconda3 directory and .bashrc to a path similar to the one I use at my home (/home/henrique/bin) have not worked.
What is the correct way to transfer my anaconda install?
Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment. This is useful when you want to reproduce an environment with limited or no internet access. All the previous methods download packages from their respective repositories to create an environment. Keep in mind that conda-pack is both platform and operating system specific and that the target computer must have the same platform and OS as the source computer.
To install conda-pack, make sure you are in the root or base environment so that it is available in sub-environments. Conda-pack is available at conda-forge or PyPI.
conda-forge:
conda install -c conda-forge conda-pack
PyPI:
pip install conda-pack
To package an environment:
# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
To install the environment:
# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
# Run Python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack
Source
I created an environment with the following command: virtualenv venv --distribute
I cannot remove it with the following command: rmvirtualenv venv -
This is part of virtualenvwrapper as mentioned in answer below for virtualenvwrapper
I do an lson my current directory and I still see venv
The only way I can remove it seems to be: sudo rm -rf venv
Note that the environment is not active. I'm running Ubuntu 11.10. Any ideas? I've tried rebooting my system to no avail.
"The only way I can remove it seems to be: sudo rm -rf venv"
That's it! There is no command for deleting your virtual environment. Simply deactivate it and rid your application of its artifacts by recursively removing it.
Note that this is the same regardless of what kind of virtual environment you are using. virtualenv, venv, Anaconda environment, pyenv, pipenv are all based the same principle here.
Just to echo what #skytreader had previously commented, rmvirtualenv is a command provided by virtualenvwrapper, not virtualenv. Maybe you didn't have virtualenvwrapper installed?
See VirtualEnvWrapper Command Reference for more details.
Use rmvirtualenv
Remove an environment, in the $WORKON_HOME.
Syntax:
rmvirtualenv ENVNAME
You must use deactivate before removing the current environment.
$ rmvirtualenv my_env
Reference: http://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html
You can remove all the dependencies by recursively uninstalling all of them and then delete the venv.
Edit including Isaac Turner commentary
source venv/bin/activate
pip freeze > requirements.txt
pip uninstall -r requirements.txt -y
deactivate
rm -r venv/
If you are using pyenv, it is possible to delete your virtual environment:
$ pyenv virtualenv-delete <name>
Simply delete the virtual environment from the system:
rm -rf venv
(There's no special command for it)
from virtualenv's official document https://virtualenv.pypa.io/en/latest/user_guide.html
Removing an Environment
Removing a virtual environment is simply done by deactivating it and deleting the environment folder with all its contents:
(ENV)$ deactivate
$ rm -r /path/to/ENV
1. Remove the Python environment
There is no command to remove a virtualenv so you need to do that by hand, you will need to deactivate if you have it on and remove the folder:
deactivate
rm -rf <env path>
2. Create an env. with another Python version
When you create an environment the python uses the current version by default, so if you want another one you will need to specify at the moment you are creating it. To make and env. with Python 3.X called MyEnv just type:
python3.X -m venv MyEnv
Now to make with Python 2.X use virtualenv instead of venv:
python2.X -m virtualenv MyEnv
3. List all Python versions on my machine
If any of the previous lines of code didn't worked you probably don't have the specific version installed. First list all your versions with:
ls -ls /usr/bin/python*
If you didn't find it, install Python 3.X using apt-get:
sudo apt-get install python3.X
I used pyenv uninstall my_virt_env_name to delete the virual environment.
Note: I'm using pyenv-virtualenv installed through the install script.
The following command works for me.
rm -rf /path/to/virtualenv
If you are a Windows user and you are using conda to manage the environment in Anaconda prompt, you can do the following:
Make sure you deactivate the virtual environment or restart Anaconda Prompt. Use the following command to remove virtual environment:
$ conda env remove --name $MyEnvironmentName
Alternatively, you can go to the
C:\Users\USERNAME\AppData\Local\Continuum\anaconda3\envs\MYENVIRONMENTNAME
(that's the default file path) and delete the folder manually.
Actually requires two deletions.
The project folder which everyone in this thread already said you simply delete manually or using rm -r projectfoldername
But then you also need to delete the actual virtualenv located in macOS /Users/edison/.pyenv/versions/3.8.0/envs/myspecialenv.
You can do that by doing pyenv virtualenv-delete myspecialenv or manual removal.
if you are windows user, then it's in C:\Users\your_user_name\Envs. You can delete it from there.
Also try in command prompt rmvirtualenv environment name.
I tried with command prompt so it said deleted but it was still existed. So i manually delete it.
cd \environmentfolder_name\Scripts\deactivate.bat
deactivate is the command you are looking for. Like what has already been said, there is no command for deleting your virtual environment. Simply deactivate it!
If you're a windows user, you can also delete the environment by going to: C:/Users/username/Anaconda3/envs Here you can see a list of virtual environment and delete the one that you no longer need.
If you are using pyenv virtualenv < https://github.com/pyenv/pyenv > to centrally manage python versions and virtual environment the solution would be
pyenv uninstall some_env
(Assuming that you have set up your bash .szh profile correctly.)
The solution to this issue is also answered here:
https://github.com/pyenv/pyenv-virtualenv/issues/17
Hope this helps 👍🏻
Just use Anaconda Navigator to remove selected env.
It is possible that some resources will be activated, making it impossible to just delete the directory. All Python processes should be stopped in advance:
pkill -9 python
rm -rf venv
You can follow these steps to remove all the files associated with virtualenv and then reinstall the virtualenv again and using it
cd {python virtualenv folder}
find {broken virtualenv}/ -type l ## to list out all the links
deactivate ## deactivate if virtualenv is active
find {broken virtualenv}/ -type l -delete ## to delete the broken links
virtualenv {broken virtualenv} --python=python3 ## recreate links to OS's python
workon {broken virtualenv} ## activate & workon the fixed virtualenv
pip3 install ... {other packages required for the project}
For the new versions do:
conda deactivate
conda env remove -n env_name
step 1: delete virtualenv virtualenvwrapper by copy and paste the following command below:
$ sudo pip uninstall virtualenv virtualenvwrapper
step 2: go to .bashrc and delete all virtualenv and virtualenvwrapper
open terminal:
$ sudo nano .bashrc
scroll down and you will see the code bellow then delete it.
# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh
next, source the .bashrc:
$ source ~/.bashrc
FINAL steps: without terminal/shell go to /home and find .virtualenv (I forgot the name so if your find similar to .virtualenv or .venv just delete it. That will work.