I have a conda environment at home that I'm using on my Ph.D., but now that I'm needing more computational power I have to transfer (or install a perfect copy) of my environment on one of the University's computers. The computers have no internet connection, all I have is SSH.
My attempt to copy the entire /anaconda3 directory and .bashrc to a path similar to the one I use at my home (/home/henrique/bin) have not worked.
What is the correct way to transfer my anaconda install?
Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment. This is useful when you want to reproduce an environment with limited or no internet access. All the previous methods download packages from their respective repositories to create an environment. Keep in mind that conda-pack is both platform and operating system specific and that the target computer must have the same platform and OS as the source computer.
To install conda-pack, make sure you are in the root or base environment so that it is available in sub-environments. Conda-pack is available at conda-forge or PyPI.
conda-forge:
conda install -c conda-forge conda-pack
PyPI:
pip install conda-pack
To package an environment:
# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
To install the environment:
# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
# Run Python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack
Source
Related
I have a Dockerfile where I try to activate python virtualenv after what, it should install all dependencies within this env. However, everything still gets installed globally. I used different approaches and non of them worked. I also do not get any errors. Where is a problem?
1.
ENV PATH $PATH:env/bin
2.
ENV PATH $PATH:env/bin/activate
3.
RUN . env/bin/activate
I also followed an example of a Dockerfile config for the python-runtime image on Google Cloud, which is basically the same stuff as above.
Setting these environment variables are the same as running source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
Additionally, what does ENV VIRTUAL_ENV /env mean and how it is used?
You don't need to use virtualenv inside a Docker Container.
virtualenv is used for dependency isolation. You want to prevent any dependencies or packages installed from leaking between applications. Docker achieves the same thing, it isolates your dependencies within your container and prevent leaks between containers and between applications.
Therefore, there is no point in using virtualenv inside a Docker Container unless you are running multiple apps in the same container, if that's the case I'd say that you're doing something wrong and the solution would be to architect your app in a better way and split them up in multiple containers.
EDIT 2022: Given this answer get a lot of views, I thought it might make sense to add that now 4 years later, I realized that there actually is valid usages of virtual environments in Docker images, especially when doing multi staged builds:
FROM python:3.9-slim as compiler
ENV PYTHONUNBUFFERED 1
WORKDIR /app/
RUN python -m venv /opt/venv
# Enable venv
ENV PATH="/opt/venv/bin:$PATH"
COPY ./requirements.txt /app/requirements.txt
RUN pip install -Ur requirements.txt
FROM python:3.9-slim as runner
WORKDIR /app/
COPY --from=compiler /opt/venv /opt/venv
# Enable venv
ENV PATH="/opt/venv/bin:$PATH"
COPY . /app/
CMD ["python", "app.py", ]
In the Dockerfile example above, we are creating a virtualenv at /opt/venv and activating it using an ENV statement, we then install all dependencies into this /opt/venv and can simply copy this folder into our runner stage of our build. This can help with with minimizing docker image size.
There are perfectly valid reasons for using a virtualenv within a container.
You don't necessarily need to activate the virtualenv to install software or use it. Try invoking the executables directly from the virtualenv's bin directory instead:
FROM python:2.7
RUN virtualenv /ve
RUN /ve/bin/pip install somepackage
CMD ["/ve/bin/python", "yourcode.py"]
You may also just set the PATH environment variable so that all further Python commands will use the binaries within the virtualenv as described in https://pythonspeed.com/articles/activate-virtualenv-dockerfile/
FROM python:2.7
RUN virtualenv /ve
ENV PATH="/ve/bin:$PATH"
RUN pip install somepackage
CMD ["python", "yourcode.py"]
Setting this variables
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
is not exactly the same as just running
RUN . env/bin/activate
because activation inside single RUN will not affect any lines below that RUN in Dockerfile. But setting environment variables through ENV will activate your virtual environment for all RUN commands.
Look at this example:
RUN virtualenv env # setup env
RUN which python # -> /usr/bin/python
RUN . /env/bin/activate && which python # -> /env/bin/python
RUN which python # -> /usr/bin/python
So if you really need to activate virtualenv for the whole Dockerfile you need to do something like this:
RUN virtualenv env
ENV VIRTUAL_ENV /env # activating environment
ENV PATH /env/bin:$PATH # activating environment
RUN which python # -> /env/bin/python
Although I agree with Marcus that this is not the way of doing with Docker, you can do what you want.
Using the RUN command of Docker directly will not give you the answer as it will not execute your instructions from within the virtual environment. Instead squeeze the instructions executed in a single line using /bin/bash. The following Dockerfile worked for me:
FROM python:2.7
RUN virtualenv virtual
RUN /bin/bash -c "source /virtual/bin/activate && pip install pyserial && deactivate"
...
This should install the pyserial module only on the virtual environment.
If you your using python 3.x :
RUN pip install virtualenv
RUN virtualenv -p python3.5 virtual
RUN /bin/bash -c "source /virtual/bin/activate"
If you are using python 2.x :
RUN pip install virtualenv
RUN virtualenv virtual
RUN /bin/bash -c "source /virtual/bin/activate"
Consider a migration to pipenv - a tool which will automate virtualenv and pip interactions for you. It's recommended by PyPA.
Reproduce environment via pipenv in a docker image is very simple:
FROM python:3.7
RUN pip install pipenv
COPY src/Pipfile* ./
RUN pipenv install --deploy
...
I want to create a bash script to install a new virtual environment "ABC" in conda. But before I go ahead and run a command to create this env, I want to check if conda is already installed on the machine. If not installed, I want to install miniconda and then create the env "ABC". If conda is already installed then I would just go ahead and create the environment. (All this should happen within the same script)
I just want to know if it is possible to check the existence of conda within a bash script and then proceed with the rest of the installations?
'''
#!/bin/bash
<code_to_check_existence_of_conda_env_here ?>
//If it does not exist, I will run the below code
mkdir -p miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O
miniconda3/miniconda.sh
bash miniconda3/miniconda.sh -b -u -p ~/miniconda3
conda env update -f my_env.yml
'''
The main issue here is that Conda has multiple components. Which components are loaded depends on the how Conda is installed and what user the BASH script is executing under. I'll try describing the components and hopefully you can decide what is suitable to verify.
Conda Components
1: Conda Package
The first is a Python package conda, which is installed in the Anaconda/Miniconda/Miniforge base environment. If the base environment is activated, one could test
python -m conda
which will give something like
/path/to/python: No module named conda
if it isn't there. Otherwise, it outputs the conda entrypoint's documentation.
2: Conda Entrypoint
The entrypoint conda, which acts as a CLI, is located under the condabin directory of the Anaconda/Miniconda/Miniforge installation. When a user runs conda init, a managed section is added to their shell initialization file (.bashrc for Linux BASH), that includes code to add the condabin to PATH. This is most likely what OP wants to identify, however, running with the shebang /bin/bash will not load the .bashrc file. Instead, one should probably be using
#!/bin/bash -l
or
#!/usr/bin/env bash -l
Then the entrypoint can be located with
which conda
3: Conda Activate
Finally, Conda also includes some shell-only functions, which are defined in the aforementioned shell initialization code. This sets up a middleman shell function, also called conda (essentially an alias), which can be viewed with
type conda
This function serves to determine whether the conda (de)?activate commands are being requested, which are pure shell functions, or something that needs to be forwarded to the entrypoint.
Recommendation
Were I designing this, I would write an interactive script that checks for #2 (which conda) and if that comes up blank then prompt the user to either provide the PATH to the Conda entrypoint (maybe they installed it in a weird place or didn't run conda init) or install Miniforge.1
I also would not use the base environment to install stuff - that is a bad idea for an end-user, let alone a third-party - and instead create a new environment. I would prompt the user with a specific default environment name, but also provide an option for them to customize.
[1] Yes, Miniforge, not Miniconda. Commercial use of the Anaconda defaults channels now requires a paid license, so better to use the free Miniforge.
I'm not really good at bash scripting but I would execute a command:
pip3 freeze | grep conda
and based on the output (if exit status was an error or not) install it or directly create the environment.
run the command conda list on your machine, the output will of be different when its installed vs not installed, then run if statement on the output.
In bash, if we want to check if a software is installed or not, the following check is used.
if dpkg -l $SOFTWARE; then
<Do stuff>
ensureconda
This sounds like the intended problem that the experimental tool ensureconda was designed to solve. This, however, requires a Python installation with pip:
pip install ensureconda
Here are the command options:
$ ensureconda --help
Usage: ensureconda [OPTIONS]
Ensures that a conda/mamba is installed.
Options:
--mamba / --no-mamba search for mamba
--micromamba / --no-micromamba search for micromamba, install if not
present
--conda / --no-conda search for conda
--conda-exe / --no-conda-exe search for conda.exe / conda-standalone,
install if not present
--no-install don't install conda/mamba if no version can
be discovered
--min-conda-version VERSIONNUMBER
minimum version of conda to accept (defaults
to 4.8.2)
--min-mamba-version VERSIONNUMBER
minimum version of mamba/micromamba to
accept (defaults to 0.7.3)
--help Show this message and exit.
Type conda activate
or conda activate env_name
The first command will activate the base environment if it is already installed. The second command will do activate the user created environment if it is installed.
For some reason virtualenv is not in my path after installing with pip3. I have a fresh install of ubuntu 16.04.
sudo apt-get install pip3
pip3 install virtualenv
virtualenv # command not found!!!
edit: I also installed jupyter notebook with pip3 and its not in the path either.
Python executables are placed in ~/.local/bin/ on Ubuntu 16.04.
This location is not in $PATH so edit your .bashrc to append it there.
# .bashrc file
export PATH=$PATH:~/.local/bin
This is Ubuntu only (didn't check other distros)
TL;DR (if you used pip to install pkg) run following command
$ source ~/.profile
If you examine .profile there is a script looks like following.
(18 version. 16 version has something different)
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
PATH="$HOME/.local/bin:$PATH"
fi
which means everything under ~/.local/bin will be added to PATH.
So, if you used pip to install pkg and try to run from prompt. As long as pip created file under the folder it will let you run commands without full path.
You can restart the session too. Whichever you feel comfortable with.
I built Python 3.3.0 from source on my Ubuntu 13.10 laptop.
When using the /usr/bin/virtualenv -p /python3.3.0/bin/python3 foo_virt command to create a virtual environemnt, I see no modules installed when running pip freeze, which is the behavior I expect.
When using /python3.3.0/bin/python3 -m venv foo_virt, I see tons of modules installed:
(foo_virt) user#laptop:/foo_virt$ /usr/bin/pip freeze --local
Jinja2==2.7
Mako==0.8.1
MarkupSafe==0.15
PAM==0.4.2
Pillow==2.0.0
Pygments==1.6
SecretStorage==1.0.0
... (total of 75 modules listed)
I tried then to install pip for that specific version of Python, by running, as per the module's documentation: python3 get-pip.py. But I still see all these modules:
(foo_virt) user#laptop:/foo_virt$ which pip
/foo_virt/bin/pip
(foo_virt) user#laptop:/foo_virt$ pip freeze --local
Jinja2==2.7
Mako==0.8.1
MarkupSafe==0.15
PAM==0.4.2
Pillow==2.0.0
Pygments==1.6
SecretStorage==1.0.0
... (still 75 modules)
How do I use venv so no modules are installed in the virtual environment? I didn't find any option in the documentation to help me. Also, this issue is not happening on Windows 7. Thanks!
bash caches commands found by searching the PATH. You can see the current cache by entering hash. Adding -r resets the cache. -d will delete an individual name. Sourcing the activate script should reset the cache:
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "$BASH" -o -n "$ZSH_VERSION" ] ; then
hash -r
fi
Maybe you ran the system pip before get-pip.py. In that case hash -d pip solves the problem.
I've already set up a Git repository on GitHub and committed a few changes from my Windows machine.
But tomorrow I'll have to work in this repository from a machine running Ubuntu with limited privilege (i.e. no sudo).
Is there a portable version of Git for Linux? Or some source that allows me to compile and install Git only for the current user?
You can download the git source and do ./configure --prefix=/home/user/myroot && make && make install to install git to your home directory provided you have the build tools. If you don't have the build-essential package installed (dpkg --list|grep build-essential), you will need to install those to your home directory as well.
I don't like link-only answers, but this link I followed step-by-step on a Fedora machine and it worked without modification. Very, very easy. The binaries end up in your ~/bin directory. You download a tarball, extract the sources, run make and run make install and that is it.
As the author states, the 2 prerequisites are gcc and ssh and if you meet these git should work for you as a non-root user.
To install git and dependencies from source the following maybe useful.
Replace with the location you are installing your non-root apps and consider checking for latest versions of source code.
wget https://curl.haxx.se/download/curl-7.47.1.tar.gz
tar -xf curl-7.47.1.tar.gz
mkdir <local_curl_dir>
cd curl-7.47.1
./configure --prefix=<local_curl_dir>
make
make install
wget http://downloads.sourceforge.net/expat/expat-2.1.0.tar.gz
tar -xf expat-2.1.0.tar.gz
mkdir <local_expat_dir>
cd expat-2.1.0
./configure --prefix=<local_expat_dir>
make
make install
wget https://github.com/git/git/archive/v2.6.4.tar.gz
tar -xf v2.6.4
mkdir <local_git_dir>
cd git-2.6.4
make configure
./configure --prefix=<local_git_dir>/git --with-curl=<local_curl_dir>/curl --with-expat=<local_expat_dir>/expat
make
make install
This is what I ended up doing, the main trick being the make flags:
wget -O git.tar.gz https://github.com/git/git/archive/v2.17.0.tar.gz
tar zxf git.tar.gz
mv git-2.17.0 git
cd git
make configure
./configure --prefix=`pwd` --with-curl --with-expat
# ./configure --prefix=`pwd`
# Make flags from https://public-inbox.org/git/CAP8UFD2gKTourXUdB_9_FZ3AEECTDc1Fx1NFKzeaTZDWHC3jxA#mail.gmail.com/
make NO_GETTEXT=Nope NO_TCLTK=Nope
make install NO_GETTEXT=Nope NO_TCLTK=Nope
Credits:
79E09796's answer above was a good tip, but didn't work for my case on Cloudways and did not require compiling curl and expat.
A random email record I found on the internet: https://public-inbox.org/git/CAP8UFD2gKTourXUdB_9_FZ3AEECTDc1Fx1NFKzeaTZDWHC3jxA#mail.gmail.com/
A related answer is https://askubuntu.com/a/350.
I could get it work with the third method proposed:
apt-get source git
cd git_vXXX
./configure --prefix=$HOME
make
make install
I don't know why, but when I had tried to install from the source download from github instead, I had a lot of problems with missing dependencies
Overkill workaround
Install Anaconda as a user and install git with conda.
Advantages
Anaconda can be installed as user, and a conda environment can be created which can help you to install other packages. This way you don't need to compile git from source, nor you need to install libcurl and perl, so you won't get the error
git: 'remote-https' is not a git command. See 'git --help'
after successfully compiling git.
Steps to install Anaconda, then git
read the Anaconda installation manual which points to the download page's shell script file.
Download the script file:
copy to your local machine and then copy with scp (e.g. winscp) to the Linux machine or
use a terminal on the Linux machine and issue wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh.
add executable rights to yourself on the file by issuing chmod +x Anaconda3-2020.11-Linux-x86_64.sh
follow the installation instructions where you can also specify into which folder you want to install anaconda
after installation, chose one of the possibilities below to successfully invoke git later:
you either activate a conda environment, which is useful e.g. if you need to create different conda environments (it is common if you work with python). To activate an environment, you need to ask the installer (at the end of the installation) to add the conda initialization code into your ~/.bashrc file [see below]. Using this method, your path won't be polluted, and you will see if conda-related binaries are also in your current path. Or you can
add the installed bin's folder to your path, e.g. if you installed anaconda into /home/myusername/anaconda3, it will be in /home/myusername/anaconda3/bin. Your new executable file conda will be also there which will help you to install packages like git, python or pandoc, or
cd into the binary folder of anaconda, e.g. cd /home/myusername/anaconda3/bin, and execute the commands below.
don't forget to take into effect the new settings by, e.g., closing and opening the terminal again if you selected method 1. or 2. in point 5. If you selected 1, you will see something like (base) myusername#servername indicating you are using the base conda environment.
Now you can install git using conda by issuing conda install -c anaconda git.
Your .bashrc will contain likes like this if you told the Anaconda installer to initialize conda for you:
# content of your .bashrc in your home dir
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/myusername/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/myusername/anaconda3/etc/profile.d/conda.sh" ]; then
. "/home/myusername/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/home/myusername/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
for the latest version(which i mean git-2.25.0-rc1 or upper), you need to
wget https://github.com/git/git/releases/tag/v2.25.0-rc1 -O git.zip
unzip git.zip
cd git-2.25.0-rc1
export PATH=`pwd`:$PATH
and of course, you can add the last line into your .bashrc or .zshrc or something else for more convenience.