Google colab conda env [duplicate] - pytorch

Question

In Google Colaboratory, I can install a new library using !pip install package-name. But when I open the notebook again tomorrow, I need to re-install it every time.
Is there a way to install a library permanently? No need to spend time installing every time to use?

Answer 1

Yes. You can install the library in Google Drive. Then add the path to sys.path.
import os, sys
from google.colab import drive
drive.mount('/content/drive')
nb_path = '/content/notebooks'
os.symlink('/content/drive/My Drive/Colab Notebooks', nb_path)
sys.path.insert(0,nb_path)
Then you can install a library, for example, jdc, and specify the target.
!pip install --target=$nb_path jdc
Later, when you run the notebook again, you can skip the !pip install line. You can just import jdc and use it. Here's an example notebook.
https://colab.research.google.com/drive/1KpMDi9CjImudrzXsyTDAuRjtbahzIVjq
BTW, I really like jdc's %%add_to. It makes working with a big class much easier.

Answer 2

If you want a no-authorization solution. You can use mounting with gcsfuse + service-account key embedded in your notebook. Like this:
# first install gcsfuse
%%capture
!echo "deb http://packages.cloud.google.com/apt gcsfuse-bionic main" > /etc/apt/sources.list.d/gcsfuse.list
!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
!apt update
!apt install gcsfuse
Then get your service account credential from google cloud console and embed it in the notebook
%%writefile /key.json
{
"type": "service_account",
"project_id": "kora-id",
"private_key_id": "xxxxxxx",
"private_key": "-----BEGIN PRIVATE KEY-----\nxxxxxxx==\n-----END PRIVATE KEY-----\n",
"client_email": "colab-7#kora-id.iam.gserviceaccount.com",
"client_id": "100380920993833371482",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/colab-7%40kora-id.iam.gserviceaccount.com"
}
Then set environment to look for this credential file
%env GOOGLE_APPLICATION_CREDENTIALS=/key.json
You must then create (or have it already) a gcs bucket. And mount it to a made-up directory.
!mkdir /content/my-bucket
!gcsfuse my-bucket /content/my-bucket
Then finally, install the library there. Like my above answer.
import sys
nb_path = '/content/my-bucket'
sys.path.insert(0, nb_path)
# Do this just once
!pip install --target=$nb_path jdc
You can now import jdc without !pip install it next time.

Answer 3

In case you need to install multiple libraries here is a snippet:
def install_library_to_drive(libraries_list):
""" Install library on gdrive. Run this only once. """
drive_path_root = 'path/to/mounted/drive/directory/where/you/will/install/libraries'
for lib in libraries_list:
drive_path_lib = drive_path_root + lib
!pip install -q $lib --target=$drive_path_lib
sys.path.insert(0, drive_path_lib)
def load_library_from_drive(libraries_list):
""" Technically, it just appends install dir to a sys.path """
drive_path_root = 'path/to/mounted/drive/directory/where/you/will/install/libraries'
for lib in libraries_list:
drive_path_lib = drive_path_root + lib
sys.path.insert(0, drive_path_lib)
libraries_list = ["torch", "jsonlines", "transformers"] # list your libraries
install_library_to_drive(libraries_list) # Run this just once
load_library_from_drive(libraries_list)