Dependency caching in Python CI pipeline in Azure DevOps? - azure

Question

I followed the pip section on the Azure documentation on pipeline caching to speed up my Azure DevOps CI pipeline (in particular the dependency installation step). However, the packages are still installed every time I execute the pipeline (which I ideally also want to cache). How can I achieve this?

Answer 1

The Azure DevOps documentation is a bit lackluster here. Following the pip section just leads to caching of the wheels, not the installation itself (which you can also cache to further improve the pipeline execution time). To enable this, you need to work with a virtual environment (such as venv or a conda environment) and cache the entire environment.
Below you can find a code example with conda on how to cache an entire installed environment:
variables:
CONDA_ENV_NAME: "unit_test"
# set $(CONDA) environment variable to your conda path (pre-populated on 'ubuntu-latest' VMs)
CONDA_ENV_DIR: $(CONDA)/envs/$(CONDA_ENV_NAME)
steps:
- script: echo "##vso[task.prependpath]$CONDA/bin"
displayName: Add conda to PATH
- task: Cache#2
displayName: Use cached Anaconda environment
inputs:
key: 'conda | "$(Agent.OS)" | requirements.txt'
path: $(CONDA_ENV_DIR)
cacheHitVar: CONDA_CACHE_RESTORED
- bash: conda create --yes --quiet --name $(CONDA_ENV_NAME)
displayName: Create Anaconda environment
condition: eq(variables.CONDA_CACHE_RESTORED, 'false')
- bash: |
source activate $(CONDA_ENV_NAME)
pip install -r requirements.txt
displayName: Install dependencies
condition: eq(variables.CONDA_CACHE_RESTORED, 'false')
# Optional step here: Install your package (do not cache this step)
- bash: |
source activate $(CONDA_ENV_NAME)
pip install --no-deps .
pytest .
displayName: Install package and execute unit tests