I built a simple ymal file for Azure pipeline.
Tasks:
Build wheel file during Azure pipeline run
Copy wheel file just created in the pipeline run to databricks' dbfs
It does write .whl file somewhere as I can see in scrip print in the pipeline, but it doesn't overwrite into the repo or I don't know where the file created is?
stages:
- stage: Build
jobs:
- job:
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'install/update pip and wheel'
python -m pip install --upgrade pip wheel
echo 'building wheel file'
python src/databricks/setup.py bdist_wheel
displayName: 'Build updated wheel file'
- stage: Deploy
jobs:
- job:
steps:
- script: |
python src/databricks/setup.py bdist_wheel
echo 'installing databricks-cli'
python -m pip install --upgrade databricks-cli
displayName: 'install databricks cli'
- script: |
echo 'copying wheel file'
databricks fs cp --overwrite src/databricks/dist/library-0.0.2-py3-none-any.whl dbfs:/FileStore/whl
echo 'copying main.py'
databricks fs cp --overwrite src/databricks/main.py dbfs:/FileStore/whl
env:
DATABRICKS_HOST: $(DATABRICKS_HOST)
DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
displayName: 'Copy wheel file and main.py to dbfs'
Related
I have the following azure task that copies my .env file from secure files (library) in Azure to the working diretory from which I execte unit tests:
steps:
- task: DownloadSecureFile#1
inputs:
secureFile: '.env'
displayName: "Download .env file"
- task: CopyFiles#2
inputs:
sourceFolder: "$(Agent.TempDirectory)"
contents: ".env"
targetFolder: "$(System.DefaultWorkingDirectory)"
displayName: "Copy .env"
- script: |
cd /$(System.DefaultWorkingDirectory)
sudo apt-get update
sudo apt-get install -y python3-dev default-libmysqlclient-dev build-essential unixodbc-dev
pip3 install -r requirements.txt
pip3 install pytest pytest-azurepipelines
pip3 install pytest-cov
python3 -m pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml
However when running pytest its failing because Python cannot find the environment variables from the file. Is there something I am missing that I need to do so that Pytest can successfully run the tests with all the environment variables that are coming from the .env file?
I am trying to set up a CI/CD pipeline for a python project using a Windows runner on GitLab.
However, when executing pytest, pytest collects 10 items, and opens the first test file. After that, the pipeline continues running, but nothing happens and the pipeline stops after time-out. All test work correctly locally and take around 30 seconds in total.
The root directoty for pytest is correct.
This is my Gitlab yml file:
image: python:3.11.0-buster
cache:
paths:
- .cache/pip
- venv/
tests:
tags:
- windows
before_script:
- mkdir -p ~/.ssh
- echo "$DEPLOY_KEY" > ~/.ssh/id_rsa
- echo "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- Import-Module "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"
- choco install python --version=3.11 -y -f
# - dir
- C:\\Python311\\python.exe -m pip install --upgrade pip
- C:\\Python311\\python.exe -m pip install --upgrade setuptools
- C:\\Python311\\python.exe -m pip install -r requirements.txt
# - where python
- C:\\Python311\\python.exe -m pip install virtualenv
script:
- C:\\Python311\\python.exe -m pytest -p no:faulthandler
I've also tried - C:\Python311\python.exe -m pytest which had the same result
I followed the pip section on the Azure documentation on pipeline caching to speed up my Azure DevOps CI pipeline (in particular the dependency installation step). However, the packages are still installed every time I execute the pipeline (which I ideally also want to cache). How can I achieve this?
The Azure DevOps documentation is a bit lackluster here. Following the pip section just leads to caching of the wheels, not the installation itself (which you can also cache to further improve the pipeline execution time). To enable this, you need to work with a virtual environment (such as venv or a conda environment) and cache the entire environment.
Below you can find a code example with conda on how to cache an entire installed environment:
variables:
CONDA_ENV_NAME: "unit_test"
# set $(CONDA) environment variable to your conda path (pre-populated on 'ubuntu-latest' VMs)
CONDA_ENV_DIR: $(CONDA)/envs/$(CONDA_ENV_NAME)
steps:
- script: echo "##vso[task.prependpath]$CONDA/bin"
displayName: Add conda to PATH
- task: Cache#2
displayName: Use cached Anaconda environment
inputs:
key: 'conda | "$(Agent.OS)" | requirements.txt'
path: $(CONDA_ENV_DIR)
cacheHitVar: CONDA_CACHE_RESTORED
- bash: conda create --yes --quiet --name $(CONDA_ENV_NAME)
displayName: Create Anaconda environment
condition: eq(variables.CONDA_CACHE_RESTORED, 'false')
- bash: |
source activate $(CONDA_ENV_NAME)
pip install -r requirements.txt
displayName: Install dependencies
condition: eq(variables.CONDA_CACHE_RESTORED, 'false')
# Optional step here: Install your package (do not cache this step)
- bash: |
source activate $(CONDA_ENV_NAME)
pip install --no-deps .
pytest .
displayName: Install package and execute unit tests
Requesting your help with Azure Artifact connection from Azure Pipelines.
My Azure Pipeline is building a image from docker file and a 'requirements' file has list of packages to be pip installed. In the pipeline I authenticate to my Azure Artifacts feed using PipAuthenticate#1 task and the authentication is successfull and the URL is passed as an argument to the docker file.
However, I can see that the packages are getting installed from external links, but are not downloaded to my artifact feed.
The artifact feed 'testartifact' is currently empty and so it is correctly going to the external link to download the package. But I was expecting the package to be then saved in 'testartifact' feed so that next docker build, it takes the package directly from testartifact feed. Is my assumption correct?
If so, could you help on if I am missing something in the code due to which the package is not getting saved to my artifact.
Here is the Azure Pipeline yaml file and the docker file. Also attached the log of package download.
Thanks for your time!
pool:
vmImage: 'ubuntu-latest'
# Set variables
variables:
imageversion: 1.0
artifactFeed: testartifact
stages:
- stage: DevDeploy
jobs:
- job: DevBuildandPushImage
steps:
- bash: echo DevDeploy
- task: PipAuthenticate#1
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: $(artifactFeed)
onlyAddExtraIndex: true
- bash: echo "##vso[task.setvariable variable=artifactoryUrl;]$PIP_EXTRA_INDEX_URL"
- bash: echo $PIP_EXTRA_INDEX_URL
- task: Docker#2
inputs:
containerRegistry: 'testcontaineregistry'
repository: 'testrepository'
command: 'build'
Dockerfile: '**/dockerfile'
arguments: '--build-arg PIP_EXTRA_URL=$(PIP_EXTRA_INDEX_URL)'
Part of the dockerfile
ARG PIP_EXTRA_URL
ENV PIP_EXTRA_INDEX_URL=$PIP_EXTRA_URL
RUN echo 'PIP_EXTRA_INDEX_URL'$PIP_EXTRA_INDEX_URL
# Install Python Packages & Requirements
COPY requirements requirements
RUN pip3 install -r requirements --extra-index-url $PIP_EXTRA_URL
part of the log
2020-07-16T17:39:05.0301632Z Step 8/28 : RUN echo 'PIP_EXTRA_INDEX_URL'$PIP_EXTRA_INDEX_URL
2020-07-16T17:39:05.4787725Z PIP_EXTRA_INDEX_URLhttps://build:***#XXXXXXX.pkgs.visualstudio.com/_packaging/testartifact/pypi/simple
2020-07-16T17:39:06.1264997Z Step 9/28 : COPY requirements requirements
2020-07-16T17:39:07.0309036Z Step 10/28 : RUN pip3 install -r requirements --extra-index-url $PIP_EXTRA_URL
2020-07-16T17:39:08.3873873Z Collecting pypyodbc (from -r requirements (line 1))
2020-07-16T17:39:08.7139882Z Downloading https://files.pythonhosted.org/packages/ea/48/bb5412846df5b8f97d42ac24ac36a6b77a802c2778e217adc0d3ec1ee7bf/pypyodbc-1.3.5.2.zip
2020-07-16T17:39:08.9900873Z Collecting pyodbc (from -r requirements (line 2))
2020-07-16T17:39:09.2421266Z Downloading https://files.pythonhosted.org/packages/81/0d/bb08bb16c97765244791c73e49de9fd4c24bb3ef00313aed82e5640dee5d/pyodbc-4.0.30.tar.gz (266kB)
2020-07-16T17:39:09.4960835Z Collecting xlrd (from -r requirements (line 3))
2020-07-16T17:39:09.6500787Z Downloading https://files.pythonhosted.org/packages/b0/16/63576a1a001752e34bf8ea62e367997530dc553b689356b9879339cf45a4/xlrd-1.2.0-py2.py3-none-any.whl (103kB)
2020-07-16T17:39:09.6782714Z Collecting pandas (from -r requirements (line 4))
2020-07-16T17:39:10.2506552Z Downloading https://files.pythonhosted.org/packages/c0/95/cb9820560a2713384ef49060b0087dfa2591c6db6f240215c2bce1f4211c/pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl (10.1MB)
2020-07-16T17:39:11.4371150Z Collecting datetime (from -r requirements (line 5))
2020-07-16T17:39:11.6083120Z Downloading https://files.pythonhosted.org/packages/73/22/a5297f3a1f92468cc737f8ce7ba6e5f245fcfafeae810ba37bd1039ea01c/DateTime-4.3-py2.py3-none-any.whl (60kB)
2020-07-16T17:39:11.6289946Z Collecting azure-storage-blob (from -r requirements (line 6))
From the task log, the Python packages are restored from external links. You need to make sure that the packages are installed from Feed upstream source . Then the package will exist in the feed after installation.
Here are the steps:
Step1: Add the Python Upstream source to feed.
Step2: Use PipAuthenticate task to get the $PIP_EXTRA_INDEX_URL
Step3: Use the $PIP_EXTRA_INDEX_URL to install package from feed.
pip install -r requirements.txt --index-url $PIP_EXTRA_INDEX_URL
Note: The step 2 and 3 are already existing in your yaml file. But the pip install script seems to have issue. You need to directly add the --index-url parameter.
Then the packages are installed from feed upstream source.
In this case, these packages will also exist in the feed.
My vmImage is 'ubuntu-latest'.
When my build comes to nlohmann I get the following error:
-- Up-to-date: /usr/local/include
CMake Error at libraries/nlohmann_json/cmake_install.cmake:41 (file):
file INSTALL cannot set permissions on "/usr/local/include": Operation not
permitted.
Call Stack (most recent call first):
libraries/cmake_install.cmake:42 (include)
cmake_install.cmake:42 (include)
It works fine locally and also on the Windows and MacOS vmImage pipelines, so I am assuming this is some type of permissions issue/setting with DevOps?
The yaml file is as follows:
# Starter pipeline
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: UsePythonVersion#0
displayName: 'install python 3.x'
inputs:
versionSpec: '3.x'
addToPath: true
architecture: 'x64'
- task: CmdLine#2
displayName: 'install opengl'
inputs:
script: 'sudo apt-get -y install freeglut3 freeglut3-dev libglew1.5 libglew1.5-dev libglu1-mesa libglu1-mesa-dev libgl1-mesa-glx libgl1-mesa-dev'
failOnStderr: true
- task: CmdLine#2
displayName: 'install sdl'
inputs:
script: 'sudo apt-get install -y libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev libsdl2-net-dev'
failOnStderr: true
- task: CmdLine#2
displayName: 'update google gpg key'
inputs:
script: 'wget -q -O - curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -'
failOnStderr: false
- task: CmdLine#2
displayName: 'install gcc 10'
inputs:
script: 'sudo apt-get update && sudo add-apt-repository ppa:ubuntu-toolchain-r/test && sudo apt-get update && sudo apt-get -y install gcc-10 g++-10'
failOnStderr: true
- task: PythonScript#0
displayName: 'run build.py'
inputs:
scriptSource: 'filePath'
scriptPath: '$(Build.SourcesDirectory)/build.py'
failOnStderr: false
In the end, I disabled the install step of nlohmann json by doing the following before I added the subdirectory:
set(JSON_Install OFF CACHE INTERNAL "")