I am trying to execute the Selenium test in my self hosted VM from Azure pipelines. I am getting below error message.
##[error]Version spec 3.8 for architecture x64 did not match any version in Agent.ToolsDirectory.
Below is my Yaml
trigger:
- main
pool: default strategy: matrix:
Python38:
python.version: '3.8'
addToPath: true
steps:
- task: UsePythonVersion#0 inputs:
versionSpec: '$(python.version)'
addToPath: true
architecture: 'x64' displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt displayName: 'Install dependencies'
- script: |
pip install pytest-azurepipelines
pytest tests\test_smoke\test_suite_SmokeTest.py -s -v --browser Chrome --html=report.html --reruns 2 displayName: 'pytest'
- task: DownloadPipelineArtifact#2 inputs:
buildType: 'specific'
project: '8801f21d-f4e9-4b65-b01e-68baf825747c'
definition: '5'
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
I have configured the agent in my VM. I have also installed the python 3.8.6 in VM and configured the python path for it.
Unable to run Python Selenium Tests on Self Hosted Azure VM
To use this task on the private agent, we need to add the desired Python version to the tool cache on the self-hosted agent in order for the task to use it.
Normally the tool cache is located under the _work/_tool directory of the agent or you could use the command line task to echo the variable $(Agent.ToolsDirectory) to get the path.
Then, under that directory, create the following directory structure based off of your Python version:
$AGENT_TOOLSDIRECTORY/
Python/
{version number}/
{platform}/
{tool files}
{platform}.complete
The version number should follow the format of 1.2.3. The platform
should either be x86 or x64. The tool files should be the unzipped
Python version files. The {platform}.complete should be a 0 byte file
that looks like x86.complete or x64.complete and just signifies the
tool has been installed in the cache properly.
My test directory structure:
$AGENT_TOOLSDIRECTORY/
Python/
3.9.0/
x64/
python-3.9.0-amd64.exe
x64.complete
Now, it works fine on my Self Hosted agent:
You could refer the FAQ in this document for some more details.
Related
I am trying to build a pipeline azure DevOps. I build a basic Flask website and wrote a Unittest script for it. It basically all works perfect. When i commit to azure repos the pipeline will do his thing and the test will run. The thing i want is to see the test results, i see all these tutorials for Pytest but not for Unittest.
trigger:
- Development
jobs:
- job: 'Test'
pool:
vmImage: 'ubuntu-latest' # other options: 'macOS-latest', 'windows-latest'
strategy:
matrix:
Python37:
python.version: '3.7'
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
- script: |
python -m pip install --upgrade pip
python -m pip install -e .
displayName: 'Install dependencies'
- script: |
python -m unittest discover -p "*.py" > results.txt
displayName: unittesting
This is my Yaml file that runs the pipeline.
This is how my results look when running the pipeline.
Pipeline results
Is there a way to publish these results with Unittest and have them in azure pipeline.
You'll first need to make your test script generate results in a format that DevOps can understand, i.e. JUnit XML.
There's an example in MS docs (which includes coverage as well):
- script: |
pip install pytest pytest-azurepipelines
pip install pytest-cov
pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml
displayName: 'pytest'
Alternatively, using unittest-xml-reporting should give you results in JUnit XML format as well.
Once you have that, then you can use Publish Test Results task to upload results and make them visible in DevOps UI, i.e.:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-*.xml'
If your unit tests are written using unittest module, dont worry
pytest can run unittest as well !!!
Add below yaml code snippet to your azure pipeline.
Run your unittest with pytest
Generate Junit XML output
Publish the output back to azure pipeline
- script: |
cd $(Build.Repository.LocalPath)
python -m pytest $(Build.Repository.LocalPath)/<unit_tests_path>/*.py --junitxml=test-unit.xml
displayName: 'Run Unit Tests'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'test-unit.xml'
I have created an Azure agent environment to a virtual machine where I download builds to the folder. I have the following yaml file
# Python package
# Create and test a Python package on multiple Python versions. OK
# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/python
trigger:
paths:
include:
- Flytteordre
pool:
vmImage: 'ubuntu-latest'
name: Azure Pipelines
variables:
python.version: '3.6'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: build
displayName: build
steps:
- task: UsePythonVersion#0
displayName: 'Use Python $(python.version) copy'
inputs:
versionSpec: '$(python.version)'
# We actually don't need to install these dependencies here. It needs to happen in the deploy yaml file.
- task: CmdLine#2
inputs:
script: |
python -m pip install --upgrade pip
python -m pip install selenium
python -m pip install pdfplumber
python -m pip install pandas
displayName: 'Install dependencies'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: dist'
inputs:
PathtoPublish: Flytteordre
ArtifactName: dist
- stage: Deployment
displayName: Deployment stage
jobs:
- deployment: deploymentJob
displayName: Deploy
environment:
name: Production
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- download: none
- downloadBuild: none
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'dist'
downloadPath: 'C:/azagent/A1/_work/myfolder/'
My problem is that each time I run the pipeline, it creates a folder inside the _work environment called 1, 2 etc.
I would like to avoid this so that I am in full control of which folders are created. As you can see I have indicated that my artifact should by downloaded to the folder path C:/azagent/A1/_work/myfolder/, however this will create two folders. One is the folder that I have indicated but the other is a folder with the title of a number. I now that this is the default but I would like to know if there is a way to turn off this default setting or at the very least to be able to change the predefined path variable PIPELINE.WORKSPACE or Agent.BuildDirectory?
How to avoid creating new folder in Azure Agent for Pipeline in Environment
According to the document Agent variables:
So, each build definition goes into its own directory within the agent's working directory.
As we know, a pipeline has input and output. Whenever we create a new pipeline, we will increment a number to name the new work folder created. The advantage of this is that multiple running builds sharing the same copy of the repository are guaranteed to step on each other sooner or later.
I would prefer the option of being able to name the directory myself
in case I need to find a particular project and open it on the local
agent.
If you want open it on the local agent, you could just use the Agent variables to show the path:
Agent.BuildDirectory
I deploy Azure functions using yaml scripts.
For some reasons, I do not put all my package requirements in the file requirements.txt, as this is used by some other processes.
Yet, when deploying my web app through a YAML pipeline, I want to install additional python packages. However, the resulting app that is deployed crashes with errors saying that it does not have those packages installed.
my pipeline:
# Python to Linux Web App on Azure
# Build your Python project and deploy it to Azure as a Linux Web App.
# Change python version to one thats appropriate for your application.
# https://learn.microsoft.com/azure/devops/pipelines/languages/python
trigger:
- master
variables:
# Azure Resource Manager connection created during pipeline creation
azureServiceConnectionId: .........................
# Web app name
webAppName: .........................
# Agent VM image name
vmImageName: 'ubuntu-latest'
# Environment name
environmentName: .........................
# Project root folder. Point to the folder containing manage.py file.
projectRoot: $(System.DefaultWorkingDirectory)
# Python version: 3.8
pythonVersion: '3.8'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: BuildJob
pool:
vmImage: $(vmImageName)
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(pythonVersion)'
displayName: 'Use Python $(pythonVersion)'
- script: |
python -m venv antenv
source antenv/bin/activate
python -m pip install --upgrade pip
pip install setup
pip install -r requirements.txt
pip install pyodbc
workingDirectory: $(projectRoot)
displayName: "Install requirements"
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(projectRoot)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- upload: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
displayName: 'Upload package'
artifact: drop
- stage: Deploy
displayName: 'Deploy Web App'
dependsOn: Build
condition: succeeded()
jobs:
- deployment: DeploymentJob
pool:
vmImage: $(vmImageName)
environment: $(environmentName)
strategy:
runOnce:
deploy:
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(pythonVersion)'
displayName: 'Use Python version'
- task: AzureWebApp#1
displayName: 'Deploy Azure Web App : .........................
inputs:
azureSubscription: $(azureServiceConnectionId)
appName: $(webAppName)
package: $(Pipeline.Workspace)/drop/$(Build.BuildId).zip
startUpCommand: 'startup-nocontainer.sh'
Here, pyodbc is installed separately in the pipeline, and not present in the file requirements.txt.
Checking the logs of the pipeline, I can see that the installation was performed successfully.
Yet when the app starts, it crashes at first import of pyodbc, as if the app that is eventually deployed only relyed on requirements.txt.
Any idea?
I think pyodbc is not being installed in the venv you create. The venv contains only the packages you specify in requirements.txt
Check the solutions proposed in this old post
I am trying to copy a file to Azure Databricks DBFS through Azure Devops pipeline. The following is a snippet from the yml file I am using:
stages:
- stage: MYBuild
displayName: "My Build"
jobs:
- job: BuildwhlAndRunPytest
pool:
vmImage: 'ubuntu-16.04'
steps:
- task: UsePythonVersion#0
displayName: 'Use Python 3.7'
inputs:
versionSpec: '3.7'
addToPath: true
architecture: 'x64'
- script: |
pip install pytest requests setuptools wheel pytest-cov
pip install -U databricks-connect==7.3.*
displayName: 'Load Python Dependencies'
- checkout: self
persistCredentials: true
clean: true
- script: |
echo "y
$(databricks-host)
$(databricks-token)
$(databricks-cluster)
$(databricks-org-id)
8787" | databricks-connect configure
databricks-connect test
env:
databricks-token: $(databricks-token)
displayName: 'Configure DBConnect'
- script: |
databricks fs cp test-proj/pyspark-lib/configs/config.ini dbfs:/configs/test-proj/config.ini
I get the following error at the stage where I am invoking the databricks fs cp command:
/home/vsts/work/_temp/2278f7d5-1d96-4c4e-a501-77c07419773b.sh: line 7: databricks: command not found
However, when I run databricks-connect test, it is able to execute the command successfully. Kindly help if I am missing some steps somewhere.
The databricks command is located in the databricks-cli package, not in the databricks-connect, so you need to change your pip install command.
Also, for databricks command you can just set the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN and it will work, like this:
- script: |
pip install pytest requests setuptools wheel
pip install -U databricks-cli
displayName: 'Load Python Dependencies'
- script: |
databricks fs cp ... dbfs:/...
env:
DATABRICKS_HOST: $(DATABRICKS_HOST)
DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
displayName: 'Copy artifacts'
P.S. Here is an example on how to do CI/CD on Databricks + notebooks. You could be also interested in cicd-templates project.
I am trying to run UI tests in Azure devops pipelines and I created the yaml file and added the command to run UItests on chrome, but gets an error "WebDriverError: unknown error: cannot find Chrome binary"
I am using node js code
Here is the YAML file:
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
name: 'Default'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run build
displayName: 'npm install and build'
- task: Npm#1
inputs:
command: 'install'
customCommand: 'iocContainer'
displayName: 'npm iocContainer'
- task: CmdLine#2
inputs:
script: '.\runUITest.bat "#RecieveOnlineOrder"'
WebDriverError: unknown error: cannot find Chrome binary
The problem seems to be that the chrome driver cannot find the chrome.exe file.
Since you are using the self-hosted agent, you need to make sure you have installed Chrome on your local machine firstly. If only the chrome driver is installed, you will face this issue.
You could also try to update the chrome driver to the latest.
If you have install the Chrome, you could check the file path of chrome.exe.
As per the ChromeDriver - Requirements:
On the other hand, this issue may be related to the version of chrome, this is an open ticket with similar issues, you could refer to it.
You could try to run the following script and check if it could work:
npm i chromedriver --chromedriver_version=LATEST --save-dev