How to publish python unittest results to azure pipeline - azure

I am trying to build a pipeline azure DevOps. I build a basic Flask website and wrote a Unittest script for it. It basically all works perfect. When i commit to azure repos the pipeline will do his thing and the test will run. The thing i want is to see the test results, i see all these tutorials for Pytest but not for Unittest.
trigger:
- Development
jobs:
- job: 'Test'
pool:
vmImage: 'ubuntu-latest' # other options: 'macOS-latest', 'windows-latest'
strategy:
matrix:
Python37:
python.version: '3.7'
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
- script: |
python -m pip install --upgrade pip
python -m pip install -e .
displayName: 'Install dependencies'
- script: |
python -m unittest discover -p "*.py" > results.txt
displayName: unittesting
This is my Yaml file that runs the pipeline.
This is how my results look when running the pipeline.
Pipeline results
Is there a way to publish these results with Unittest and have them in azure pipeline.

You'll first need to make your test script generate results in a format that DevOps can understand, i.e. JUnit XML.
There's an example in MS docs (which includes coverage as well):
- script: |
pip install pytest pytest-azurepipelines
pip install pytest-cov
pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml
displayName: 'pytest'
Alternatively, using unittest-xml-reporting should give you results in JUnit XML format as well.
Once you have that, then you can use Publish Test Results task to upload results and make them visible in DevOps UI, i.e.:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-*.xml'

If your unit tests are written using unittest module, dont worry
pytest can run unittest as well !!!
Add below yaml code snippet to your azure pipeline.
Run your unittest with pytest
Generate Junit XML output
Publish the output back to azure pipeline
- script: |
cd $(Build.Repository.LocalPath)
python -m pytest $(Build.Repository.LocalPath)/<unit_tests_path>/*.py --junitxml=test-unit.xml
displayName: 'Run Unit Tests'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'test-unit.xml'

Related

How to pass flags into docker run command in Azure Pipelines?

I am trying to get tfsec running in my Azure Pipeline. I have been able to get this to run with the following commands without any additional flags:
stages:
- stage: QualityCheckStage
displayName: Quality Check Stage
jobs:
- job: QualityTestJob
pool:
vmImage: ubuntu-latest
displayName: Run TFSec
steps:
- bash: |
mkdir TFSecReport
docker pull tfsec/tfsec:latest
docker run --rm -v $(System.DefaultWorkingDirectory):/src tfsec/tfsec ./src --format JUnit > $(System.DefaultWorkingDirectory)/TFSecReport/junit.xml
displayName: TFSec Static Code Analysis
- task: PublishTestResults#2
displayName: Publish TFSecReport Test Results
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/junit.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/TFSecReport'
mergeTestResults: false
failTaskOnFailedTests: false
publishRunAttachments: true
This operates as I expect and successfully completes the job, checking my terraform code and showing results during the pipeline run.
I would like to add additional tfsec flags such as --include-passed, --verbose, or --config-file to make the pipeline more robust. I am struggling to find a way to pass any of these tfsec flags into the docker run command. I've tried adding the flags to every part of the command and Azure spits out the following error ##[error]Bash exited with code '1'..
How can I pass flags into the tfsec image through the docker run command?

Databricks command not found in azure devops pipeline

I am trying to copy a file to Azure Databricks DBFS through Azure Devops pipeline. The following is a snippet from the yml file I am using:
stages:
- stage: MYBuild
displayName: "My Build"
jobs:
- job: BuildwhlAndRunPytest
pool:
vmImage: 'ubuntu-16.04'
steps:
- task: UsePythonVersion#0
displayName: 'Use Python 3.7'
inputs:
versionSpec: '3.7'
addToPath: true
architecture: 'x64'
- script: |
pip install pytest requests setuptools wheel pytest-cov
pip install -U databricks-connect==7.3.*
displayName: 'Load Python Dependencies'
- checkout: self
persistCredentials: true
clean: true
- script: |
echo "y
$(databricks-host)
$(databricks-token)
$(databricks-cluster)
$(databricks-org-id)
8787" | databricks-connect configure
databricks-connect test
env:
databricks-token: $(databricks-token)
displayName: 'Configure DBConnect'
- script: |
databricks fs cp test-proj/pyspark-lib/configs/config.ini dbfs:/configs/test-proj/config.ini
I get the following error at the stage where I am invoking the databricks fs cp command:
/home/vsts/work/_temp/2278f7d5-1d96-4c4e-a501-77c07419773b.sh: line 7: databricks: command not found
However, when I run databricks-connect test, it is able to execute the command successfully. Kindly help if I am missing some steps somewhere.
The databricks command is located in the databricks-cli package, not in the databricks-connect, so you need to change your pip install command.
Also, for databricks command you can just set the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN and it will work, like this:
- script: |
pip install pytest requests setuptools wheel
pip install -U databricks-cli
displayName: 'Load Python Dependencies'
- script: |
databricks fs cp ... dbfs:/...
env:
DATABRICKS_HOST: $(DATABRICKS_HOST)
DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
displayName: 'Copy artifacts'
P.S. Here is an example on how to do CI/CD on Databricks + notebooks. You could be also interested in cicd-templates project.

Publish PHPUnit code coverage result in Pipeline Azure

Azure DevOps Pipelines only supports JaCoCo and Cobertura coverage report formats :
PHPUnit only supports Clover, Crap4jn PHP, (custom) XML, HTML and TXT coverage report formats :
How can I publish the coverage result of my PHPUnit tests in my Pipeline ?
As of this time, however, publishing PHPUnit code coverage result in Pipeline is not supported.
PHPUnit 9.4 added support for Cobertura coverage output. However the default Ubuntu build agents that Azure Pipelines provides atm. only support phpunit 8.5. But you can get coverage reports by running phpunit 9.4+ inside a docker container instead. Here is a snippet of my current azure build pipeline that does that:
trigger:
- master
pool:
vmImage: ubuntu-latest
variables:
phpVersion: 7.4
phpunitImage: jitesoft/phpunit:7.4-9
steps:
- script: |
sudo update-alternatives --set php /usr/bin/php$(phpVersion)
sudo update-alternatives --set phar /usr/bin/phar$(phpVersion)
sudo update-alternatives --set phpdbg /usr/bin/phpdbg$(phpVersion)
sudo update-alternatives --set php-cgi /usr/bin/php-cgi$(phpVersion)
sudo update-alternatives --set phar.phar /usr/bin/phar.phar$(phpVersion)
php -version
displayName: 'Use PHP version $(phpVersion)'
# Do a composer install to get an autoloader that phpunit can use
- script: composer install --no-interaction --prefer-dist
displayName: 'composer install'
# Run the test using the jitesoft phpunit docker image to get support
# for phpunit 9+ and that way cobertura reports for code coverage.
- script: |
docker run --rm -v $(pwd):/app ${{ variables.phpunitImage }} phpunit --log-junit .junit/TEST-phpunit-junit.xml --coverage-cobertura=.coverage/COVERAGE-phpunit-cobertura.xml
displayName: 'Run tests with phpunit docker container'
- task: PublishTestResults#2
displayName: 'Publish test report'
condition: always()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/TEST-phpunit-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/.junit'
failTaskOnFailedTests: true
- task: PublishCodeCoverageResults#1
displayName: 'Publish coverage report'
condition: always()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(System.DefaultWorkingDirectory)/.coverage/COVERAGE-phpunit-*.xml'
pathToSources: '$(System.DefaultWorkingDirectory)/src'
failIfCoverageEmpty: true
note the always() condition in the Publish* tasks. This is needed because if a test fails, then the docker run step will fail with a bash exit code 1, which in turn would prevent the report publishing unless those steps are forced. There might be a way to handle the exit code cleaner, but I haven't figured it out yet.
ps. ideally you'd do some caching as well so the docker image is not always downloaded, but I skipped that part to keep the example focused on the actual running of unit tests and coverage reports.

Running Azure DevOps pipeline exists with ##[error]Bash exited with code '1'

Introduction:
I created a simple add function to get more insight in how Azure Dev Ops is working. When I let test_method2() fail on purpose the pipeline exists with ##[error]Bash exited with code '1'. Therefore in Azure Dev Ops an overview of Tests (which ones are fine and which ones are not) and Code Coverage is missing. I would expect that overview Test and Code Coverage is still provided in case of a failing test. To summarize, it's fine that a test fails but this should be reported and the process should not be stopped.
Note that when I repair the test, i.e. 'def test_method2(): assert add(15,5) == 20 ' the pipeline works as expected.
My question is what is needed to make sure that overview of Tests and Code Coverage is still provided in case of failing tests?
calc_lean.py:
def add(x, y):
"""Add Function"""
return x + y
test_calc_lean.py:
import pytest
from calc_lean import add
def test_method1():
assert add(3,5) == 8
def test_method2():
assert add(15,5) == 21
azure-pipelines.yml:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
Python37:
python.version: '3.7'
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: |
pip install pytest
pip install pytest-cov
pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml
displayName: pytest
- task: PublishTestResults#2
displayName: 'Publish Test Results **/test-results.xml'
inputs:
testResultsFiles: '**/test-results.xml'
testRunTitle: 'Python $(python.version)'
- task: PublishCodeCoverageResults#1
displayName: 'Code Coverage'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'
Solution: Problem was already discussed in other posts. Amongst others Failed Cypress test exit in 'bash exited with code 1' in Azure DevOps
Adding ' || true' adressed it. That is,
pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml || true
supresses the err

Azure DevOps PublishTestResults task -- how to publish results if tests fail

I am running a pytest-based suite of tests during my Azure DevOps build process. I have two jobs arranged to run these tests against two different environments.
In each job, I run the pytest tests using a script task and generate a junit-style xml output file, then have a PublishTestResults task publish that xml file. This is working great, and I'm able to peruse my test results in the azure build tests report UI -- but only if all the tests pass. If any tests fail, the publish task is skipped, and the tests aren't reported in the UI.
YML extract:
- job: 'RunTestsQA'
continueOnError: True
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.6'
architecture: 'x64'
- task: DownloadSecureFile#1
inputs:
secureFile: 'ConfigFile'
- script: pip install -r requirements.txt
displayName: 'Install Requirements'
- script: |
pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
displayName: 'Test with pytest'
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
condition: succeededOrFailed()
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
#searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
mergeTestResults: false # Optional
testRunTitle: 'API_CHECK QA'
#buildPlatform: # Optional
#buildConfiguration: # Optional
publishRunAttachments: true # Optional
Through some experimentation, I've been able to confirm the XML file is always created. What do I need to fix here? A test report isn't super helpful if it only shows up when the tests pass.
In your task description, the condition is effectively listed as a task input, and hence won't be taken into account at all.
You had:
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
condition: succeededOrFailed()
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
The correct setup is
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
condition: succeededOrFailed()
The full list of things you can do with conditions is here
I'm using Ruby and Minitest, but I have found that the following setting allows the PublishTestResults task to run:
- script: |
pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
displayName: 'Test with pytest'
continueOnError: true
The only issue that I have found with this setting is that if the build fails, it reports as "Partially Succeeded" and not "Failed".
edit:
Of course, if your build process has any deploy tasks after the test task, you may not want to use this setting.

Resources