Publish PHPUnit code coverage result in Pipeline Azure - azure

Azure DevOps Pipelines only supports JaCoCo and Cobertura coverage report formats :
PHPUnit only supports Clover, Crap4jn PHP, (custom) XML, HTML and TXT coverage report formats :
How can I publish the coverage result of my PHPUnit tests in my Pipeline ?

As of this time, however, publishing PHPUnit code coverage result in Pipeline is not supported.

PHPUnit 9.4 added support for Cobertura coverage output. However the default Ubuntu build agents that Azure Pipelines provides atm. only support phpunit 8.5. But you can get coverage reports by running phpunit 9.4+ inside a docker container instead. Here is a snippet of my current azure build pipeline that does that:
trigger:
- master
pool:
vmImage: ubuntu-latest
variables:
phpVersion: 7.4
phpunitImage: jitesoft/phpunit:7.4-9
steps:
- script: |
sudo update-alternatives --set php /usr/bin/php$(phpVersion)
sudo update-alternatives --set phar /usr/bin/phar$(phpVersion)
sudo update-alternatives --set phpdbg /usr/bin/phpdbg$(phpVersion)
sudo update-alternatives --set php-cgi /usr/bin/php-cgi$(phpVersion)
sudo update-alternatives --set phar.phar /usr/bin/phar.phar$(phpVersion)
php -version
displayName: 'Use PHP version $(phpVersion)'
# Do a composer install to get an autoloader that phpunit can use
- script: composer install --no-interaction --prefer-dist
displayName: 'composer install'
# Run the test using the jitesoft phpunit docker image to get support
# for phpunit 9+ and that way cobertura reports for code coverage.
- script: |
docker run --rm -v $(pwd):/app ${{ variables.phpunitImage }} phpunit --log-junit .junit/TEST-phpunit-junit.xml --coverage-cobertura=.coverage/COVERAGE-phpunit-cobertura.xml
displayName: 'Run tests with phpunit docker container'
- task: PublishTestResults#2
displayName: 'Publish test report'
condition: always()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/TEST-phpunit-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/.junit'
failTaskOnFailedTests: true
- task: PublishCodeCoverageResults#1
displayName: 'Publish coverage report'
condition: always()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(System.DefaultWorkingDirectory)/.coverage/COVERAGE-phpunit-*.xml'
pathToSources: '$(System.DefaultWorkingDirectory)/src'
failIfCoverageEmpty: true
note the always() condition in the Publish* tasks. This is needed because if a test fails, then the docker run step will fail with a bash exit code 1, which in turn would prevent the report publishing unless those steps are forced. There might be a way to handle the exit code cleaner, but I haven't figured it out yet.
ps. ideally you'd do some caching as well so the docker image is not always downloaded, but I skipped that part to keep the example focused on the actual running of unit tests and coverage reports.

Related

How to publish python unittest results to azure pipeline

I am trying to build a pipeline azure DevOps. I build a basic Flask website and wrote a Unittest script for it. It basically all works perfect. When i commit to azure repos the pipeline will do his thing and the test will run. The thing i want is to see the test results, i see all these tutorials for Pytest but not for Unittest.
trigger:
- Development
jobs:
- job: 'Test'
pool:
vmImage: 'ubuntu-latest' # other options: 'macOS-latest', 'windows-latest'
strategy:
matrix:
Python37:
python.version: '3.7'
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
- script: |
python -m pip install --upgrade pip
python -m pip install -e .
displayName: 'Install dependencies'
- script: |
python -m unittest discover -p "*.py" > results.txt
displayName: unittesting
This is my Yaml file that runs the pipeline.
This is how my results look when running the pipeline.
Pipeline results
Is there a way to publish these results with Unittest and have them in azure pipeline.
You'll first need to make your test script generate results in a format that DevOps can understand, i.e. JUnit XML.
There's an example in MS docs (which includes coverage as well):
- script: |
pip install pytest pytest-azurepipelines
pip install pytest-cov
pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml
displayName: 'pytest'
Alternatively, using unittest-xml-reporting should give you results in JUnit XML format as well.
Once you have that, then you can use Publish Test Results task to upload results and make them visible in DevOps UI, i.e.:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-*.xml'
If your unit tests are written using unittest module, dont worry
pytest can run unittest as well !!!
Add below yaml code snippet to your azure pipeline.
Run your unittest with pytest
Generate Junit XML output
Publish the output back to azure pipeline
- script: |
cd $(Build.Repository.LocalPath)
python -m pytest $(Build.Repository.LocalPath)/<unit_tests_path>/*.py --junitxml=test-unit.xml
displayName: 'Run Unit Tests'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'test-unit.xml'

How to pass flags into docker run command in Azure Pipelines?

I am trying to get tfsec running in my Azure Pipeline. I have been able to get this to run with the following commands without any additional flags:
stages:
- stage: QualityCheckStage
displayName: Quality Check Stage
jobs:
- job: QualityTestJob
pool:
vmImage: ubuntu-latest
displayName: Run TFSec
steps:
- bash: |
mkdir TFSecReport
docker pull tfsec/tfsec:latest
docker run --rm -v $(System.DefaultWorkingDirectory):/src tfsec/tfsec ./src --format JUnit > $(System.DefaultWorkingDirectory)/TFSecReport/junit.xml
displayName: TFSec Static Code Analysis
- task: PublishTestResults#2
displayName: Publish TFSecReport Test Results
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/junit.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/TFSecReport'
mergeTestResults: false
failTaskOnFailedTests: false
publishRunAttachments: true
This operates as I expect and successfully completes the job, checking my terraform code and showing results during the pipeline run.
I would like to add additional tfsec flags such as --include-passed, --verbose, or --config-file to make the pipeline more robust. I am struggling to find a way to pass any of these tfsec flags into the docker run command. I've tried adding the flags to every part of the command and Azure spits out the following error ##[error]Bash exited with code '1'..
How can I pass flags into the tfsec image through the docker run command?

Why is test automation only partially successful with Cypress and Azure DevOps?

I am using Cypress.io (Version 5.1.0) for testing my project.
My project is in azure DevOps. Now i want to include my cypress tests in Azure DevOps so my tests will run automatically.
I set up the JUnit reporter on my Cypress project:
into my “package.json” file i added
"cypress-multi-reporters": "^1.2.4",
"mocha-junit-reporter": "^1.23.3"
then run
npm install
than added
"scripts": {
"scripts": "cypress run",
"test": "npm run scripts"
}
Into my “cypress.json” file i added
"reporter": "mocha-junit-reporter",
"reporterOptions": {
"mochaFile": "cypress/reports/junit/test-results.[hash].xml",
"testsuitesTitle": false
}
After this I created a new Pipeline using Azure Repos in Azure DevOps.
For Pipeline Configuration i selected Node.js.
Now I have a YAML file. Here i removed npm build from the first script.
Now I picked npm from the assisstant. On the npm configurations, I selected custom and write the command run test . Now I Select the result format as “JUnit” and set Test results files to “*.xml”
At last I selected the option "Merge test results".
Now I saved and run the pipeline.
This is what my Job does:
Pool: Azure Pipelines
Image: ubuntu-latest
Agent: Hosted Agent
Started: Yesterday at 17:31
Expanded: Object
Result: True
Evaluating: not(containsValue(job['steps']['*']['task']['id'], '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Expanded: not(containsValue(Object, '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Result: True
Evaluating: resources['repositories']['self']['checkoutOptions']
Result: Object
Finished evaluating template 'system-pre-steps.yml'
********************************************************************************
Template and static variable resolution complete. Final runtime YAML document:
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4#1
inputs:
repository: self
MaxConcurrency: 0
What is wrong with my automation? How can I fix this?
Update:
Thats my yml file:
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- task: Npm#1
inputs:
command: 'custom'
customCommand: 'run test'
continueOnError: true
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/cypress/reports/junit'
mergeTestResults: true
testRunTitle: 'Publish Test Results'
I got an email with this Details
Job 1 error(s), 1 warning(s) Error: Npm failed with return code: 254
The issue may be due to the agent rather than your code and scripts.
You can try the following solutions:
Change your agent image. As you are currently using the ubuntu-latest, it is recommanded to try the ubuntu-20.04 or ubuntu-16.04.
Use a self-hosted agent. If you don't have a self-hosted agent, click Self-hosted Linux agent for detailed steps.
Change the orgnization. Choose another organization that can run the build correctly, and just in case, it is better to create a new organization. Then create a new project and try your tests.
As stated already, the problem most likely lies with the Azure environment. Cypress has a dependency on a browser (electron, chrome) in order to execute. For example, if you are using docker, they provide an official image called cypress/browsers:node14.7.0-chrome84 that has everything you need out of the box. The Dockerfile also has useful info on the environment needed. Make sure to provide a headless configuration as well, something like:
cypress run --headless --browser chrome --spec yourSpecHere.js

Azure DevOps Build pipline

I am new to AzureDevOps, i am trying to use Azure DevOps for one of my C++ project. My requirements are below.
1: Build C++ and publish to Artifactory(Azure),This i am able to do and binary able to publish in Artifactory.
2: I want to use that Artifactory(Binary file) while building Docker image with help of binary file. But i am unable to achieve in Azure DevOps. Locally i am able to build docker image with binary and running file.
Just As summary i am writing.
I need to create simple release pipeline using the build artifacts from the previous task
● The Release Pipeline should build a Docker image with the following requirements
○ must contain Build artifacts from the build pipeline.
Please find below code:
trigger:
- master
jobs:
- job: gcctest
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: sudo apt-get update && sudo apt-get install libboost-all-dev
- script: g++ -std=c++11 -I/usr/include/boost/asio -I/usr/include/boost -o binary.out main.cpp
connection.cpp connection_manager.cpp mime_types.cpp reply.cpp request_handler.cpp request_parser.cpp server.cpp -lboost_system -lboost_thread -lpthread
- powershell:
$commitId= "$env:BUILD_SOURCEVERSION"
$definitionName= "$env:BUILD_DEFINITIONNAME"
$shortId= $commitId.Substring(1, 8)
$buildName="$definitionName.$shortId"
Write-Host $buildName
Write-Output "$buildName">readme.txt
# echo "##vso[task.setvariable variable=text]$buildName"
- task: CopyFiles#2
inputs:
sourceFolder: '$(Build.SourcesDirectory)'
contents: '?(*.out|*.txt)'
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: $(Build.ArtifactStagingDirectory)
artifactName: result
- task: Docker#1
displayName: 'Build using multi-stage'
inputs:
containerregistrytype: 'Container Registry'
dockerRegistryEndpoint: 'My Container Registry'
arguments: '--build-arg ARTIFACTS_ENDPOINT=$(ArtifactFeed)'
Docker File:
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y libboost-all-dev
COPY . /app
EXPOSE 80
CMD /app/binary.out 0.0.0.0 80 .```
~
Expected result:
Docker image should build by artifact and able to publish on DockerHub repo.

Azure DevOps PublishTestResults task -- how to publish results if tests fail

I am running a pytest-based suite of tests during my Azure DevOps build process. I have two jobs arranged to run these tests against two different environments.
In each job, I run the pytest tests using a script task and generate a junit-style xml output file, then have a PublishTestResults task publish that xml file. This is working great, and I'm able to peruse my test results in the azure build tests report UI -- but only if all the tests pass. If any tests fail, the publish task is skipped, and the tests aren't reported in the UI.
YML extract:
- job: 'RunTestsQA'
continueOnError: True
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.6'
architecture: 'x64'
- task: DownloadSecureFile#1
inputs:
secureFile: 'ConfigFile'
- script: pip install -r requirements.txt
displayName: 'Install Requirements'
- script: |
pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
displayName: 'Test with pytest'
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
condition: succeededOrFailed()
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
#searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
mergeTestResults: false # Optional
testRunTitle: 'API_CHECK QA'
#buildPlatform: # Optional
#buildConfiguration: # Optional
publishRunAttachments: true # Optional
Through some experimentation, I've been able to confirm the XML file is always created. What do I need to fix here? A test report isn't super helpful if it only shows up when the tests pass.
In your task description, the condition is effectively listed as a task input, and hence won't be taken into account at all.
You had:
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
condition: succeededOrFailed()
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
The correct setup is
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
condition: succeededOrFailed()
The full list of things you can do with conditions is here
I'm using Ruby and Minitest, but I have found that the following setting allows the PublishTestResults task to run:
- script: |
pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
displayName: 'Test with pytest'
continueOnError: true
The only issue that I have found with this setting is that if the build fails, it reports as "Partially Succeeded" and not "Failed".
edit:
Of course, if your build process has any deploy tasks after the test task, you may not want to use this setting.

Resources