I have some unit tests written with jest. Obviously, sometimes they fail. I want to publish the result to the build Test section.
The problem is task PublishTestResults#2 never runs if a previous task fails, hence no data in test section of the build.
In this case, what I get is the message ##[error]Bash exited with code '1'..
Here is my azure-pipelines.yml:
- script: npm run test:publish
displayName: "Run Unit Tests"
- script: npm run build
displayName: "Build"
- task: PublishTestResults#2
displayName: "Publish unit test result"
inputs:
testResultsFiles: "unit_test_report.xml"
testRunTitle: "Unit tests with Jest"
mergeTestResults: false
searchFolder: "$(Build.SourcesDirectory)"
condition: always()
failTaskOnFailedTests: false
What am I missing?
You have indented condition: always() one place too far in your YAML, it should sit at the same level as displayName
Related
I have a react app, created with npx create-react-app command, and configured a yaml based azure devops pipeline for CI/CD. Then I added some simple unit tests.
I added a script task in my yaml pipeline to generate coverage results as follows.
- script: |
cd reactonazure
npm run test -- --coverage --ci --reporters=default --reporters=jest-junit --coverageReporters=cobertura
displayName: 'test code coverage'
But this tasks displays the following output and then task continues to run and never ends. I have to cancel the pipeline run to ultimately stop it. Not sure whats happening. Any ideas?
No tests found related to files changed since last commit.
But when I prefix with CI=true to make the command as follows, then the command works fine and I get test results, but no where I find coverage report(cobertura-coverage.xml).
CI=true npm test -- --reporters=jest-junit --reporters=default --coverageReporters=cobertura
I added a simple script task shown below to list out the files and folders to enable me search for cobertura-coverage.xml. But no where I found the file. Note that jest.xml file is being created. And that is the reason why I am able to see the test results shown above. The problem is with test coverage report.
- script: |
pwd
cd ..
pwd
ls -Rla
displayName: 'list out the files and folders for inspection'
Also when I run the following command on vs code command prompt on my machine(Windows 10), I do see a file named cobertura-coverage.xml in coverage folder.
npm run test -- --coverage --ci --reporters=default --reporters=jest-junit --coverageReporters=cobertura
The full pipeline yaml is pasted below.
trigger:
- master
variables:
azureSubscription: 'AzureServiceConnection'
# Web app name
webAppName: BasicReactApp
# Environment name
environmentName: Dev
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: NodeTool#0
inputs:
versionSpec: '16.x'
displayName: 'Install Node.js'
- script: |
cd reactonazure
npm install
npm run build --if-present
displayName: 'npm install, build '
- script: |
cd reactonazure
CI=true npm test -- --reporters=jest-junit --reporters=default --coverageReporters=cobertura
displayName: 'test code coverage'
- task: PublishTestResults#2
displayName: "Publish Test Results"
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'reactonazure/junit.xml'
failTaskOnFailedTests: true
mergeTestResults: true
condition: succeededOrFailed()
# The following, two are not functioning as no coverage.cobertura.xml file is being created.
- task: PublishCodeCoverageResults#1
displayName: "Publish code coverage"
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: "reactonazure/$(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml"
reportDirectory: "reactonazure/$(System.DefaultWorkingDirectory)/coverage"
failIfCoverageEmpty: false
- task: PublishCodeCoverageResults#1
displayName: 'Publish code coverage report'
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
failIfCoverageEmpty: true
Are there any other ways to generate report on the pipeline?
The path you provided to the coverage file is incorrect. It should be $(System.DefaultWorkingDirectory)/reactonazure/coverage/cobertura-coverage.xml.
The task would look like this:
- task: PublishCodeCoverageResults#1
displayName: "Publish code coverage"
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: "$(System.DefaultWorkingDirectory)/reactonazure/coverage/cobertura-coverage.xml"
reportDirectory: "$(System.DefaultWorkingDirectory)/reactonazure/coverage"
failIfCoverageEmpty: false
The command to generate the report that I presented earlier is not working. The following is finally working.
CI=true npm run test -- --coverage --watchAll=false --ci --reporters=default --reporters=jest-junit --coverageReporters=cobertura
The full and final yaml after taking into account the suggested answer is below.
trigger:
- master
variables:
azureSubscription: 'AzureServiceConnection'
webAppName: BasicReactApp
environmentName: Dev
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: NodeTool#0
inputs:
versionSpec: '16.x'
displayName: 'Install Node.js'
- script: |
cd reactonazure
npm install
npm run build --if-present
displayName: 'npm install, build '
- script: |
cd reactonazure
CI=true npm run test -- --coverage --watchAll=false --ci --reporters=default --reporters=jest-junit --coverageReporters=cobertura
displayName: 'Test with code coverage'
- task: PublishTestResults#2
displayName: "Publish Test Results"
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'reactonazure/junit.xml'
failTaskOnFailedTests: true
mergeTestResults: true
condition: succeededOrFailed()
- task: PublishCodeCoverageResults#1
displayName: "Publish code coverage"
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: "$(System.DefaultWorkingDirectory)/reactonazure/coverage/cobertura-coverage.xml"
reportDirectory: "$(System.DefaultWorkingDirectory)/reactonazure/coverage"
failIfCoverageEmpty: false
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(System.DefaultWorkingDirectory)/reactonazure/build'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- upload: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: $(environmentName)
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy'
inputs:
azureSubscription: $(azureSubscription)
#appType: webAppLinux
appType: webApp
appName: $(webAppName)
runtimeStack: 'NODE|16.x'
package: $(Pipeline.Workspace)/drop/$(Build.BuildId).zip
startUpCommand: 'npm run start'
How do we include unit test in .Net azure build pipeline. Unit testing should happen before before the build process and when the unit test is Succeeded only then the build should happen.
You need to build your app first to be able to run unit tests. Or you can just run dotnet test command which makes sure that first your nuget packages are restored, app is build and then unit test run.
Please check this documentation
Here is an example pipeline:
- task: UseDotNet#2
inputs:
version: '5.0.x'
includePreviewVersions: true # Required for preview versions
- task: DotNetCoreCLI#2
displayName: 'dotnet build'
inputs:
command: 'build'
configuration: $(buildConfiguration)
- task: DotNetCoreCLI#2
displayName: 'dotnet test'
inputs:
command: 'test'
arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
publishTestResults: true
projects: 'MyTestLibrary' # update with your test project directory
I have a build pipeline as shown below
variables:
- name: BuildParameters.solution
value: '**/*.sln'
- name: buildConfiguration
value: Release
name: $(date:yyyyMMdd)$(rev:.r)
trigger: none
resources:
repositories:
- repository: self
type: git
stages:
- stage: __default
jobs:
- job: Job_1
displayName: Agent job 1
pool:
vmImage: windows-2019
steps:
- task: UseDotNet#2
displayName: '.NET Core 3.1.404'
inputs:
version: '3.1.404'
packageType: sdk
- task: DotNetCoreCLI#2
enabled: true
displayName: dotnet test
continueOnError: true
timeoutInMinutes: 0
inputs:
command: test
projects: '**/*test/*.csproj'
arguments: --configuration $(buildConfiguration) --verbosity n --collect "Code coverage"
publishTestResults: true
- task: PublishTestResults#2
inputs:
testResultsFormat: XUnit
testResultsFiles: '**/TEST.xml'
mergeTestResults: true
failTaskOnFailedTests: true
I have broken a test on purpose
This test fails, but the publish test results task doesnt fail outright
I get the warning
##[warning]No test result files matching **/TEST.xml were found. xunit
Has anyone ever had this?
Annoyingly if a test fails the Azure pipeline classes this as a partial success which is not ideal. I am trying to get into a situation where if a test fails, my pipeline fails
This pipeline is used to verify pull requests - i.e. do a build of the code in the PR and ensure that all tests pass
When I searched for this there were references to installing .NET 4.6.2 or later, but this is being used inside a build pipeline, so not sure how to do that
Have I missed something in the dot net test task?
Paul
When you set publishTestResults field to true for dotnet test task. See below extract from here.
This option appends --logger trx --results-directory $(Agent.TempDirectory) to the command line arguments.
Above commands will generate the test result in VSTest(trx) format and output the result to folder $(Agent.TempDirectory)
So you need to change the PublishTestResults#2 task like below:
- task: PublishTestResults#2
inputs:
testResultsFormat: VSTest #change the format to VSTest
testResultsFiles: '**/*.trx' #change to find .trx file
mergeTestResults: true
failTaskOnFailedTests: true
searchFolder: '$(Agent.TempDirectory)' #search the .trx file in $(Agent.TempDirectory)
Usually the continueOnError field is set to false by default. And the dotnet test task will fail if there is any test fails, See below:
If the continueOnError field is set to true. The dotnet test task and the pipeline will show Warnig state.
So you can have a try setting the continueOnError field to false, which will fail the task and pipeline if there is any test fails. And there will not be no need to use PublishTestResults task. For publishTestResults is set to true for dotnet test task, the test result will be automatically published by the dotnet test task. See below:
- task: DotNetCoreCLI#2
enabled: true
displayName: dotnet test
continueOnError: false
timeoutInMinutes: 0
inputs:
command: test
projects: '**/*test/*.csproj'
arguments: --configuration $(buildConfiguration) --verbosity n --collect "Code coverage"
publishTestResults: true
I have created the below YAML file with the PublishTestResults task to display the mocha test results in Azure DevOps portal
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.14.1'
displayName: 'Install Node.js'
- script: |
npm install
mocha test --reporter mocha-junit-reporter
npm test
displayName: 'npm install and test'
- task: PublishTestResults#2
condition: succeededOrFailed()
inputs:
testRunner: JUnit
testResultsFiles: '**/TEST-RESULTS.xml'
- task: ArchiveFiles#2
displayName: 'Archive $(System.DefaultWorkingDirectory)'
inputs:
rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
includeRootFolder: false
archiveFile: '$(Build.ArtifactStagingDirectory)/node.zip'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
but whenever I run the build I am getting the following warning
"No test result files matching **/TEST-RESULTS.xml were found."
The main motive is to display the mocha test results separately in a dashboard or a test tab. So that we don't have to check the test task in build process to see the test results.
I encountered same issue,
After trying bunch of things could solve it as below.
Step 1. Update package.json
Include in scripts section:
"test": "cross-env CI=true react-scripts test --env=jsdom
--testPathIgnorePatterns=assets --reporters=default --reporters=jest-junit"
Include below in jest config or jest section in package.json
"jest-junit": {
"suiteNameTemplate": "{filepath}",
"outputDirectory": ".",
"outputName": "junit.xml"
}
Run npm run test locally and see if junit.xml is created.
Step 2. In Azure Pipeline YAML file
Include an Npm task with custom command as below
- task: Npm#1
displayName: 'Run Tests'
inputs:
command: 'custom'
workingDir: "folder where your package.json exists"
customCommand: 'test'
Task to publish Results
- task: PublishTestResults#2
displayName: 'Publish Unit Test Results'
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/junit.xml'
mergeTestResults: true
failTaskOnFailedTests: true
testRunTitle: 'React App Test'
Publish Test Results
I am running a pytest-based suite of tests during my Azure DevOps build process. I have two jobs arranged to run these tests against two different environments.
In each job, I run the pytest tests using a script task and generate a junit-style xml output file, then have a PublishTestResults task publish that xml file. This is working great, and I'm able to peruse my test results in the azure build tests report UI -- but only if all the tests pass. If any tests fail, the publish task is skipped, and the tests aren't reported in the UI.
YML extract:
- job: 'RunTestsQA'
continueOnError: True
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.6'
architecture: 'x64'
- task: DownloadSecureFile#1
inputs:
secureFile: 'ConfigFile'
- script: pip install -r requirements.txt
displayName: 'Install Requirements'
- script: |
pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
displayName: 'Test with pytest'
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
condition: succeededOrFailed()
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
#searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
mergeTestResults: false # Optional
testRunTitle: 'API_CHECK QA'
#buildPlatform: # Optional
#buildConfiguration: # Optional
publishRunAttachments: true # Optional
Through some experimentation, I've been able to confirm the XML file is always created. What do I need to fix here? A test report isn't super helpful if it only shows up when the tests pass.
In your task description, the condition is effectively listed as a task input, and hence won't be taken into account at all.
You had:
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
condition: succeededOrFailed()
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
The correct setup is
# PUBLISH JUNIT RESULTS
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/TEST-*.xml'
condition: succeededOrFailed()
The full list of things you can do with conditions is here
I'm using Ruby and Minitest, but I have found that the following setting allows the PublishTestResults task to run:
- script: |
pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
displayName: 'Test with pytest'
continueOnError: true
The only issue that I have found with this setting is that if the build fails, it reports as "Partially Succeeded" and not "Failed".
edit:
Of course, if your build process has any deploy tasks after the test task, you may not want to use this setting.