I've found some questions on SO specific to version for jest unit test to publish its result in VSTS build Test Results tab. But no proper solution is found.
I've used a different approach, b/c after some research I found that the Jest testResultsProcessor property is deprecated. I'm using the jest-junit package for test reports (which has been worked on more recently than the jest-trx-results-processor, fwiw):
Add jest-junit to package.json
Eg yarn add -D jest-junit or npm add --save-dev jest-junit
Add a VSTS task to run Jest using the jest-junit results reporter
I used the Yarn task, but you can alternately use the npm task. I used these task arguments:
jest --ci --reporters=jest-junit --reporters=default --coverage --coverageReporters=cobertura --coverageReporters=html
because I also wanted code coverage. To skip code coverage reporting, use these (npm or yarn) task arguments:
jest --ci --reporters=jest-junit --reporters=default
Note that --reporters=default is there b/c I wanted the default stdout in my build log.
Add a Publish Test Results task
Since we're using the default path, the test results file will be written to ~/junit.xml
(Optional) Add a publish code coverage task, too
If you're running code coverage, you might as well add a task for publishing the code coverage results, too:
If you're using a YAML pipeline, here's equivalent YAML (note that we're using the yarn task instead of npm tasks, but that can be changed):
- task: geeklearningio.gl-vsts-tasks-yarn.yarn-task.Yarn#2
displayName: 'Install dependencies'
inputs:
Arguments: install --no-progress
- task: geeklearningio.gl-vsts-tasks-yarn.yarn-task.Yarn#2
displayName: 'Unit tests'
inputs:
Arguments: 'test --ci --reporters=jest-junit --reporters=default --coverage --coverageReporters=cobertura'
continueOnError: true # Test failures should be published before failing the build
- task: PublishTestResults#2
displayName: 'Publish Jest Unit Test Results'
inputs:
testResultsFiles: junit.xml
mergeTestResults: true
testRunTitle: 'Jest Unit Tests'
failTaskOnFailedTests: true
- task: PublishCodeCoverageResults#1
displayName: 'Publish code coverage from Jest tests'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml'
# reportDirectory: '$(System.DefaultWorkingDirectory)/coverage'
failIfCoverageEmpty: true
I've gone throw some jest npm packages like tap-xunit and jest-json-to-tap but couldn't figure out it to work. Following steps worked for me to get the results to review under Test of VSTS build.
Install jest-trx-results-processor
# NPM
npm install jest-trx-results-processor --save-dev
# Yarn
yarn add -D jest-trx-results-processor
Create jestTrxProcessor.js file with following content:
var builder = require('jest-trx-results-processor');
var processor = builder({
outputFile: 'jestTestresults.trx'
});
module.exports = processor;
Updated package.json file should look like:
"devDependencies": {
"jest": "^23.4.1",
"jest-trx-results-processor": "0.0.7",
"jsdom": "^11.12.0",
...
},
"scripts": {
"test": "jest"
},
"jest": {
...,
"testResultsProcessor": "./jestTrxProcessor.js"
}
Add npm task to VSTS build for npm test. This will run jest tests and publish results to jestTestresults.trx
Add Publish Test Results task of VSTS to add jestTestresults.trx results in VSTS test.
You will be able to see JEST tests along with other tests.
Evaldas' solution is obsolete, so I'm going to add a few modifications.
The more modern solution is a combination between Evaldas' here, as well as the one from the maintainer: https://www.npmjs.com/package/jest-trx-results-processor
I'll describe it as such below.
Install jest-trx-results-processor
# NPM
npm install jest-trx-results-processor --save-dev
# Yarn
yarn add -D jest-trx-results-processor
Updated package.json file should look like:
"devDependencies": {
"jest": "^24.9.0",
"jest-trx-results-processor": "^1.0.2"
...
},
"scripts": {
"test": "jest"
},
"jest": {
...,
"reporters": [
"default",
[
"jest-trx-results-processor",
{
"outputFile": "./src/jestTestresults.trx",
"defaultUserName": "user name to use if automatic detection fails"
}
]]}
Add npm task to VSTS build for npm test in the build pipeline. It should look like this:
Add Publish Test Results task of VSTS to add jestTestresults.trx results in VSTS test. To do this, in the build pipeline, click on the 'add sign'. Look for "Publish Test Results". It'll bring up a menu like this. Since it's a .trx file, you'll need to use VSTest instead of JTest.
Finally, the build pipeline for your frontend project will look like this:
Related
I have set up a cron job for my cypress tests however I want to run it on a specific test alone.
This is my cron job:
name: Cypress Tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
# Install NPM dependencies, cache them correctly
# and run all Cypress tests
- name: Cypress run
uses: cypress-io/github-action#v4.x.x # use the explicit version number
with:
build: npm run build
start: npm start
I just want to run this one spec file:
Is there a way to run this spec file alone?
Try this out!
Add new command in script section of package.json. Use file or folder here.
"scripts": {
"cypress:spec-run": "cypress run --browser chrome --spec test.spec.js"
},
Run the commond in respective crone job
npm run cypress:spec-run
I am trying to setup task PublishTestResults#2 so it shows xunit style mocha test results in azure devlops pipeline test tab.
Unfortunately I am getting an error:
Obtained XUnit Test Run Start Date: 2020-09-18T12:48:36.0000000Z and Completed Date: 2020-09-18T12:48:36.0470000Z
No Result Found to Publish '/home/vsts/work/1/s/portal/test-result.xml'.
Async Command Start: Publish test results
Async Command End: Publish test results
Finishing: Publish Test Results
Is there something wrong with the configuration? Perhaps I am missing some additional things.
I was using mocha-junit-reporter before but some reason it couldn't display all of the tests cases inside xml file. It only displayed 112 test cases out of 1800+.
Here's the configuration
package.json
{
"scripts": {
"test:pipeline": "mocha --reporter mocha-xunit-reporter --reporter-options \"addTags=true,assemblyName=Tests,mochaFile=./test-result.xml\" --require test-setup.js --recursive ./ClientApp/**/*.spec.ts*",
},
"dependencies": {
"mocha": "6.2.0",
"mocha-xunit-reporter": "2.2.0",
}
}
yaml file
- job: build_web
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Npm#1
displayName: 'Restore NPM packages'
inputs:
command: 'install'
workingDir: './portal'
- task: Npm#1
displayName: 'Run Unit Tests'
inputs:
command: 'custom'
workingDir: './portal'
customCommand: 'run test:pipeline'
- task: PublishTestResults#2
displayName: 'Publish Test Results'
inputs:
testRunTitle: 'Mocha Tests'
testResultsFormat: 'xUnit'
testResultsFiles: '**/test-*.xml'
Thanks for help
--Edit
Run Unit Tests step was creating the file, if it couldn't find it I would receive an error:
##[warning]No test result files matching **/test-*.xml were found.
That happened when I was fiddling with a yaml.
Logs of the step.
Starting: Run Unit Tests
==============================================================================
Task : npm
Description : Install and publish npm packages, or run an npm command. Supports npmjs.com and authenticated registries like Azure Artifacts.
Version : 1.175.0
Author : Microsoft Corporation
Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/package/npm
==============================================================================
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
/usr/local/bin/npm --version
6.14.8
/usr/local/bin/npm config list
; cli configs
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.14.8 node/v12.18.3 linux x64"
; environment configs
userconfig = "/home/vsts/work/1/npm/722.npmrc"
; node bin location = /usr/local/bin/node
; cwd = /home/vsts/work/1/s/portal
; HOME = /home/vsts
; "npm config ls -l" to show all defaults.
/usr/local/bin/npm run test:pipeline
> portal#1.0.0 test:pipeline /home/vsts/work/1/s/portal
> mocha --reporter mocha-xunit-reporter --reporter-options "addTags=true,assemblyName=Portal Tests,mochaFile=test-result.xml" --require test-setup.js --recursive ./ClientApp/**/*.spec.ts*
Finishing: Run Unit Tests
I switched vmImage to windows, mocha reporter to mocha-junit-reporter and now it runs fine.
I didn't bother checking other reporters after changing the image.
Here's the diff.
- job: build_web
pool:
vmImage: 'windows-2019'
...
- task: PublishTestResults#2
displayName: 'Publish Test Results'
inputs:
testRunTitle: 'Mocha Tests'
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-*.xml'
"scripts": {
"test:pipeline": "mocha --reporter mocha-junit-reporter --reporter-options mochaFile=./test-result.xml --timeout 60000 --require test-setup.js --recursive ./ClientApp/**/*.spec.ts*",
},
"dependencies": {
"mocha": "6.2.0",
"mocha-junit-reporter": "1.23.1",
}
}
I am using Cypress.io (Version 5.1.0) for testing my project.
My project is in azure DevOps. Now i want to include my cypress tests in Azure DevOps so my tests will run automatically.
I set up the JUnit reporter on my Cypress project:
into my “package.json” file i added
"cypress-multi-reporters": "^1.2.4",
"mocha-junit-reporter": "^1.23.3"
then run
npm install
than added
"scripts": {
"scripts": "cypress run",
"test": "npm run scripts"
}
Into my “cypress.json” file i added
"reporter": "mocha-junit-reporter",
"reporterOptions": {
"mochaFile": "cypress/reports/junit/test-results.[hash].xml",
"testsuitesTitle": false
}
After this I created a new Pipeline using Azure Repos in Azure DevOps.
For Pipeline Configuration i selected Node.js.
Now I have a YAML file. Here i removed npm build from the first script.
Now I picked npm from the assisstant. On the npm configurations, I selected custom and write the command run test . Now I Select the result format as “JUnit” and set Test results files to “*.xml”
At last I selected the option "Merge test results".
Now I saved and run the pipeline.
This is what my Job does:
Pool: Azure Pipelines
Image: ubuntu-latest
Agent: Hosted Agent
Started: Yesterday at 17:31
Expanded: Object
Result: True
Evaluating: not(containsValue(job['steps']['*']['task']['id'], '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Expanded: not(containsValue(Object, '6d15af64-176c-496d-b583-fd2ae21d4df4'))
Result: True
Evaluating: resources['repositories']['self']['checkoutOptions']
Result: Object
Finished evaluating template 'system-pre-steps.yml'
********************************************************************************
Template and static variable resolution complete. Final runtime YAML document:
steps:
- task: 6d15af64-176c-496d-b583-fd2ae21d4df4#1
inputs:
repository: self
MaxConcurrency: 0
What is wrong with my automation? How can I fix this?
Update:
Thats my yml file:
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- task: Npm#1
inputs:
command: 'custom'
customCommand: 'run test'
continueOnError: true
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/cypress/reports/junit'
mergeTestResults: true
testRunTitle: 'Publish Test Results'
I got an email with this Details
Job 1 error(s), 1 warning(s) Error: Npm failed with return code: 254
The issue may be due to the agent rather than your code and scripts.
You can try the following solutions:
Change your agent image. As you are currently using the ubuntu-latest, it is recommanded to try the ubuntu-20.04 or ubuntu-16.04.
Use a self-hosted agent. If you don't have a self-hosted agent, click Self-hosted Linux agent for detailed steps.
Change the orgnization. Choose another organization that can run the build correctly, and just in case, it is better to create a new organization. Then create a new project and try your tests.
As stated already, the problem most likely lies with the Azure environment. Cypress has a dependency on a browser (electron, chrome) in order to execute. For example, if you are using docker, they provide an official image called cypress/browsers:node14.7.0-chrome84 that has everything you need out of the box. The Dockerfile also has useful info on the environment needed. Make sure to provide a headless configuration as well, something like:
cypress run --headless --browser chrome --spec yourSpecHere.js
It seems that coverage report with coveralls is not possible for VSCode extension built with TypeScript.
Currently, I am adding test cases to our project https://github.com/PicGo/vs-picgo/pull/42, I have found several ways to report coverages, but none of them work for me.
Using custom TestRunner
The official documentation mentions little about custom test runners, but I found a post here. It works when I use F5 to launch an Extension Test, but does not work when I run npm run test in the console (Got no coverage output at all).
I have also tried to understand the custom runner (source code) in the blog post, but I found I have nothing to do because I do not know why it works.
Using nyc
nyc with mocha is very powerful, but we cannot take advantage of it. When I run nyc ./node_modules/vscode/bin/test, I will got 0% coverage:
I have searched the issue page of nyc, lots of the same 0% coverage problems about TS projects exist, but none of them are the same with our environment. The main difference is that they are using mocha for testing, not like VSCode's ./node_modules/vscode/bin/test script, it will create a new process to run the test js files. I don't know how to deal with this.
I searched all the issues (mocha, tyc, istanbul, vscode, etc...), and there are few (I did not find any 😭 ) vscode TypeScripts are using coverage report for me to copy from. So my question is: how do I get the coverage report for my VSCode TS extension?
I have struggled with this myself for some time until I got it working properly. There were three main challenges in getting it working:
Proper setup of the nyc instance
Preventing race conditions on startup
Capturing nyc output and displaying it in the debug console
You can find my working test runner here. I'm also sharing additional insights on my blog.
I got everything working with Mocha, NYC, and VSCode!
You can see my solution to this in https://github.com/jedwards1211/vscode-extension-skeleton.
Basically, I use Babel to transpile my .ts code with #babel/preset-typescript and babel-plugin-istanbul before running the tests. This allows me to skip the convoluted extra steps of instrumenting the tsc output and using remap-istanbul.
Then in the test runner, I use the (not really documented) NYC API to write the coverage to disk after tests finish.
Finally, in my package scripts, I run nyc report after the test command finishes.
UPDATE: you need to delete the .nyc_output folder before each test run too.
src/test/index.js
import NYC from 'nyc'
export async function run(): Promise<void> {
const nyc = new NYC()
await nyc.createTempDirectory()
// Create the mocha test
const mocha = new Mocha({
ui: 'tdd',
})
mocha.useColors(true)
const testsRoot = path.resolve(__dirname, '..')
const files: Array<string> = await new Promise((resolve, reject) =>
glob(
'**/**.test.js',
{
cwd: testsRoot,
},
(err, files) => {
if (err) reject(err)
else resolve(files)
}
)
)
// Add files to the test suite
files.forEach(f => mocha.addFile(path.resolve(testsRoot, f)))
const failures: number = await new Promise(resolve => mocha.run(resolve))
await nyc.writeCoverageFile()
if (failures > 0) {
throw new Error(`${failures} tests failed.`)
}
}
Add custom test runner
See this post for more information, you can just copy the test runner code to your project's test/index.ts file.
Demo azure pipeline configurations
variables:
system.debug: true
jobs:
- job: Windows
pool:
name: Hosted VS2017
demands: npm
steps:
- task: NodeTool#0
displayName: 'Use Node 12.3.1'
inputs:
versionSpec: 12.3.1
- task: Npm#1
displayName: 'Install dependencies'
inputs:
verbose: false
- task: Npm#1
displayName: 'Compile sources and run tests'
inputs:
command: custom
verbose: false
customCommand: 'test'
# https://stackoverflow.com/questions/45602358/lcov-info-has-absolute-path-for-sf
- script: 'sed -i -- 's/..\\..\\//g' coverage/lcov.info && npm run coveralls'
displayName: 'Publish code coverage'
env:
COVERALLS_SERVICE_NAME: $(COVERALLS_SERVICE_NAME)
COVERALLS_REPO_TOKEN: $(COVERALLS_REPO_TOKEN)
- script: 'npm install -g vsce && vsce package'
displayName: 'Build artifact'
- task: CopyFiles#2
inputs:
contents: '*.vsix'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: vs-picgo-dev-build
trigger:
branches:
include:
- '*' # must quote since "*" is a YAML reserved character; we want a string
pr:
- dev*
Note that you have to use sed to delete the ..\..\ prefix of SF paths in lcov.info:
Before:
SF:..\..\src\vs-picgo\index.ts
After:
SF:src\vs-picgo\index.ts
Demo project: https://github.com/PicGo/vs-picgo
I'm trying to run an npm install and test on Visual Studio Team Services Build (configuring the build using YAML). I have the following definition.
** updated **
queue:
name: Hosted VS2017
demands: npm
steps:
- task: NodeTool#0
inputs:
versionSpec: "8.x"
task: Npm#1
inputs:
command: "custom"
verbose: "false"
customCommand: "install #angular/cli -g"
task: Npm#1
inputs:
verbose: "false"
task: CmdLine#1
inputs:
filename: "ng"
arguments: "test --watch=false --single-run=true --reporters=junit,progress"
task: PublishTestResults#2
inputs:
testResultsFiles: "**\test.xml"
testRunTitle: "Jasmine Tests"
The install runs fine, but I get an error on the test step:
85% chunk id optimization 86% hashing 87% module assets processing 88% chunk assets processing 89% additional chunk assets processing 90% recording 91% additional asset processing 92% chunk asset optimization 94% asset optimization 95% emitting
2017-12-06T06:21:34.7643927Z ##[error]Process completed with exit code 1.
2017-12-06T06:21:34.7682485Z ##[section]Finishing: Run Tests
I'm using puppeteer to workaround lack of chrome to run the tests.
Using this code instead:
queue: Hosted VS2017
steps:
- script: echo installing packages
- task: Npm#1
displayName: npm install
inputs:
command: install
- task: Npm#1
displayName: npm test
inputs:
command: custom
customCommand: run test
BTW, you need add #angular/cli dependency in package.json to install angular cli during running npm install command.
Finally got it working, but not 100% satisfied with the solutiuon, posting here the progress and open points plus detailed info to reproduce it, hope it can help other and that we can refine it.
Scenario: I'm in the need of configuring CI on an Angular 4 project and running it under Visual Studio Team Service CI hosted environment, and I want to use the yaml based config.
Issues that I have found:
Hosted enviroment come with old node version (1).
Chrome is not installed by default on the hosted environment, phantom browser is not anymore a solution since it's an abandoned project.
You need to install angular-cli globally.
The current working solution I got, steps:
On your project:
Open your command prompt, install puppeteer, karma-junit-reporter
npm install puppeteer karma-junit-reporter --save-dev
Configure your karma.conf.js
At the top of your file add the following line of code
process.env.CHROME_BIN = require('puppeteer').executablePath();
Then on the pluging sections (add karma-junit-reporter)
plugins: [
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-junit-reporter'), // add karma-unit reporter
require('karma-coverage-istanbul-reporter'),
require('#angular/cli/plugins/karma')
],
On the reporters section set junit reporter as a valid one,and configure it:
reporters: config.angularCli && config.angularCli.codeCoverage
? ['progress', 'coverage-istanbul', 'junit']
: ['progress', 'kjhtml', 'junit'],
junitReporter: {
outputDir: '',
outputFile: 'test.xml',
},
Now it's time to add on the root of your project a file named .vsts-ci.yml (just pasted the raw content in this link: https://pastebin.com/0XpmpPrp)
queue:
name: Hosted VS2017
demands: npm
steps:
- task: NodeTool#0
displayName: "Updating node to version 8.x"
inputs:
versionSpec: "8.x"
task: Npm#1
displayName: "installing angular-cli"
inputs:
command: "custom"
verbose: "false"
customCommand: "install #angular/cli#1.0.2 -g"
task: Npm#1
inputs:
verbose: "false"
task: CmdLine#1
displayName: "run tests: ng test"
inputs:
filename: "ng"
arguments: "test --watch=false --single-run=true --reporters=junit,progress"
task: PublishTestResults#2
displayName: "Publish Test Results (test.xml)"
inputs:
testResultsFiles: "**\test.xml"
testRunTitle: "Jasmine Tests"
What it does:
Update node version.
Globally install angular-cli (I stick to the version that is being used in the package).
Execute a noop command (I don't know why it's needed, but if I don't add this it breaks).
Execute ng test (single run and using junit as reporter).
Publish test results.
Tips and interesting posts I have found:
Use the UI build tool to create the build, then there is an option to convert it to yaml.
Node tool updated task: https://learn.microsoft.com/en-us/vsts/build-release/tasks/tool/node-js
Puppeteer and VSTS: http://benjaminspencer.me/post/14/headless-chrome-vsts
Angular 4 and VSTS (I): https://csharperimage.jeremylikness.com/2016/12/integrating-angular-2-unit-tests-with.html
Angular 4 and VSTS (II): https://blogs.msdn.microsoft.com/premier_developer/2017/05/17/integrating-angular-4-unit-tests-with-visual-studio-team-services-vsts/
Room for improvement:
Avoid having to install angular-cli globally, tried to add a custom script to my package-json, but when I call something like "npm run test:ci" it's not recognized by the visual studio build runner.
I would like to get rid of puppeteer, is karma-jsdom-launcher an option?