I'm trying to run an azure devops pipeline that contains a vitest run with coverage. The issue is that the azure coverage collector plugin accepts only jacoco/cobertura formats.
I've seen that for jest is it possible to run with a cobertura reporter. Is there anyway of doing this for vitest?
Thank you
Vitest supports c8 and istanbul for coverage, they both support Cobertura format.
Wherever you define your test settings vite.config.ts or vitest.config.ts set the coverage reporter to generate the cobertura format.
import { defineConfig } from 'vitest/config'
export default defineConfig({
test: {
coverage: {
reporter: ['cobertura', 'text'],
},
},
})
It will output by default at /coverage/cobertura-coverage.xml. You can then feed this into the PublishCodeCoverageResults task.
Related
I have written my first E2E tests with Cypress, and so far I am using this cypress.json file:
{
"baseUrl": "http://localhost/",
"video": false
}
Now, I want to run my E2E tests on Gitlab CI pipeline. What baseURL I use? Any help?
I'm using pytest with gitlab and I'm wondering if there's a way to automatically parse test results in the pipeline, so that I don't have to go manually in the terminal output and search for test names that have failed. Teamcity has such a feature by using teamcity-messages.
Does anybody know if such a feature is available for gitlab as well?
Test summary in Merge Request view
Gitlab supports parsing and rendering test results from a JUnit report file. The reserved word for that is artifacts:reports:junit. Here is an example CI config that generates a JUnit report on a pytest run and makes it available to Gitlab:
stages:
- test
test:
stage: test
script:
- pytest --junitxml=report.xml
artifacts:
reports:
junit: report.xml
Here is what the results would look like rendered in the Merge Request view:
More info (and examples for other languages) can be found in Gitlab docs: JUnit test reports.
Preview feature: test summary in the Pipeline view
On the doc page linked above, you can also find a preview feature of an extra Tests card in the pipeline view:
This feature is available since 12.5 and currently should be explicitly enabled by an admin via the :junit_pipeline_view flag.
Edit: your case
To sum up, I would rework the pytest invocation command and add the reports section to artifacts in the .gitlab-ci.yml:
test:
script:
- pytest -vv
--cov=${ROOT_MODULE}
--cov-branch
--cov-report term-missing
--cov-report xml:artifacts/coverage.xml
--junitxml=artifacts/junit.xml
artifacts:
paths:
- artifacts/coverage.xml
- artifacts/junit.xml # if you want the JUnit report to be also downloadable
reports:
junit: artifacts/junit.xml
It seems that coverage report with coveralls is not possible for VSCode extension built with TypeScript.
Currently, I am adding test cases to our project https://github.com/PicGo/vs-picgo/pull/42, I have found several ways to report coverages, but none of them work for me.
Using custom TestRunner
The official documentation mentions little about custom test runners, but I found a post here. It works when I use F5 to launch an Extension Test, but does not work when I run npm run test in the console (Got no coverage output at all).
I have also tried to understand the custom runner (source code) in the blog post, but I found I have nothing to do because I do not know why it works.
Using nyc
nyc with mocha is very powerful, but we cannot take advantage of it. When I run nyc ./node_modules/vscode/bin/test, I will got 0% coverage:
I have searched the issue page of nyc, lots of the same 0% coverage problems about TS projects exist, but none of them are the same with our environment. The main difference is that they are using mocha for testing, not like VSCode's ./node_modules/vscode/bin/test script, it will create a new process to run the test js files. I don't know how to deal with this.
I searched all the issues (mocha, tyc, istanbul, vscode, etc...), and there are few (I did not find any 😠) vscode TypeScripts are using coverage report for me to copy from. So my question is: how do I get the coverage report for my VSCode TS extension?
I have struggled with this myself for some time until I got it working properly. There were three main challenges in getting it working:
Proper setup of the nyc instance
Preventing race conditions on startup
Capturing nyc output and displaying it in the debug console
You can find my working test runner here. I'm also sharing additional insights on my blog.
I got everything working with Mocha, NYC, and VSCode!
You can see my solution to this in https://github.com/jedwards1211/vscode-extension-skeleton.
Basically, I use Babel to transpile my .ts code with #babel/preset-typescript and babel-plugin-istanbul before running the tests. This allows me to skip the convoluted extra steps of instrumenting the tsc output and using remap-istanbul.
Then in the test runner, I use the (not really documented) NYC API to write the coverage to disk after tests finish.
Finally, in my package scripts, I run nyc report after the test command finishes.
UPDATE: you need to delete the .nyc_output folder before each test run too.
src/test/index.js
import NYC from 'nyc'
export async function run(): Promise<void> {
const nyc = new NYC()
await nyc.createTempDirectory()
// Create the mocha test
const mocha = new Mocha({
ui: 'tdd',
})
mocha.useColors(true)
const testsRoot = path.resolve(__dirname, '..')
const files: Array<string> = await new Promise((resolve, reject) =>
glob(
'**/**.test.js',
{
cwd: testsRoot,
},
(err, files) => {
if (err) reject(err)
else resolve(files)
}
)
)
// Add files to the test suite
files.forEach(f => mocha.addFile(path.resolve(testsRoot, f)))
const failures: number = await new Promise(resolve => mocha.run(resolve))
await nyc.writeCoverageFile()
if (failures > 0) {
throw new Error(`${failures} tests failed.`)
}
}
Add custom test runner
See this post for more information, you can just copy the test runner code to your project's test/index.ts file.
Demo azure pipeline configurations
variables:
system.debug: true
jobs:
- job: Windows
pool:
name: Hosted VS2017
demands: npm
steps:
- task: NodeTool#0
displayName: 'Use Node 12.3.1'
inputs:
versionSpec: 12.3.1
- task: Npm#1
displayName: 'Install dependencies'
inputs:
verbose: false
- task: Npm#1
displayName: 'Compile sources and run tests'
inputs:
command: custom
verbose: false
customCommand: 'test'
# https://stackoverflow.com/questions/45602358/lcov-info-has-absolute-path-for-sf
- script: 'sed -i -- 's/..\\..\\//g' coverage/lcov.info && npm run coveralls'
displayName: 'Publish code coverage'
env:
COVERALLS_SERVICE_NAME: $(COVERALLS_SERVICE_NAME)
COVERALLS_REPO_TOKEN: $(COVERALLS_REPO_TOKEN)
- script: 'npm install -g vsce && vsce package'
displayName: 'Build artifact'
- task: CopyFiles#2
inputs:
contents: '*.vsix'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: vs-picgo-dev-build
trigger:
branches:
include:
- '*' # must quote since "*" is a YAML reserved character; we want a string
pr:
- dev*
Note that you have to use sed to delete the ..\..\ prefix of SF paths in lcov.info:
Before:
SF:..\..\src\vs-picgo\index.ts
After:
SF:src\vs-picgo\index.ts
Demo project: https://github.com/PicGo/vs-picgo
I've found some questions on SO specific to version for jest unit test to publish its result in VSTS build Test Results tab. But no proper solution is found.
I've used a different approach, b/c after some research I found that the Jest testResultsProcessor property is deprecated. I'm using the jest-junit package for test reports (which has been worked on more recently than the jest-trx-results-processor, fwiw):
Add jest-junit to package.json
Eg yarn add -D jest-junit or npm add --save-dev jest-junit
Add a VSTS task to run Jest using the jest-junit results reporter
I used the Yarn task, but you can alternately use the npm task. I used these task arguments:
jest --ci --reporters=jest-junit --reporters=default --coverage --coverageReporters=cobertura --coverageReporters=html
because I also wanted code coverage. To skip code coverage reporting, use these (npm or yarn) task arguments:
jest --ci --reporters=jest-junit --reporters=default
Note that --reporters=default is there b/c I wanted the default stdout in my build log.
Add a Publish Test Results task
Since we're using the default path, the test results file will be written to ~/junit.xml
(Optional) Add a publish code coverage task, too
If you're running code coverage, you might as well add a task for publishing the code coverage results, too:
If you're using a YAML pipeline, here's equivalent YAML (note that we're using the yarn task instead of npm tasks, but that can be changed):
- task: geeklearningio.gl-vsts-tasks-yarn.yarn-task.Yarn#2
displayName: 'Install dependencies'
inputs:
Arguments: install --no-progress
- task: geeklearningio.gl-vsts-tasks-yarn.yarn-task.Yarn#2
displayName: 'Unit tests'
inputs:
Arguments: 'test --ci --reporters=jest-junit --reporters=default --coverage --coverageReporters=cobertura'
continueOnError: true # Test failures should be published before failing the build
- task: PublishTestResults#2
displayName: 'Publish Jest Unit Test Results'
inputs:
testResultsFiles: junit.xml
mergeTestResults: true
testRunTitle: 'Jest Unit Tests'
failTaskOnFailedTests: true
- task: PublishCodeCoverageResults#1
displayName: 'Publish code coverage from Jest tests'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage/cobertura-coverage.xml'
# reportDirectory: '$(System.DefaultWorkingDirectory)/coverage'
failIfCoverageEmpty: true
I've gone throw some jest npm packages like tap-xunit and jest-json-to-tap but couldn't figure out it to work. Following steps worked for me to get the results to review under Test of VSTS build.
Install jest-trx-results-processor
# NPM
npm install jest-trx-results-processor --save-dev
# Yarn
yarn add -D jest-trx-results-processor
Create jestTrxProcessor.js file with following content:
var builder = require('jest-trx-results-processor');
var processor = builder({
outputFile: 'jestTestresults.trx'
});
module.exports = processor;
Updated package.json file should look like:
"devDependencies": {
"jest": "^23.4.1",
"jest-trx-results-processor": "0.0.7",
"jsdom": "^11.12.0",
...
},
"scripts": {
"test": "jest"
},
"jest": {
...,
"testResultsProcessor": "./jestTrxProcessor.js"
}
Add npm task to VSTS build for npm test. This will run jest tests and publish results to jestTestresults.trx
Add Publish Test Results task of VSTS to add jestTestresults.trx results in VSTS test.
You will be able to see JEST tests along with other tests.
Evaldas' solution is obsolete, so I'm going to add a few modifications.
The more modern solution is a combination between Evaldas' here, as well as the one from the maintainer: https://www.npmjs.com/package/jest-trx-results-processor
I'll describe it as such below.
Install jest-trx-results-processor
# NPM
npm install jest-trx-results-processor --save-dev
# Yarn
yarn add -D jest-trx-results-processor
Updated package.json file should look like:
"devDependencies": {
"jest": "^24.9.0",
"jest-trx-results-processor": "^1.0.2"
...
},
"scripts": {
"test": "jest"
},
"jest": {
...,
"reporters": [
"default",
[
"jest-trx-results-processor",
{
"outputFile": "./src/jestTestresults.trx",
"defaultUserName": "user name to use if automatic detection fails"
}
]]}
Add npm task to VSTS build for npm test in the build pipeline. It should look like this:
Add Publish Test Results task of VSTS to add jestTestresults.trx results in VSTS test. To do this, in the build pipeline, click on the 'add sign'. Look for "Publish Test Results". It'll bring up a menu like this. Since it's a .trx file, you'll need to use VSTest instead of JTest.
Finally, the build pipeline for your frontend project will look like this:
I am running jest test with code coverage in GitLab CI and GitLab captures the percentage from stdout of a runner in gitlab.
jest --coverage produces the coverage in stdout and gitlab captures it using /All files[^|]*\|[^|]*\s+([\d\.]+)/ regexp but when I run
jest --coverage --json --outputFile=xyz.json sadly jest doesn't print the code coverage to stdout.
What can I do to get code coverage in stdout from jest when --json arguments is given to jest?
jest version : v22.4.3 same for jest-cli
The following configuration will let GitLab interpret the coverage report generated by Jest:
stages:
- test
Unit tests:
image: node:12.17.0
stage: test
script:
- jest --coverage
coverage: /All\sfiles.*?\s+(\d+.\d+)/
There's an open issue on GitLab which contains the correct regex for coverage reports generated using Jest (which is used by Create React App).
I'm using the following regex to parse the text-summary coverage reports from Jest for Gitlab: ^(?:Statements|Branches|Functions|Lines)\s*:\s*([^%]+)
Note that Gitlab will only consider the last match though. So above could be written as ^Lines\s*:\s*([^%]+). I included the full example so that you can choose the one that makes the most sense for your project.
The "text-summary" report looks like this in StdOut:
=============================== Coverage summary ===============================
Statements : 80.49% ( 2611/3244 )
Branches : 65.37% ( 923/1412 )
Functions : 76.48% ( 582/761 )
Lines : 80.44% ( 2583/3211 )
================================================================================
Make sure you have included text-summary as a coverage reporter in your jest.config.js:
coverageReporters: ['text-summary', 'lcov', 'cobertura'],
I'm not familiar with Jest, but if you are creating a JSON the simplest way would be to simply cat the JSON then change the regex accordingly