I am running jest test with code coverage in GitLab CI and GitLab captures the percentage from stdout of a runner in gitlab.
jest --coverage produces the coverage in stdout and gitlab captures it using /All files[^|]*\|[^|]*\s+([\d\.]+)/ regexp but when I run
jest --coverage --json --outputFile=xyz.json sadly jest doesn't print the code coverage to stdout.
What can I do to get code coverage in stdout from jest when --json arguments is given to jest?
jest version : v22.4.3 same for jest-cli
The following configuration will let GitLab interpret the coverage report generated by Jest:
stages:
- test
Unit tests:
image: node:12.17.0
stage: test
script:
- jest --coverage
coverage: /All\sfiles.*?\s+(\d+.\d+)/
There's an open issue on GitLab which contains the correct regex for coverage reports generated using Jest (which is used by Create React App).
I'm using the following regex to parse the text-summary coverage reports from Jest for Gitlab: ^(?:Statements|Branches|Functions|Lines)\s*:\s*([^%]+)
Note that Gitlab will only consider the last match though. So above could be written as ^Lines\s*:\s*([^%]+). I included the full example so that you can choose the one that makes the most sense for your project.
The "text-summary" report looks like this in StdOut:
=============================== Coverage summary ===============================
Statements : 80.49% ( 2611/3244 )
Branches : 65.37% ( 923/1412 )
Functions : 76.48% ( 582/761 )
Lines : 80.44% ( 2583/3211 )
================================================================================
Make sure you have included text-summary as a coverage reporter in your jest.config.js:
coverageReporters: ['text-summary', 'lcov', 'cobertura'],
I'm not familiar with Jest, but if you are creating a JSON the simplest way would be to simply cat the JSON then change the regex accordingly
Related
We have jest unit testing for our react app and need to set a threshold value of 80% test case coverage. I know that we can get the coverage report in npm test -- --coverage --watchAll=false but I am now tasked with failing the pipeline if the coverage goes below 80%. I saw that there is a test pipeline stage which is commented right now.
I have the following script, I need to somehow get the coverage, and compare if it is 80 or more else fail the pipleline, what should I do
test:
stage: test
image: node:16.13.1
before_script:
- npm i
- npx node -v
- npx npm -v
script:
- echo "running test coverage"
- npm test -- --coverage --watchAll=false
coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
Within your script section you should be able to use a Regex expression to retrieve the coverage value from the report.
Then you can use some bash scripting to remove the % sign from the output and compare the value with your required minimum. If the value is below, simply return false which I think gitlab should interpret as a failure.
If it’s above your minimum value return true and it should pass the job.
I am trying to split my pytests in a gitlab stage to reduce the time it takes to run them. However, I am having difficulties getting the full coverage report. I am unable to use pytest-xdist or pytest-parallel due to the way our database is set up.
Build:
stage: Build
script:
- *setup-build
- *build
- touch banana.xml # where I write the code coverage collected by pytest-cov
- *push
artifacts:
paths:
- banana.xml
reports:
cobertura: banana.xml
Unit Test:
stage: Test
script:
- *setup-build
- *pull
- docker-compose run $DOCKER_IMG_NAME pytest -m unit --cov-report xml:banana.xml --cov=app --cov-append;
needs:
- Build
Validity Test:
stage: Test
script:
- *setup-build
- *pull
- docker-compose run $DOCKER_IMG_NAME pytest -m validity --cov-report xml:banana.xml --cov=app --cov-append;
needs:
- Build
After these two stages run (Build - 1 job, Test - 2 jobs), I go to download the banana.xml file from Gitlab but there's nothing in it, even though the jobs say Coverage XML written to file banana.xml
Am I missing something with how to get the total coverage written to an artifact file when splitting up marked tests in a gitlab pipeline stage?
If you want to combine the coverage reports of several different jobs, you will have to add another stage which will run after your tests. Here is a working example for me :
# You need to define the Test stage before the Coverage stage
stages:
- Test
- Coverage
# Your first test job
unit_test:
stage: Test
script:
- COVERAGE_FILE=.coverage.unit coverage run --rcfile=.coveragerc -m pytest ./unit
artifacts:
paths:
- .coverage.unit
# Your second test job which will run in parallel
validity_test:
stage: Test
script:
- COVERAGE_FILE=.coverage.validity coverage run --rcfile=.coveragerc -m pytest ./validity
artifacts:
paths:
- .coverage.validity
# Your coverage job, which will combine the coverage data from the two tests jobs and generate a report
coverage:
stage: Coverage
script:
- coverage combine --rcfile=.coveragerc
- coverage report
- coverage xml -o coverage.xml
coverage: '/\d+\%\s*$/'
artifacts:
reports:
cobertura: coverage.xml
You also need to create a .coveragerc file in your repository with the following content, to specify that coverage.py needs to use relative file paths, because your tests were run on different gitlab runners, so their full path don't match :
[run]
relative_files = True
source =
./
Note : In your case it's better to use the coverage command directly (so coverage run -m pytest instead of pytest) because it provides more options, and it's what pytest uses under the hood anyway.
The issue in your file is that you start with creating an empty file, try to generate a report from that (which won't generate anything since the file is empty), and then pass it over to both test jobs, which both overwrite it with their local coverage report separately, and then never use it.
You need to do it the other way round, as shown in my example : run the tests first, and in a later stage, get both the test coverage data, and generate a report from that.
I am trying to show test coverage visualization in Gitlab for our monorepo as described here Test Coverage Visualization
We are using a self-managed gitlab runner with the docker+machine executor hosted on AWS EC2 instances. We are using Gitlab SaaS. The job from the gitlab-ci.yml is below
sdk:
stage: test
needs: []
artifacts:
when: always
reports:
cobertura: /builds/path/to/cobertura/coverage/cobertura-coverage.xml
<<: *main
<<: *tests
<<: *rules
<<: *tags
The line in the script that runs the tests and outputs code coverage...
- npm run test -- --watchAll=false --coverage --coverageReporters=cobertura
The artifact gets saved just fine and looks normal when I download it, but I don't get any visualization as described in the documentation linked above. I just updated the gitlab runner to V14.0.0 thinking that might be the problem, it's not.
I don't have any sort of regex pattern setup, as from my understanding that is only for printing the coverage to stdout.
I'm sure I am overlooking something trivial and I really need a sanity check here as I have already spent way more time on this than I can afford.
The issue was that the regex pattern needed to be set in the repository settings. I had experimented with adding a regex pattern, but it hadn't worked by the time I posted this question because the regex pattern I was using was not correct.
I'm using pytest with gitlab and I'm wondering if there's a way to automatically parse test results in the pipeline, so that I don't have to go manually in the terminal output and search for test names that have failed. Teamcity has such a feature by using teamcity-messages.
Does anybody know if such a feature is available for gitlab as well?
Test summary in Merge Request view
Gitlab supports parsing and rendering test results from a JUnit report file. The reserved word for that is artifacts:reports:junit. Here is an example CI config that generates a JUnit report on a pytest run and makes it available to Gitlab:
stages:
- test
test:
stage: test
script:
- pytest --junitxml=report.xml
artifacts:
reports:
junit: report.xml
Here is what the results would look like rendered in the Merge Request view:
More info (and examples for other languages) can be found in Gitlab docs: JUnit test reports.
Preview feature: test summary in the Pipeline view
On the doc page linked above, you can also find a preview feature of an extra Tests card in the pipeline view:
This feature is available since 12.5 and currently should be explicitly enabled by an admin via the :junit_pipeline_view flag.
Edit: your case
To sum up, I would rework the pytest invocation command and add the reports section to artifacts in the .gitlab-ci.yml:
test:
script:
- pytest -vv
--cov=${ROOT_MODULE}
--cov-branch
--cov-report term-missing
--cov-report xml:artifacts/coverage.xml
--junitxml=artifacts/junit.xml
artifacts:
paths:
- artifacts/coverage.xml
- artifacts/junit.xml # if you want the JUnit report to be also downloadable
reports:
junit: artifacts/junit.xml
I'm using mocha to run nodeJS tests. All tests seems to pass, but gitlab runner keeps waiting for something.
Anyone have had similar problem?
My test stage implementation looks like this.
test:
stage: test
script:
### Run with debugging
- mocha -R mocha-pretty-bunyan-nyan build/test/v1/
- mocha -R mocha-pretty-bunyan-nyan build/test/v2/
#- mocha build/test/v1/
I've also tried without the pretty reporter. Result is same. Runner gets stuck and keeps waiting for something.
Found out it is needed to run mocha tests with --exit flag.