Running stack test --coverage generates a nice HTML report showing what lines your test suite covers. How can I achieve the same thing using cabal new-test?
I'm able to pass --enable-coverage to generate a .tix file but I'm not sure what to run on the .tix file to generate the HTML report. I'm pretty sure it involves hpc but I haven't been able to work out the right command.
I have the standard Cabal configuration of my application being a library, with a test-suite for that library.
It appears it's as easy as passing --enable-coverage to cabal new-test. I had previously been running tests with cabal new-run test:test to workaround some limitations of new-test (e.g. lacking streaming and colors), so the fix is to use new-test instead of new-run.
Cabal 3.6 should be able to generate the HPC report. There is one caveat; this error may appear:
Error:
Internal libraries only supported with per-component builds.
Per-component builds were disabled because program coverage is enabled
https://github.com/haskell/cabal/issues/6440
To avoid the error, add to cabal.project:
package *
coverage: True
library-coverage: True
then cabal test (without --enable-coverage). The report should be somewhere in dist-newstyle.
Related
Several builds, like this one, fail when executing cabal check:
++cabal check
These warnings may cause trouble when distributing the package:
* 'ghc-options: -O2' is rarely needed. Check that it is giving a real benefit
and not just imposing longer compile times on your users.
However, most of the other builds in the matrix do not fail after this check.
I'm using the complex Travis configuration suggested at the stack docs, and this is the Travis configuration specific for the project I'm trying to get on CI.
Any ideas on what might be causing this behavior?
There are two types of build in your travis config:
Stack based build
Cabal based build
If you follow the script code, you will see that only Cabal based build has the command cabal check in it. That will explain why all of your Stack based builds are working fine. Now, let's see the cabal check command line in detail:
cabal check || [ "$CABALVER" == "1.16" ]
So, if your installed cabal version is 1.16, it will ignore the output of cabal check and that command is treated as a success. And infact, that's what is happening. Only one Cabal based build job is success in your travis, because it's version is 1.16.
cabal test will compile the test suite executable and then run it. However, it removes the console colouring (because it logs the result). I'd like to use cabal to build the executable and then run it from a script, but I can't figure out how to just build the executable.
If you cabal configure --enable-tests, then cabal build will build not just the library/executables, but also the test suites. You can also build individual test suites by name, saying cabal build name-of-test-suite.
You can run them manually from the appropriate subdirectories under dist/build if you don't want to use cabal test.
Furthermore, if it's e.g. a tasty test suite, you may be able to get color output by saying something like
cabal test --show-details=always --test-option=--color --test-option=always
You can also try --show-details=streaming. I don't know how robust this is, though, and whether it works may also depend on the platform you are on.
Yesod provides the yesod test tool. A test is a value of type Spec which can be executed by hspec.
The scaffolding comes with a predefined spec in tests/HomeTest.hs which is explicitly called in tests/main.hs, the file that is apparently compiled and executed when you run yesod test. I assume you are supposed to manually add all your specs to main.hs as you create them, although perhaps there is a way to use hspec's automated test discovery.
This is great for regression testing, but what about test-driven development? What is the proper way to write a single test and then run it repeatedly during development? The help text for yesod test suggests that there is no way to do this:
$ yesod test --help
Usage: yesod test
Build and run the integration tests
Running a spec file directly doesn't work (and I suppose you wouldn't expect it to, since it doesn't contain a main definition):
$ runhaskell tests/HomeTest.hs
tests/HomeTest.hs:4:8:
Could not find module ‘TestImport’
Use -v to see a list of the files searched for.
$ cabal run tests/HomeTest.hs
cabal: Cannot build the executable 'myproject' because the component is marked as disabled in the .cabal file.
I'm not sure why my project's executable is disabled. Should I change that? Or should I create a temporary copy of main.hs and comment out all the other tests? What's the cleanest solution?
When using cabal install --enable-tests --enable-library-coverage to generate code coverage reports it will also generate code coverage reports for the testcode. Is there an option for cabal to let hpc ignore testcode?
I'd be pleasantly surprised if cabal yet has an option for this. Getting HPC to ignore things is (or perhaps used to be) moderately complicated as this example shows: http://www.haskell.org/haskellwiki/Haskell_program_coverage#Example
When i google for how to integrate unit tests with cabal files, i either find
http://www.haskell.org/haskellwiki/How_to_write_a_Haskell_program which does not seem to describe the integration of HUnit/QuickCheck with the Cabal file
or i see messages like "wait for Cabal x.y which will support cabal test" but i can not find any documentation for this either
How would you run all unit test using cabal (for example everytime i do a "cabal build") today?
Make sure you have the latest version of Cabal and cabal-install installed.
Have a test-suite section in your .cabal file. See this section of cabal's documentation for an explanation of how to write a test-suite section in your Cabal file and this section for instructions on how to run it.
I've been using the built-in test support for some time and it has saved me from having to maintain fragile Makefiles just for my tests. There are still some rough edges in the command line output of cabal test, but they have been fixed in HEAD so in the next Cabal/cabal-install release everything should be very smooth.