When I run the test suite using cabal test, I got the following message:
Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to: my-lib-tests.log
But when I looked at the log file, the content was:
Test suite tests: RUNNING...
*** Failed! Falsified (after 1 test):
[]
Test suite tests: PASS
Test suite logged to: my-lib-tests.log
Why did I get a pass message when the tests clearly failed?
cabal test works under the assumption that a failing test suite will exit with a non-zero error code.
quickCheck prints a counterexample but returns normally.
To make the test executable fail when a counterexample is found, you can wrap QuickCheck tests using quickCheckResult and isSuccess.
There are test frameworks that do this for you, with a lot of useful functionality on top (like command-line arguments to select the tests to run), like tasty, with tasty-quickcheck.
Related
I've made a very simple project with a failing test suite https://github.com/k-bx/noruntests-play
Now when I run stack --test --no-run-tests build I would expect it to build the project, but not run the tests. Instead, it runs the tests:
➜ noruntests-play git:(master) stack --test --no-run-tests build
noruntests-play-0.1.0.0: test (suite: test)
test: error
CallStack (from HasCallStack):
error, called at tests/Tests.hs:4:8 in main:Main
Test suite failure for package noruntests-play-0.1.0.0
test: exited with: ExitFailure 1
Logs printed to console
What am I doing wrong here? Thank you!
You should put build option before --test like this:
$ stack build --test --no-run-tests
I'm not sure whether it's bug or feature. You can open issue here if you're interested in feedback from developers. Personally for me it seems strange to pass --test before build. In some reasonable sense --test is subpart of build and subparts or options are usually specified to the right of main option.
Also there's shorter version of what you want (because build --test is just test):
$ stack test --no-run-tests
Setup: .cabal file with library, executable that depends on the library and a test-suite. The test-suite calls integration test written in python. However, that gives me empty coverage reports from cabal test. I think that's because python calls the executable not the test-suite?
Questions
What is the difference between "Package coverage report" and the "Test coverage report"?
Should I call the test-suite binary from python, where the test-suite has a flag such that it behaves like the executable?
I want to generate code coverage report using Stack. I run command that
amounts to (omitting options passed to test suite via --test-arguments):
$ stack test --coverage
This performs the testing and then outputs the following:
Error: The coverage report for myproject's test-suite "tests" did not
consider any code. One possible cause of this is if your test-suite builds
the library code (see stack issue #1008). It may also indicate a bug in
stack or the hpc program. Please report this issue if you think your
coverage report should have meaningful results.
I think it should (this creates empty report). GHC options are identical for
all components of my package. There is no need for test suite to rebuild the
library. After all, if Cabal can generate the report, Stack should be able
to do it given the same Cabal config or am I mistaken?
I've opened
an issue on Stack
GitHub repo as suggested.
After a while I decided to create good old sandbox and generate the report
using Cabal instead (I really need to see the report, you know). It worked
previously, but now I get:
$ cabal sandbox init
… <everything OK>
$ cabal update
… <everything OK>
$ cabal install --only-dependencies --enable-tests
… <everything OK>
$ cabal configure --enable-tests --enable-coverage
… <everything OK>
$ cabal build
… <everything OK>
$ cabal test
Running 2 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to: dist/test/myproject-0.1.0-tests.log
hpc: can not find HUnit_DDLSMCRs3jyLBDbJPCH01j/Test.HUnit.Lang in ["./.hpc","./dist/hpc/vanilla/mix/myproject-0.1.0","./dist/hpc/vanilla/mix/tests"]
What? I've never seen this, although I generated many reports
before. Someone up there just decided that I won't get that report today,
it seems.
Do you know how to generate coverage report using Stack? Has anyone
succeeded at this?
In my case I was still getting this error. Running:
stack clean
stack test --coverage
solved the problem, as reported here.
Recent changes upstream fixed it. Should be resolved for users of 0.1.7.0 and later.
I have this Spec.hs:
import qualified OtherSpec
main :: IO ()
main = do
putStrLn "Test..."
OtherSpec.main
And I am planning to add more test suites from other modules to its main.
When I run tests as:
$ stack test --file-watch
A few things happen:
Tests are re-run when OtherSpec.hs is changed, but the test executable is not rebuilt.
When I change Spec.hs test executable is rebuilt and changes in OtherSpec.hs is also picked.
Is there a way to trigger a rebuild when OtherSpec.hs changes?
I am using stack version 0.1.2.0.
I am trying to test my project which uses "data-binary-ieee754" with doctests.
I uses cabal-dev, instead of cabal, to manage package dependencies.
I can build the project, but doctest seems not to recognize that package.
doctests definition in .cabal:
test-suite doctests
type: exitcode-stdio-1.0
hs-source-dirs: test
main-is: doctests.hs
ghc-options: -Wall -threaded
build-depends: base,
doctest >= 0.7,
data-binary-ieee754
test/doctests.hs:
module Main where
import Test.DocTest
main :: IO ()
main = doctest ["src/Pattern.hs"]
The error message for cabal-dev test doctests is:
Running 1 test suites...
Test suite doctests: RUNNING...
src/Pattern.hs:13:8:
Could not find module `Data.Binary.IEEE754'
Use -v to see a list of the files searched for.
Test suite doctests: FAIL
Test suite logged to: dist/test/othello-0.1.0-doctests.log
0 of 1 test suites (0 of 1 test cases) passed.
I tried to add some options to doctests.hs, like
main = doctest ["--optghc=-Lcabal-dev/lib",
"--optghc=-packagedata-binary-ieee754",
"src/Pattern.hs"]
but the result is
Running 1 test suites...
Test suite doctests: RUNNING...
doctests: <command line>: cannot satisfy -package data-binary-ieee754
(use -v for more information)
Test suite doctests: FAIL
Test suite logged to: dist/test/othello-0.1.0-doctests.log
0 of 1 test suites (0 of 1 test cases) passed.
Tell me how to do this correctly. Thanks.
I found the answer by myself.
http://hackage.haskell.org/trac/ghc/ticket/6133 was helpful.
main :: IO ()
main = doctest ["--optghc=-Lcabal-dev/lib",
"--optghc=-packagedata-binary-ieee754",
"--optghc=-package-conf=cabal-dev/packages-7.4.1.conf",
"src/Pattern.hs"]
Just to update this answer a bit, doctest now allows you to call ghc options directly.
You can also load a sandboxed package database and call from cabal.
doctest [ "-package-db .cabal-sandbox/x86_64-linux-ghc-7.10.3-packages.conf.d"
, "-isrc"
, "src/<path-to-file>"]
This cleared up missing package problems for me.