Haskell QuickCheck Test not running properly when run with Cabal - haskell

Question

Something did not make sense with a Cabal package I was developing, and I have boiled the issue down to the following example:
I have the following simple test module:
module Main where
import Test.QuickCheck (quickCheck)
main = quickCheck False
And the following Cabal file in the same directory:
name: project
version: 0.1.0.0
cabal-version: >= 1.10
build-type: Simple
executable project
main-is: Main.hs
build-depends: base, QuickCheck
default-language: Haskell2010
test-suite project-test
type: exitcode-stdio-1.0
main-is: Main.hs
build-depends: base, QuickCheck
default-language: Haskell2010
The only other files in the directory are dist (created by Cabal upon build) and the cabal sandbox files. Both the executable and the test-suite refer to Main.hs, and therefore I would expect to get the same test results when running "cabal run" as when running "cabal test". Apparently, however, that is not the case.
"cabal run" gives:
Preprocessing executable 'project' for project-0.1.0.0...
Running project...
*** Failed! Falsifiable (after 1 test):
which makes sense because the property test 'quickCheck False' should fail. As expected this is the same result I get when running main in ghci.
"cabal test", however, gives:
Test suite project-test: RUNNING...
Test suite project-test: PASS
Test suite logged to: dist/test/project-0.1.0.0-project-test.log
1 of 1 test suites (1 of 1 test cases) passed.
Why the heck does "cabal test" pass the test case, whereas "cabal run" fails it as expected?

Answer 1

quickCheck doesn't exit the program, therefore, it doesn't set the exit code. After all, you could have several quickCheck tests, which should be independent of each other:
main = do
quickCheck prop1 -- should this exit if the test fails?
quickCheck prop2 -- should this exit if the test fails?
quickCheck prop3 -- should this exit if the test fails?
However, you can easily fix this if you a) exit as soon as one of your tests doesn't pass or b) remember whether a single test hasn't passed and then exit with the correct code.
Using only QuickCheck, but no other library
Exit with failure as soon as a test fails
For this, you simply use quickCheckResult, which you can check with isSuccess from Test.QuickCheck.Test. If you want to use your current quickCheck definition, you can use qualified includes to exchange the default implementation with your special one:
import Control.Monad (when)
import System.Exit (exitFailure)
import Test.QuickCheck hiding (quickCheck, quickCheckWith)
import qualified Test.QuickCheck.Test as Q
quickCheckWith :: Testable prop => Args -> prop -> IO ()
quickCheckWith args p = do
success <- fmap Q.isSuccess $ quickCheckWithResult args p
when (not success) $ exitFailure -- exit if the result is a failure
quickCheck :: Testable prop => prop -> IO ()
quickCheck p = quickCheckWith stdArgs p
You should probably use another name though, especially if other people work on the same project. checkOrExit would be plausible.
Return failure code but run all tests
This is essentially the same, but runs all test. You must use quickCheckResult or quickCheckWithResult again:
import Control.Monad (when)
import System.Exit (exitFailure)
import Test.QuickCheck (quickCheckResult)
import Test.QuickCheck.Test (isSuccess)
main :: IO ()
main = do
let tests = [ quickCheckResult prop1
, quickCheckResult prop2
, quickCheckResult prop3
, quickCheckResult prop4
]
success <- fmap (all isSuccess) . sequence $ tests
when (not success) $ exitFailure
Using additional test libraries
While quickCheck is great for property checking, it doesn't provide a complete testing framework. That's where other full-fledged frameworks such as tasty or hspec come in handy. They can take a Testable a and inspect QuickCheck's result accordingly. An example using hspec:
module Main where
import Test.Hspec
import Test.QuickCheck (property)
main = hspec $ do
describe "<method you would like to test>" $
it "<property/assumption you would like to test>" $
property $ False -- quickCheck
This gives the (more verbose) output
<method you would like to test>
- <property/assumption you would like to test> FAILED [1]
1) <method you would like to test> <property/assumption you would like to test>
Falsifiable (after 1 test):
[remark: the used values would be here, but `False` is a boolean]
Randomized with seed 2077617428
Finished in 0.0019 seconds
1 example, 1 failure
Also, it exits with an error code, so your cabal test will recognize this failed test correctly. Those test frameworks also have additional features, which go beyond of the scope of this answer.
TL;DR
QuickCheck doesn't exit your program, but Cabal only inspects the exit code to determine whether the tests have passed. Since any regularly ending Haskell program returns zero (aka no error), you either need to use exitFailure (or something similar), or a framework that uses exitFailure behind the scenes.
Note that this only holds for tests with type: exitcode-stdio-1.0.