How to measure sequential and parallel runtimes of Haskell program - haskell

Question

I am taking measurement of the haskell program from this question to produce the following table with runtimes and speedups summary so I can plot in a graph.
#Cores Runtimes Speedups
Absolute Relative
Seq ? .. ..
1 3.712 .. ..
2 1.646 .. ..
First question
While the runtimes on 1 and 2 cores are taken by compiling the program with the -threaded flag on ([3] and [4] below), I am not sure which time to take for the sequential one ([1] or [2] below):
should it be the time obtained by compiling without the -threaded flag, or
that obtained with the flag on but then NOT specifying any number of cores i.e. with no -Nx
Compiling without -threaded flag
$ ghc --make -O2 test.hs
[1] $ time ./test ## number of core = 1
102334155
real 0m4.194s
user 0m0.015s
sys 0m0.046s
Compiling with -threaded flag
$ ghc --make -O2 test.hs -threaded -rtsopts
[2] $ time ./test ## number of core = not sure?
102334155
real 0m3.547s
user 0m0.000s
sys 0m0.078s
[3] $ time ./test +RTS -N1 ## number of core = 1
102334155
real 0m3.712s
user 0m0.016s
sys 0m0.046s
[4] $ time ./test +RTS -N2 ## number of core = 2
102334155
real 0m1.646s
user 0m0.016s
sys 0m0.046s
Second question
As can be seen from above, I am using the time command to measure the runtimes. I am taking the 'real' time. But if I run the program with the -sstderr flag on, I get more detailed information:
$ ghc --make -O2 test.hs -rtsopts
$ ./test +RTS -sstderr
102334155
862,804 bytes allocated in the heap
2,432 bytes copied during GC
26,204 bytes maximum residency (1 sample(s))
19,716 bytes maximum slop
1 MB total memory in use (0 MB lost due to fragmentation)
Generation 0: 1 collections, 0 parallel, 0.00s, 0.00s elapsed
Generation 1: 1 collections, 0 parallel, 0.00s, 0.00s elapsed
INIT time 0.00s ( 0.00s elapsed)
MUT time 3.57s ( 3.62s elapsed)
GC time 0.00s ( 0.00s elapsed)
EXIT time 0.00s ( 0.00s elapsed)
Total time 3.57s ( 3.62s elapsed)
%GC time 0.0% (0.0% elapsed)
Alloc rate 241,517 bytes per MUT second
Productivity 100.0% of total user, 98.6% of total elapsed
I believe that the -sstderr provides a more accurate time which I should use instead of the time command. Am I correct? Also, which of the 'Total time' (3.57s or 3.62s) should I use?
And finally, any general advice/good practice while taking measurements like this? I am aware that there are some packages that allow us to benchmark our program, but I am mainly interested in taking the measurements manually (or using a script to do that for me).
Also: the runtimes are the median of running the program 3 times.

Answer 1

I would use -N1 for the single-core time. I believe that also constrains the GC to use one core (which seems fitting for the benchmark, I think?), but others may know more.
As for your second question, the answer to benchmarking in Haskell is nearly always to use criterion. Criterion will allow you to time one run of the program, and you can then wrap it in a script which runs the program with -N1, -N2, etc. Taking the median of 3 runs is okay as a very quick and rough indicator, but if you want to rely on the results then you'll need a lot more runs than that. Criterion runs your code enough and performs the appropriate statistics to give you a sensible average time, as well as confidence intervals and standard deviation (and it tries to correct for how busy your machine is). I know you asked about best practice for doing it yourself, but Criterion already embodies a lot of it: use clock time, benchmark a lot, and as you realised, don't just take a simple mean of the results.
Criterion requires very little change to your program if you want to benchmark the whole thing. Add this:
import Criterion.Main
main :: IO ()
main = defaultMain [bench "My program" oldMain]
where oldMain is whatever your main function used to be.