Performance issue when calling isSpaceAscii - nim-lang

Question

I tried to call the isSpaceAscii from the standard library but got worse performance than with my own proc.
Code to reproduce:
import strutils
import std/monotimes
import stats
template timeIt(tag: string, iter: untyped, body: untyped) =
var st: RunningStat
for i in countup(1, iter):
let t0 = getMonoTime().ticks
body
let t1 = getMonoTime().ticks
let d = t1 - t0
st.push(d.float64)
echo tag, ": ", st.min
proc isSpace(c: char): bool =
result = c in Whitespace
when isMainModule:
# check eqaulity
for i in 1..255:
let c = char(i)
doAssert isSpace(c) == isSpaceAscii(c)
timeIt "isSpaceAscii", 1000:
for i in 1..255:
let c = char(i)
discard isSpaceAscii(c)
timeIt "isSpace", 1000:
for i in 1..255:
let c = char(i)
discard isSpace(c)
Benchmark results:
$ nim compile -d:release --verbosity:0 --hints:off --run test.nim
isSpaceAscii: 380.0
isSpace: 20.0
Compiler version:
$ nim -V
Nim Compiler Version 1.4.2 [Linux: amd64]
Why is isSpace faster than isSpaceAscii?

Answer 1

Benchmarking is hard, because you're not always measuring what you think you're measuring.
The incredibly stark differences you're seeing are because the isSpace loop doesn't do anything, and is in the same compilation unit as the isSpace function, so the compiler can optimize it away
as you can see on godbolt
if you instead compile with -d:release -d:lto the compiler will perform link-time optimization, and will optimize away both versions.
$ nim c -d:release -d:lto -r test.nim
isSpaceAscii: 16
isSpace: 16
We're just measuring the loop overhead.
To actually compare isSpaceAscii with isSpace, they need to do actual work as far as the compiler is concerned.
import std/[strutils,monotimes,stats]
template timeIt(tag: string, iters: int, body: untyped) =
var st: RunningStat
when declared(warmup): #BUG
for i in 1..iters:
body
for i in 1..iters:
let t0 = getMonoTime().ticks
body
let t1 = getMonoTime().ticks
st.push((t1-t0).float64)
echo tag,": ", st.min
proc isSpace(c:char):bool = c in Whitespace
template badloop(procname: untyped) =
for i in 1..255:
let c = char(i)
discard procname(c)
template goodloop(procname: untyped) =
var x: int
for i in 1..255:
let c = char(i)
if procname(c): inc x
when isMainModule:
let nruns = 1000
for i in 1..255:
doAssert isSpace(i.char) == isSpaceAscii(i.char)
timeit "isSpaceAscii, good",nruns:
goodloop(isSpaceAscii)
timeit "isSpace, good",nruns:
goodloop(isSpace)
timeit "isSpaceAscii, bad",nruns:
badloop(isSpaceAscii)
timeit "isSpace, bad",nruns:
badloop(isSpace)
result:
$ nim -d:release -d:lto -r test.nim
isSpaceAscii, good: 439.0
isSpace, good: 382.0
isSpaceAscii, bad: 17.0
isSpace, bad: 17.0
Closer, but there still seems to be a discrepancy, what's that about?
T̶h̶e̶ ̶c̶p̶u̶ ̶h̶a̶s̶ ̶w̶a̶r̶m̶e̶d̶ ̶u̶p̶ ̶b̶y̶ ̶t̶h̶e̶ ̶t̶i̶m̶e̶ ̶i̶t̶ ̶g̶e̶t̶s̶ ̶t̶o̶ ̶t̶h̶e̶ ̶s̶e̶c̶o̶n̶d̶ ̶t̶e̶s̶t̶,̶ ̶a̶n̶d̶ ̶i̶t̶ ̶g̶o̶e̶s̶ ̶f̶a̶s̶t̶e̶r̶.̶ (Edit due to a bug -d:warmup didn't change the code, differences were due to the compiler making different optimization choices.)
Try again, t̶u̶r̶n̶i̶n̶g̶ ̶o̶n̶ ̶t̶h̶e̶ ̶w̶a̶r̶m̶u̶p̶ ̶l̶o̶o̶p̶ we've added to timeit, and this time lets go all in for speed and use -d:danger
$ nim c -d:danger -d:lto -d:warmup -r test.nim
isSpaceAscii, good: 237.0
isSpace, good: 234.0
isSpaceAscii, bad: 15.0
isSpace, bad: 15.0
pretty much the same.
Edit as if to highlight the inscrutibility of microbenchmarks, I was completely wrong about the warmup section. I should have written when defined(warmup), and because of that bug, my "warmup" code never actually ran. Indeed, since we took the fastest time, the first few runs were sufficient warmup.
I've run several versions of the code since then, and the results vary too widely to draw any more conclusions, other than maybe:
compilers' optimization choices are fickle
Benchmarking is hard