Why do strace's timings for -c and -T disagree? - linux

Question

I have an rsync running a no-op (all files are already there) directory copy operation to a network-mounted file system.
Because all files are already there, the only thing that rsync does is lstat() syscalls.
If I strace -c this, I get this:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.076780 30 2524 lstat
------ ----------- ----------- --------- --------- ----------------
100.00 0.076780 2524 total
real 0m5.451s
But if I strace -T (showing time per syscall), I get this:
lstat("file1", {st_mode=S_IFREG|0644, st_size=32820, ...}) = 0 <0.005523>
lstat("file2", {st_mode=S_IFREG|0644, st_size=20816, ...}) = 0 <0.001529>
lstat("file3", {st_mode=S_IFREG|0644, st_size=1828312, ...}) = 0 <0.001991>
lstat("file4", {st_mode=S_IFREG|0644, st_size=1823258, ...}) = 0 <0.001326>
lstat("file5", {st_mode=S_IFREG|0644, st_size=32820, ...}) = 0 <0.006562>
lstat("file6", {st_mode=S_IFREG|0644, st_size=22578, ...}) = 0 <0.002151>
lstat("file7", {st_mode=S_IFREG|0644, st_size=32835, ...}) = 0 <0.001705>
lstat("file8", {st_mode=S_IFREG|0644, st_size=25493, ...}) = 0 <0.001492>
lstat("file9", {st_mode=S_IFREG|0644, st_size=1783930, ...}) = 0 <0.001974>
The times are completely off!
-c claims each lstat takes roughly 30 usecs/call, while -T shows about 2 ms/call.
The 2 ms make sense, that's in the order of ping speed for the network mount, but 30 microseconds is just plain impossible.
Why is the value in the usecs/call column bogus? Am I misunderstanding it?

Answer 1

From the strace man page:
-c Count time, calls, and errors for each system call and report a summary on program exit. On Linux, this attempts to show
system time (CPU time
spent running in the kernel) independent of wall clock time. If -c is used with -f or -F (below), only aggregate
totals for all traced processes
are kept.
(Emphasis added by me.) Most I/O will just make the actual asynchronous call and context switch away, rather than doing some kind of busy loop. -T will instead show the wall clock time duration between calling into the kernel, and that call returning.
Edit: In later versions, -w gives you wait times, rather than system times, so -c -w will give you times that should match -T.