Measure total time spent by a process on IO - linux

Question

While running time command, one of the programs gives following output:
real 1m33.523s
user 0m15.156s
sys 0m1.312s
Here the real and user+sys time have a lot of difference. This is most likely due to time spent on IO wait/calls. I want to measure total time spend by program in IO wait or IO calls. Is there any way to do that?
I tried using iotop. However, it doesnot report total time spent by the program performing IO.

Answer 1

Yes, strace - which can provide per-system-call statistics.
Example 1
I want to measure time spent on I/O while accessing stackoverflow.com:
$ time curl stackoverflow.com >/dev/null 2>&1
curl stackoverflow.com > /dev/null 2>&1 0.00s user 0.01s system 2% cpu 0.392 total
OK, 2% CPU and 0.01 s in system. Let's find out:
$ strace -c curl stackoverflow.com >/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 240k 0 240k 0 0 127k 0 --:--:-- 0:00:01 --:--:-- 130k
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
54.12 0.005497 11 506 write
18.16 0.001845 43 43 fstat
11.95 0.001214 30 41 poll
5.75 0.000584 32 18 recvfrom
3.40 0.000345 3 101 mmap
2.51 0.000255 4 62 mprotect
1.98 0.000201 4 50 close
1.84 0.000187 31 6 getsockname
0.29 0.000029 1 42 1 open
Especially useful compare this results with results measured for runing curl without args.
Anyway. strace shows that curl mostly spends time in write, fstat and poll.
Another example
The first approach seems show incorrect results for sleep. If you are not satisfied with the first approach you can just print get times of each syscall (strace -T). Get this data and process them to find summary time of each syscall.
$ strace 2>&1 -T curl stackoverflow.com >/dev/null | head -n 20
execve("/usr/bin/curl", ["curl", "stackoverflow.com"], [/* 62 vars */]) = 0 <0.000219>
brk(0) = 0x186e000 <0.000175>
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc04c9e6000 <0.000166>
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000238>
open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000144>
fstat(3, {st_mode=S_IFREG|0644, st_size=96498, ...}) = 0 <0.000175>
mmap(NULL, 96498, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc04c9ce000 <0.000164>
close(3) = 0 <0.000160>
open("/usr/lib64/libcurl.so.4", O_RDONLY) = 3 <0.000047>
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\333\300\">\0\0\0"..., 832) = 832 <0.000160>
fstat(3, {st_mode=S_IFREG|0755, st_size=346008, ...}) = 0 <0.000216>
mmap(0x3e22c00000, 2438600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e22c00000 <0.000189>
mprotect(0x3e22c51000, 2097152, PROT_NONE) = 0 <0.000032>
mmap(0x3e22e51000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x51000) = 0x3e22e51000 <0.000119>
close(3) = 0 <0.000110>
open("/lib64/libidn.so.11", O_RDONLY) = 3 <0.000257>
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0/#U1\0\0\0"..., 832) = 832 <0.000051>
fstat(3, {st_mode=S_IFREG|0755, st_size=209088, ...}) = 0 <0.000041>
mmap(0x3155400000, 2301736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3155400000 <0.000037>
mprotect(0x3155432000, 2093056, PROT_NONE) = 0 <0.000037>