Oprofile and gprof output varies for same code - linux

Question

I am running my code AMD optron 6270 machine. OS is Centos 6.2 release.
I have made a simple program as
#include<stdio.h>
#include<stdlib.h>
int calling (long a);
int calling1 (long a);
int calling2 (long a);
int calling3 (long a);
int calling4 (long a);
int calling5 (long a);
void main()
{
long a,b=0;
printf("hi");
for (a=0; a<10000000; a++) b++;
b =Calling(a);
b =Calling5(a);
b =Calling4(a);
}
int Calling(long a)
{
long b=0;
for (a=0; a<100; a++) b = Calling1(a);
return 0;
}
int Calling1(long a)
{
long b=0;
for (a=0; a<10000000; a++) b++;
b =Calling2(a);
return 0;
}
int Calling2(long a)
{
long b=0;
for (a=0;a<10000000;a++) b++;
b =Calling3(a);
return 0;
}
int Calling3(long a)
{
long b=0;
for (a=0;a<10000000;a++) b++;
b =Calling4(a);
return 0;
}
int Calling4(long a)
{
long b=0;
for (a=0; a<10000000; a++) b++;
return 0;
}
int Calling5(long a)
{
long b=0;
for (a=0; a<10000000; a++) b++;
b=0;
for (a=0; a<10000000; a++) b++;
b=0;
for (a=0 ;a<10000000; a++) b++;
b=0;
for (a=0; a<10000000; a++) b++;
b=0;
return 0;
}
While profinling this code with gprof and Oprofile. I got different reports. Lets say i run the main.exe with gprof two times:
1st report with gprof
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
24.80 2.96 2.96 101 0.03 0.03 Calling4
24.71 5.91 2.95 100 0.03 0.12 Calling1
24.63 8.84 2.94 100 0.03 0.06 Calling3
23.78 11.68 2.84 100 0.03 0.09 Calling2
1.01 11.80 0.12 1 0.12 0.12 Calling5
0.34 11.84 0.04 main
0.00 11.84 0.00 1 0.00 11.65 Calling
2nd report with gprof
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
25.13 2.99 2.99 100 0.03 0.12 Calling1
24.88 5.95 2.96 101 0.03 0.03 Calling4
24.80 8.89 2.95 100 0.03 0.06 Calling3
23.48 11.69 2.79 100 0.03 0.09 Calling2
1.02 11.81 0.12 1 0.12 0.12 Calling5
0.17 11.83 0.02 main
0.00 11.83 0.00 1 0.00 11.66 Calling
Both the reports differ. Everytime I run my main.exe i get different profiling reports.
When I tried Oprofile I also get different result as:
Oprofile report1
Using /var/lib/oprofile/samples/ for samples directory.
warning: /no-vmlinux could not be found.
CPU: AMD64 family15h, speed 2.2e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (CPU Clocks not Halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % image name symbol name
92552 24.7672 main Calling4
91610 24.5151 main Calling3
91566 24.5033 main Calling1
91469 24.4774 main Calling2
3665 0.9808 main Calling5
1892 0.5063 no-vmlinux /no-vmlinux
916 0.2451 main main
10 0.0027 libc-2.12.so profil_counter
1 2.7e-04 ld-2.12.so _dl_cache_libcmp
1 2.7e-04 ld-2.12.so _dl_relocate_object
1 2.7e-04 ld-2.12.so _dl_sysdep_start
1 2.7e-04 ld-2.12.so strcmp
1 2.7e-04 libc-2.12.so __libc_fini
1 2.7e-04 libc-2.12.so _dl_addr
1 2.7e-04 libc-2.12.so _int_malloc
1 2.7e-04 libc-2.12.so exit
Oprofile report2
Using /var/lib/oprofile/samples/ for samples directory.
warning: /no-vmlinux could not be found.
CPU: AMD64 family15h, speed 2.2e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (CPU Clocks not Halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % image name symbol name
92254 24.7719 main Calling4
91482 24.5646 main Calling1
91480 24.5641 main Calling3
91340 24.5265 main Calling2
3658 0.9822 main Calling5
1270 0.3410 no-vmlinux /no-vmlinux
916 0.2460 main main
6 0.0016 libc-2.12.so profil_counter
1 2.7e-04 ld-2.12.so _dl_lookup_symbol_x
1 2.7e-04 ld-2.12.so _dl_setup_hash
1 2.7e-04 ld-2.12.so _dl_sysdep_start
1 2.7e-04 ld-2.12.so bcmp
1 2.7e-04 libc-2.12.so __mcount_internal
1 2.7e-04 libc-2.12.so _dl_addr
1 2.7e-04 libc-2.12.so _int_free
1 2.7e-04 libc-2.12.so mcount
Can any one tell em why this happens? What are the possible reasons which cause this?
How I can avoid this situation so that I can get a constant profiling results?

Answer 1

I wouldn't be concerned about subsequent reports differing. Reports can vary drastically depending on how the program is executed. Moreover, it's difficult to say much about what occurs between the two profiles. Depending on what other processes are running, both your system's cache and TLB will most certainly be in a different state than they were during the first profile. Unless you can ensure a controlled machine state, don't expect consistent results.
It's simple to understand why reports from each tool don't agree. The two tools are fundamentally different. Oprofile is a sampling-based profiler that, in essence, periodically interrupts the CPU. Gprof is instrumentation based; it must be compiled into your program. This produces a different binary than would otherwise run had gprof not been used. As a result, gprof will over estimate timing. Use oprofile for CPU-bound processes, and gprof for I/O-bound processes.