I would like to find out how to trace rescheduling interrupts.
As I noticed that my application had some involuntary context switches and then cat /proc/interrupts shows that rescheduling interrupts happened.
I suspected that it has nothing to do with my application and so I created a dummy application:
#include <cstdio>
#include <cstdlib>
#include <ctime>
#include <cstdint>
int main(int argc, char** argv) {
srand(time(nullptr));
int32_t i = 0;
while (i != rand()) i = rand() * -1;
printf("%d", i);
}
Which basically never exits.
I compiled it:
/opt/rh/devtoolset-8/root/usr/bin/g++ -Wall dummy.cpp -o a -march=native -mtune=native -O0 -fno-omit-frame-pointer -std=c++17
Then, I ran taskset -c 5 ./a and then cat /proc/interrupts every second or so. I notice that Rescheduling Interrupts increases by 1-2 every second, which I don't expect. Local Timer Interrupt also increases by 1 every second, which is expected. I already isolated core 5 in boot parameter.
Another machine's rescheduling interrupts only increases by 1 every 30 mins or so.
Hence, I am looking for a generic way to trace down this kind of interrupt issues so that I can reapply the same methodology in the future for different kinds of unexpected interrupts.
My kernel version:
# uname -a
Linux localhost 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
I also have tried taskset -c 4 perf record -e cycles:pp -C 5 -g --freq=12000 to record the call graph but for some reason the call graph of the kernel functions wasn't created...only userspace's.
Update 1:
Followed #osgx's suggestion, I ran
taskset -c 4 perf record -e irq_vectors:reschedule_entry -c 1 --call-graph dwarf,4096 taskset -c 5 ./a
And then perf report --call-graph
- 100.00% 100.00% vector=253 ▒
- 76.47% _start ▒
__libc_start_main ▒
- main ▒
- 64.71% rand ▒
- 58.82% __random ▒
- 29.41% 0xffffffffb298e06a ▒
0xffffffffb2991864 ▒
0xffffffffb22a4565 ◆
0xffffffffb222f675 ▒
- 0xffffffffb299042c ▒
- 23.53% 0xffffffffb22a41e5 ▒
0xffffffffb298f8da ▒
0xffffffffb2258ec2 ▒
- 5.88% 0xffffffffb298f8da
kernel-debuginfo, kernel-debuginfo-common, glibc-debuginfo and glibc-debuginfo-common have been installed and -fno-omit-frame-pointer have been specified when compiling. Not sure why addresses show in the report instead of symbols. Idea?
Related
fio supports a whole bunch of io engines - all supported engines are present here : https://github.com/axboe/fio/tree/master/engines
I have been trying to understand the internals of how fio works and got stuck on how fio loads all the io engines.
For example I see every engine has a method to register and unregister itself, for example sync.c registers and unregisters using the following methods
fio_syncio_register : https://github.com/axboe/fio/blob/master/engines/sync.c#L448
and fio_syncio_unregister :
https://github.com/axboe/fio/blob/master/engines/sync.c#L461
My question is who calls these methods ?
To find answer I tried running fio under gdb - placed a break point in fio_syncio_register and in the main function, fio_syncio_register gets called even before main which tells me it has something to do with __libc_csu_init
and backtrace confirmed that
(gdb) bt
#0 fio_syncio_register () at engines/sync.c:450
#1 0x000000000047fb9d in __libc_csu_init ()
#2 0x00007ffff6ee27bf in __libc_start_main (main=0x40cd90 <main>, argc=2, argv=0x7fffffffe608, init=0x47fb50 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5f8)
at ../csu/libc-start.c:247
#3 0x000000000040ce79 in _start ()
I spent sometime reading about __libc_csu_init and __libc_csu_fini and every single description talks about methods being decorated with __attribute__((constructor)) will be called before main, but in the case of fio sync.c I dont see fio_syncio_register decorated with __attribute__
Can someone please help me out in understanding how this flow works ? Are there other materials I should be reading to understand this ?
Thanks
Interesting question. I couldn't figure out the answer looking at the source, so here are the steps I took:
$ make
$ find . -name 'sync.o'
./engines/sync.o
$ readelf -WS engines/sync.o | grep '\.init'
[12] .init_array INIT_ARRAY 0000000000000000 0021f0 000008 00 WA 0 0 8
[13] .rela.init_array RELA 0000000000000000 0132a0 000018 18 36 12 8
This tells us that global initializers are present in this object. These are called at program startup. What are they?
$ objdump -Dr engines/sync.o | grep -A4 '\.init'
Disassembly of section .init_array:
0000000000000000 <.init_array>:
...
0: R_X86_64_64 .text.startup
Interesting. There is apparently a special .text.startup section. What's in it?
$ objdump -dr engines/sync.o | less
...
Disassembly of section .text.startup:
0000000000000000 <fio_syncio_register>:
0: 48 83 ec 08 sub $0x8,%rsp
4: bf 00 00 00 00 mov $0x0,%edi
5: R_X86_64_32 .data+0x380
9: e8 00 00 00 00 callq e <fio_syncio_register+0xe>
a: R_X86_64_PC32 register_ioengine-0x4
...
Why, it's exactly the function we are looking for. But how did it end up in this special section? To answer that, we can look at preprocessed source (in retrospect, I should have started with that).
How could we get it? The command line to compile sync.o is hidden. Looking in Makefile, we can unhide the command line with QUIET_CC=''.
$ rm engines/sync.o && make QUIET_CC=''
gcc -o engines/sync.o -std=gnu99 -Wwrite-strings -Wall -Wdeclaration-after-statement -g -ffast-math -D_GNU_SOURCE -include config-host.h -I. -I. -O3 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -DBITS_PER_LONG=64 -DFIO_VERSION='"fio-2.16-5-g915ca"' -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DFIO_INTERNAL -DFIO_INC_DEBUG -c engines/sync.c
LINK fio
Now we know the command line, and can produce preprocessed file:
$ gcc -E -dD -std=gnu99 -ffast-math -D_GNU_SOURCE -include config-host.h -I. -I. -O3 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -DBITS_PER_LONG=64 -DFIO_VERSION='"fio-2.16-5-g915ca"' -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DFIO_INTERNAL -DFIO_INC_DEBUG engines/sync.c -o /tmp/sync.i
Looking in /tmp/sync.i, we see:
static void __attribute__((constructor)) fio_syncio_register(void)
{
register_ioengine(&ioengine_rw);
register_ioengine(&ioengine_prw);
...
Hmm, it is __attribute__((constructor)) after all. But how did it get there? Aha! I missed the fio_init on this line:
static void fio_init fio_syncio_register(void)
What does fio_init stand for? Again in /tmp/sync.i:
#define fio_init __attribute__((constructor))
So that is how it works.
I am using Docker containers based on the "ubuntu" tag and cannot get linux perf tool to display debugging symbols.
Here is what I'm doing to demonstrate the problem.
First I start a container, here with an interactive shell.
$ docker run -t -i ubuntu:14.04 /bin/bash
Then from the container prompt I install linux perf tool.
$ apt-get update
$ apt-get install -y linux-tools-common linux-tools-generic linux-tools-`uname -r`
I can now use the perf tool. My kernel is 3.16.0-77-generic.
Now I'll install gcc, compile a test program, and try to run it under perf record.
$ apt-get install -y gcc
I paste in the test program into test.c:
#include <stdio.h>
int function(int i) {
int j;
for(j = 2; j <= i / 2; j++) {
if (i % j == 0) {
return 0;
}
}
return 1;
}
int main() {
int i;
for(i = 2; i < 100000; i++) {
if(function(i)) {
printf("%d\n", i);
}
}
}
Then compile, run, and report:
$ gcc -g -O0 test.c && perf record ./a.out && perf report
The output looks something like this:
72.38% a.out a.out [.] 0x0000000000000544
8.37% a.out a.out [.] 0x000000000000055a
8.30% a.out a.out [.] 0x000000000000053d
7.81% a.out a.out [.] 0x0000000000000551
0.40% a.out a.out [.] 0x0000000000000540
This does not have symbols, even though the executable does have symbol information.
Doing the same general steps outside the container works fine, and shows something like this:
96.96% a.out a.out [.] function
0.35% a.out libc-2.19.so [.] _IO_file_xsputn##GLIBC_2.2.5
0.14% a.out [kernel.kallsyms] [k] update_curr
0.12% a.out [kernel.kallsyms] [k] update_cfs_shares
0.11% a.out [kernel.kallsyms] [k] _raw_spin_lock_irqsave
In the host system I have already turned on kernel symbols by becoming root and doing:
$ echo 0 > /proc/sys/kernel/kptr_restrict
How do I get the containerized version to work properly and show debugging symbols?
Running the container with -v /:/host flag and running perf report in the container with --symfs /host flag fixes it:
96.59% a.out a.out [.] function
2.93% a.out [kernel.kallsyms] [k] 0xffffffff8105144a
0.13% a.out [nvidia] [k] 0x00000000002eda57
0.11% a.out libc-2.19.so [.] vfprintf
0.11% a.out libc-2.19.so [.] 0x0000000000049980
0.09% a.out a.out [.] main
0.02% a.out libc-2.19.so [.] _IO_file_write
0.02% a.out libc-2.19.so [.] write
Part of the reason why it doesn't work as is? The output from perf script sort of sheds some light on this:
...
a.out 24 3374818.880960: cycles: ffffffff81141140 __perf_event__output_id_sample ([kernel.kallsyms])
a.out 24 3374818.881012: cycles: ffffffff817319fd _raw_spin_lock_irqsave ([kernel.kallsyms])
a.out 24 3374818.882217: cycles: ffffffff8109aba3 ttwu_do_activate.constprop.75 ([kernel.kallsyms])
a.out 24 3374818.884071: cycles: 40053d [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
a.out 24 3374818.885329: cycles: 400544 [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
...
Note the /var/lib/docker/aufs path. That's from the host so it won't exist in the container and you need to help perf report to locate it. This likely happens because the mmap events are tracked by perf outside of any cgroup and perf does not attempt to remap the paths.
Another option is to run perf host-side, like sudo perf record -a docker run -ti <container name>. But the collection has to be system-wide here (the -a flag) as containers are spawned by docker daemon process which is not in the process hierarchy of the docker client tool we run here.
Another way that doesn't require changing how you run the container (so you can profile an already running process) is to mount container's root on host using bindfs:
bindfs /proc/$(docker inspect --format {{.State.Pid}} $CONTAINER_ID)/root /foo
Then run perf report as perf report --symfs /foo
You'll have to run perf record system wide, but you can restrict it to only collect events for the specific container:
perf record -g -a -F 100 -e cpu-clock -G docker/$(docker inspect --format {{.Id}} $CONTAINER_ID) sleep 90
This question already has an answer here:
Error during Cross-compiling C code with Dynamic libraries
(1 answer)
Closed 6 years ago.
I am using VAR-SOM-AM33 board and want to run sample code like hello world run on device and it gives -sh:./test:not found error
Toolchain used for compile code is gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux.
Code is as below
#include <stdio.h>
int main(){
printf("hello world");
return 0;
}
for crosscompile file
arm-linux-gnueabihf-gcc test.c -march=armv7-a -marm -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a8 -o test
output binary is shows as follows
test: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.31, BuildID[sha1]=2ce1c5b3d97dac2093fe2cd2d340cdaa9989923f, not stripped
After copy that file into hardware and run it shows following error
root#am335x-evm:~# ./test
-sh: ./test: not found
File permission also change by
root#am335x-evm:~# chmod +x test
but result shows same not found error.
Demo file which is running on hardware,its architecture as follows
root#am335x-evm:~# readelf -A /usr/bin/hello
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_VFP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align8_needed: Yes
Tag_ABI_align8_preserved: Yes, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: SP and DP
and file which is cross compiled,its architecture as follows
root#am335x-evm:~# readelf -A test
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_VFP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align8_needed: Yes
Tag_ABI_align8_preserved: Yes, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: SP and DP
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6
Tag_unknown_44: 1 (0x1)
Also hardware cpuinfo is as follows
root#am335x-evm:~# cat /proc/cpuinfo
Processor : ARMv7 Processor rev 2 (v7l)
BogoMIPS : 718.02
Features : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part : 0xc08
CPU revision : 2
Hardware : Variscite VAR-SOM-AM33
Revision : 0000
Serial : 0000000000000000
I have tried running ldd command on target device.
root#am335x-evm:~# ldd
-sh: ldd: not found
So I suspect that the issue is related to the linker.
If I simply compile the file, without linking it.
arm-linux-gnueabihf-gcc -mtune=cortex-a8 -march=armv7 -O -c test.c -o test
Now if I run this file I get this error.
root#am335x-evm:~# chmod +x test
root#am335x-evm:~# ./test
./test: line 1: syntax error: word unexpected (expecting ")")
please suggest how to resolve this.
Thanks for solution, I found error with linker.
existing file has linker ld-linux.so.3 and crosscompiled file has linker ld-linux-armhf.so.3
root#am335x-evm:/usr/bin# readelf -l hello
Elf file type is EXEC (Executable file)
Entry point 0x82fc
There are 8 program headers, starting at offset 52
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
EXIDX 0x00044c 0x0000844c 0x0000844c 0x00008 0x00008 R 0x4
PHDR 0x000034 0x00008034 0x00008034 0x00100 0x00100 R E 0x4
INTERP 0x000134 0x00008134 0x00008134 0x00013 0x00013 R 0x1
[Requesting program interpreter: /lib/ld-linux.so.3]
crosscompiled file program header
root#am335x-evm:~# readelf -l test
Elf file type is EXEC (Executable file)
Entry point 0x82f9
There are 8 program headers, starting at offset 52
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
EXIDX 0x000450 0x00008450 0x00008450 0x00008 0x00008 R 0x4
PHDR 0x000034 0x00008034 0x00008034 0x00100 0x00100 R E 0x4
INTERP 0x000134 0x00008134 0x00008134 0x00019 0x00019 R 0x1
[Requesting program interpreter: /lib/ld-linux-armhf.so.3]
after chang that linker program runs on target device.
root#am335x-evm:~# cd /lib/
root#am335x-evm:/lib# ls -l ld-linux.so.3
lrwxrwxrwx 1 1000 1000 12 Aug 7 2012 ld-linux.so.3 -> ld-2.12.2.so
root#am335x-evm:/lib# ln -s /li ld-2.12.2.so ld-linux-armhf.so.3
/lib/ /linuxrc
root#am335x-evm:/lib# ln -s /lib/ld-2.12.2.so ld-linux-armhf.so.3
root#am335x-evm:/lib# ldconfig
root#am335x-evm:/lib# cd
root#am335x-evm:~# ./test
hello worldroot#am335x-evm:~#
on FreeBSD I started to play around with LLDB, but it crashes right at the start.
user#host ~/sandbox % rake hello
cc -I/usr/local/include -g -O0 -o hello.o -c hello.c
cc -Wl,-L/usr/local/lib -o hello hello.o
user#host ~/sandbox % lldb
(lldb) target create hello
Current executable set to 'hello' (i386).
(lldb) source list
8 {
9 printf( "Hello, world!\n");
10 return 0;
11 }
12
(lldb) breakpoint set -f hello.c -l 9
Breakpoint 1: where = hello`main + 31 at hello.c:9, address = 0x080485af
(lldb) process launch
Process 2409 launching
Process 2409 stopped
(lldb) Process 2409 launched: '/usr/home/user/sandbox/hello' (i386)
Process 2409 stopped
* thread #1: tid = 100224, 0x0818188f, stop reason = hardware error
frame #0: 0x0818188f
-> 0x818188f: addb %al, (%eax)
0x8181891: addb %al, (%eax)
0x8181893: addb %al, (%eax)
0x8181895: addb %al, (%eax)
(lldb)
It is the same on three machines.
I have also tried Gdb on Linux. There, everything worked fine.
What did I do wrong?
Thanks in advance,
Bertram
LLDB doesn't support FreeBSD/i386 host for now. Use recent gdb from ports or switch to amd64.
I'm trying to learn about ARM assembler programming using the GNU assembler. I've setup my PC with QEmu and have a Debian ARM-HF chroot environment.
If I assemble and link my test program:
.text
.global _start
_start:
mov r0, #6
bx lr
with:
as test.s -o test.o
ld test.o -o test
Then load the file into gdb and set a breakpoint on _start:
root#Latitude-E6420:/root# gdb test
GNU gdb (GDB) 7.6.1 (Debian 7.6.1-1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
...
Reading symbols from /root/test...(no debugging symbols found)...done.
(gdb) break _start
Breakpoint 1 at 0x8054
(gdb)
How do I single step the code, display the assembler source code and monitor the registers?
I tried some basic commands and they did not work:
(gdb) break _start
Breakpoint 1 at 0x8054
(gdb) info regi
The program has no registers now.
(gdb) stepi
The program is not being run.
(gdb) disas
No frame selected.
(gdb) r
Starting program: /root/test
qemu: Unsupported syscall: 26
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
qemu: Unsupported syscall: 26
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb)
Your problem here is that you're trying to run an ARM gdb under QEMU's user-mode emulation. QEMU doesn't support the ptrace syscall (that's what syscall number 26 is), so this is never going to work.
What you need to do is run your test binary under QEMU with the QEMU options to enable QEMU's own builtin gdb stub which will listen on a TCP port. Then you can run a gdb compiled to run on your host system but with support for ARM targets, and tell that to connect to the TCP port.
(Emulating ptrace within QEMU is technically very tricky, and it would not provide much extra functionality that you can't already achieve via the QEMU builtin gdbstub. It's very unlikely it'll ever be implemented.)
Minimal working QEMU user mode example
I was missing the -fno-pie -no-pie options:
sudo apt-get install gdb-multiarch gcc-arm-linux-gnueabihf qemu-user
printf '
#include <stdio.h>
#include <stdlib.h>
int main() {
puts("hello world");
return EXIT_SUCCESS;
}
' > hello_world.c
arm-linux-gnueabihf-gcc -fno-pie -ggdb3 -no-pie -o hello_world hello_world.c
qemu-arm -L /usr/arm-linux-gnueabihf -g 1234 ./hello_world
On another terminal:
gdb-multiarch -q --nh \
-ex 'set architecture arm' \
-ex 'set sysroot /usr/arm-linux-gnueabihf' \
-ex 'file hello_world' \
-ex 'target remote localhost:1234' \
-ex 'break main' \
-ex continue \
-ex 'layout split'
;
This leaves us at main, in a split code / disassembly view due to layout split. You will also interested in:
layout regs
which shows the registers.
At the end of the day however, GDB Dashboard is more flexible and reliable: gdb split view with code
-fno-pie -no-pie is required because the packaged Ubuntu GCC uses -fpie -pie by default, and those fail due to a QEMU bug: How to GDB step debug a dynamically linked executable in QEMU user mode?
There was no gdbserver --multi-like functionality for the QEMU GDB stub on QEMU 2.11: How to restart QEMU user mode programs from the GDB stub as in gdbserver --multi?
For those learning ARM assembly, I am starting some runnable examples with assertions and using the C standard library for IO at: https://github.com/cirosantilli/arm-assembly-cheat
Tested on Ubuntu 18.04, gdb-multiarch 8.1, gcc-arm-linux-gnueabihf 7.3.0, qemu-user 2.11.
Freestanding QEMU user mode example
This analogous procedure also works on an ARM freestanding (no standard library) example:
printf '
.data
msg:
.ascii "hello world\\n"
len = . - msg
.text
.global _start
_start:
/* write syscall */
mov r0, #1 /* stdout */
ldr r1, =msg /* buffer */
ldr r2, =len /* len */
mov r7, #4 /* Syscall ID. */
swi #0
/* exit syscall */
mov r0, #0 /* Status. */
mov r7, #1 /* Syscall ID. */
swi #0
' > hello_world.S
arm-linux-gnueabihf-gcc -ggdb3 -nostdlib -o hello_world -static hello_world.S
qemu-arm -g 1234 ./hello_world
On another terminal:
gdb-multiarch -q --nh \
-ex 'set architecture arm' \
-ex 'file hello_world' \
-ex 'target remote localhost:1234' \
-ex 'layout split' \
;
We are now left at the first instruction of the program.
QEMU full system examples
Linux kernel: How to debug the Linux kernel with GDB and QEMU?
Bare metal: https://github.com/cirosantilli/newlib-examples/tree/f70f8a33f8b727422bd6f0b2975c4455d0b33efa#gdb
Single step of an assembly instruction is done with stepi. disas will disassemble around the current PC. info regi will display the current register state. There are some examples for various processors on my blog for my ELLCC cross development tool chain project.
You should add the -g option too to the assembling. Otherwise the codeline info is not included.
That crash probably comes from running some garbage after the code lines.
Maybe you should add the exit system call:
mov eax, 1 ; exit
mov ebx, 0 ; returm value
int 0x80 ; system call