I am using QEMU, buildroot-2018.02.6 and Linux kernel (4.18.14), compiled all these on Ubuntu for aarch64, using gdb-multiarch to debug, but not hitting the break point in kernel code; when I set break point, I get below message:
warning: Breakpoint address adjusted from 0xffff000008f90990 to 0xff000008f90990.
Breakpoint 1 at 0xff000008f90990: file init/main.c, line 532.
Never hit the break point, tried with both hbreak and break. Please help.
Which gdb version are you using? There was a bug in gdb in this area in gdb 8.1: https://sourceware.org/bugzilla/show_bug.cgi?id=23127 -- it has been fixed in newer gdb version I think, and it wasn't present in older ones like gdb 7.
Related
I am newbie in Linux kernel and I'm trying to debug it with gdb via QEMU. My problem is gdb doesn't stop after break start_kernel. More details is below.
My host system is ArchLinux 5.0.10-arch1-1-ARCH x86_64.
Qemu is QEMU emulator version 4.0.0
gdb is GNU gdb (GDB) 8.2.1.
Debugged kernel is linux-4.20.12
I have done following steps:
Compile kernel with
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
CONFIG_GDB_SCRIPTS=y
# CONFIG_DEBUG_INFO_REDUCED is not set
Try to run LFS-8.4 system with it kernel via QEMU
qemu-system-x86_64 lfs-8.4-08052019.raw
System starts good.
Run this system with stop CPU and gdbserver options via QEMU
qemu-system-x86_64 lfs-8.4-08052019.raw -S -s
Start gdb
$ gdb
Load symbols
(gdb) file /mnt/lfs/sources/linux-4.20.12/vmlinux
Reading symbols from /mnt/lfs/sources/linux-4.20.12/vmlinux...done.
Connect to QEMU
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x000000000000fff0 in cpu_hw_events ()
Set break point
(gdb) break start_kernel
Breakpoint 1 at 0xffffffff82761ab5: file init/main.c, line 538.
(I have tried hardware break point hbreak, but result is the same)
Continue
(gdb) c
Continuing.
Further system loads login prompt without any breaks and gdb doesn't show any new messages.
What I do wrong?
UPD: starting QEMU with embed -kernel facility give the same result
qemu-system-x86_64 -kernel /mnt/lfs/sources/linux-4.20.12/arch/x86/boot/bzImage -append 'root=/dev/sda3' -drive file=lfs-8.4-08052019.raw -S -s
UPD2: I have tried to start QEMU without -S key and run target remote localhost:1234 in gdb immediately. QEMU have stopped loading at
Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting kernel.
When I type s in gdb it says
(gdb) s
Cannot find bounds of current function
I suspect that debug symbols in vmlinux isn't related to bzImage. Maybe they was made wrong, but I don't how to make them another way.
UPD3: I have built kernel 2.6 in LFS-8.4 chroot environment. System isn't loaded, but kernel is successfully debugged by method described above! So, I think that it is problem of new kernels. Maybe I should switch off/on something in my 4.20.12 kernel, but I don't know what is exactly. For my purpose (reading Robert Love "Linux Kernel Development") kernel version 2.6 is enough.
You need to pass nokaslr to kernel cmdline.
qemu-system-x86_64 -kernel /mnt/lfs/sources/linux-4.20.12/arch/x86/boot/bzImage -append 'root=/dev/sda3 nokaslr' -drive file=lfs-8.4-08052019.raw -S -s
I have setup Linux Kernel debug environment with VMware Workstation. But When I tried to connect with gdb that connects correctly but I can't set any breakpoint or examine any kernel symbol.
Target Machine (debugee) Ubuntu 18:
I have compiled linux kernel 5.0-0 with the following directives:
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_FS=y
# CONFIG_DEBUG_SECTION_MISMATCH is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
Also my VMX file configuration:
debugStub.listen.guest64 = "TRUE"
debugStub.listen.guest64.remote="TRUE"
After that I transfered vmlinux to debugger machine and use gdb:
bash$ gdb vmlinux
gdb-peda$ target remote 10.251.31.28:8864
Remote debugging using 10.251.31.28:8864
Warning: not running or target is remote
0xffffffff9c623f36 in ?? ()
gdb-peda$ disas sys_open
No symbol "do_sys_open" in current context.
First you need to install kernel-debug-devel, kernel-debuginfo, kernel-debuginfo-common for corresponding kernel version.
Then you can use crash utility to debug kernel, which internally uses gdb
The symbol name you're looking for is sometimes not exactly what you expect it to be. You can use readelf or other similar tools to find the full name of the symbol in the kernel image. These names sometimes differ from the names in the code because of various architecture level differences and their related header and C definitions in kernel code. For example you might be able to disassemble the open() system call by using:
disas __x64_do_sys_open
if you've compiled it for x86-64 architecture.
Also keep in mind that these naming conventions are subject to change in different versions of kernel.
I'm using Qt Creator 3.2.1 and gdb 7.6.2 (powerpc64-fsl-linux-gdb from QorIQ SDK 1.8) for debugging cross compiled applications.
On my ppc64 test machine (Yocto) I run gdbserver: gdbserver :666 /path/to/myapp
Then I connect using Qt Creator on my x86_64 dev machine (Debian) to gdbserver. I've specified Kit (sysroot is /opt/fsl-networking/QorIQ-SDK-V1.8/sysroots/ppc64e6500-fsl-linux), the toolchain gdb, etc... This works so far. I can debug any application.
But if my application links dynamically to a library and I want to debug this library (/path/to/libmylib.so), gdb within Qt Creator fails to step into the breakpoints of mylib.
What I want to do with Qt Creator is like this:
Yocto: start the gdbserver
Debian: run powerpc64-fsl-linux-gdb /path/to/myapp and enter
set solib-search-path /path/to/mylib/
target remote 192.168.120.211:666
break MyLibraryFunctionIWantToDebug
run
c
Log:
GNU gdb (GDB) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-fslsdk-linux --target=powerpc64-fsl-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /path/to/myapp...done.
(gdb) show solib-search-path
The search path for loading non-absolute shared library symbol files is .
(gdb) set solib-search-path /path/to/mylib/
(gdb) show solib-search-pat
The search path for loading non-absolute shared library symbol files is /path/to/mylib/.
(gdb) target remote 192.168.120.211:666
Remote debugging using 192.168.120.211:666
(gdb) break MyLibraryFunctionIWantToDebug
Breakpoint 1 at 0x10001088: file /path/to/myapp/myapp.c, line 196.
(gdb) run
The "remote" target does not support "run". Try "help target" or "continue".
(gdb) c
...
Research so far:
gdb has a bug in version 7.4 (I'm using 7.6.2 right now)
I can put the set solib-search-path /path/to/mylib/ command into .gdbinit. In Qt Creator every .gdbinit file is ignored. I've tried multiple locations, e.g. ~/.gdbinit, /path/to/myapp/.gdbinit, command line arguments option in Qt creator: -x ~/.gdbinit, Tools > Options > Debuggers putting the command into every additional field. None works :-(
I'm not alone with my question, see here or here
Any ideas how to cross debug with gdb in Qt Creator when using shared libraries?
Maybe it's changed with QtCreator 3.6 (what I'm using), but if I put the set solib-search-path command in the "Additional Attach Commands" fiend and only that field, I got it to work.
Granted, my case may not be fully applicable to yours as it's not a full cross-compilation, but a case of running GDB on a different version of Linux than my remote target runs.
I am working on academic project that modifies some Kernel Networking code as well as include a new Kernel module.
I am using QEMU to load modified kernel and test.
However, i find that a complete OS is required in some .img to debug.
Is it possible without it ?
Or, which is the distro that can be used with Kernel 2.6 for system. The distro need not have any features, except ability to run programs, including networking support.
The easiest way in my opinion is to use buildroot
http://buildroot.uclibc.org/
clone it, configure it to use your custom kernel (default userspace is fine for a start, you might want to change it later).
it will build your kernel and root filesystem. the entire process takes about half an hour, twenty minutes of which is compiling the monster
my run line looks something:
qemu-system-i386
-hda rootfs.ext2
-kernel bzImage
-m 512M
-append "root=/dev/sda console=ttyS0"
-localtime
-serial stdio
and some more options regarding a tap device
Minimal fully automated QEMU + GDB + Buildroot example
QEMU + GDB on non-module Linux kernel is covered in detail at: How to debug the Linux kernel with GDB and QEMU? and building the kernel modules inside QEMU at: How to add Linux driver as a Buildroot package Get those working first.
Next, I have also fully automated GDB module debugging at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1c29163c3919d4168d5d34852d804fd3eeb3ba67#kernel-module-debugging
These are the main steps you have to take:
Compile the kernel module with debug symbols:
ccflags-y += -g -DDEBUG
as mentioned at: kernel module no debugging symbols found
Stop GDB with Ctrl + C and run:
lx-symbols path/to/parent/of/modules/
This amazing command, which is defined in a GDB Python script inside the Linux kernel source tree, automatically loads symbols for loaded modules present under the given directory recursively whenever GDB stops.
The best way to make that command available is to use:
gdb -ex add-auto-load-safe-path /full/path/to/linux/kernel
as explained at: GDB: lx-symbols undefined command
insmod the kernel module.
This must be done before setting breakpoints, because we don't know where the kernel will insert the module in memory beforehand.
lx-symbols automatically takes care of finding the module location (in host filesystem and guest memory!) for us.
Break GDB again with Ctrl + C, set breakpoints, and enjoy.
If were feeling hardcore, you could also drop lx-symbols entirely, and find the module location after insmod with:
cat /proc/modules
and then add the .ko manually with:
add-symbol-file path/to/mymodule.ko 0xfffffffa00000000
I' ve a cross-gdb configured with --host=i686-pc-linux-gnu --target=powerpc-e300c3-linux-gnu. i can correctly debug an application on a remote board with gdbserver but i get an error about a version mismatch between libthread_db and libpthread so i can't debug threads correctly (gdb recognizes only one thread instead of three threads). Maybe it's is due to a different version of libc: on host machine i've libc2.15 and on target machine lib2.5. I tried to rebuid libc2.5 for host in order to link gdb against to it but it's an hell. Before i get crazy to rebuild it, could someone confirm that it's a libc problem?
I tried to rebuid libc2.5 for host in order to link gdb against to it
That's not what you need.
What you need, is for gdb to find and load libthread_db.so.1, that matches your target libpthread.so.0.
For this, you need to
build libc-2.5 for host, and
set GDB's libthread-db-search-path such that it finds the libthread_db.so.1 built in step 1.
You don't actually need to build the entire libc in step 1. Something like this should suffice:
mkdir build && cd build
../configure --prefix=/usr
make -C ../nptl_db objdir=`pwd`
Update:
i have GDB 6.6 and there isn't libthread-db-search-path. What is another way to specify that path?
That GDB will just dlopen("libthread_db.so.1", ...). So to make it find the right libthread_db.so.1, you need to adjust LD_LIBRARY_PATH. Using bash:
LD_LIBRARY_PATH=/tmp/glibc-2.5/build/nptl_db gdb /path/to/target/a.out