I have setup Linux Kernel debug environment with VMware Workstation. But When I tried to connect with gdb that connects correctly but I can't set any breakpoint or examine any kernel symbol.
Target Machine (debugee) Ubuntu 18:
I have compiled linux kernel 5.0-0 with the following directives:
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_FS=y
# CONFIG_DEBUG_SECTION_MISMATCH is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
Also my VMX file configuration:
debugStub.listen.guest64 = "TRUE"
debugStub.listen.guest64.remote="TRUE"
After that I transfered vmlinux to debugger machine and use gdb:
bash$ gdb vmlinux
gdb-peda$ target remote 10.251.31.28:8864
Remote debugging using 10.251.31.28:8864
Warning: not running or target is remote
0xffffffff9c623f36 in ?? ()
gdb-peda$ disas sys_open
No symbol "do_sys_open" in current context.
First you need to install kernel-debug-devel, kernel-debuginfo, kernel-debuginfo-common for corresponding kernel version.
Then you can use crash utility to debug kernel, which internally uses gdb
The symbol name you're looking for is sometimes not exactly what you expect it to be. You can use readelf or other similar tools to find the full name of the symbol in the kernel image. These names sometimes differ from the names in the code because of various architecture level differences and their related header and C definitions in kernel code. For example you might be able to disassemble the open() system call by using:
disas __x64_do_sys_open
if you've compiled it for x86-64 architecture.
Also keep in mind that these naming conventions are subject to change in different versions of kernel.
Related
I am working on academic project that modifies some Kernel Networking code as well as include a new Kernel module.
I am using QEMU to load modified kernel and test.
However, i find that a complete OS is required in some .img to debug.
Is it possible without it ?
Or, which is the distro that can be used with Kernel 2.6 for system. The distro need not have any features, except ability to run programs, including networking support.
The easiest way in my opinion is to use buildroot
http://buildroot.uclibc.org/
clone it, configure it to use your custom kernel (default userspace is fine for a start, you might want to change it later).
it will build your kernel and root filesystem. the entire process takes about half an hour, twenty minutes of which is compiling the monster
my run line looks something:
qemu-system-i386
-hda rootfs.ext2
-kernel bzImage
-m 512M
-append "root=/dev/sda console=ttyS0"
-localtime
-serial stdio
and some more options regarding a tap device
Minimal fully automated QEMU + GDB + Buildroot example
QEMU + GDB on non-module Linux kernel is covered in detail at: How to debug the Linux kernel with GDB and QEMU? and building the kernel modules inside QEMU at: How to add Linux driver as a Buildroot package Get those working first.
Next, I have also fully automated GDB module debugging at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1c29163c3919d4168d5d34852d804fd3eeb3ba67#kernel-module-debugging
These are the main steps you have to take:
Compile the kernel module with debug symbols:
ccflags-y += -g -DDEBUG
as mentioned at: kernel module no debugging symbols found
Stop GDB with Ctrl + C and run:
lx-symbols path/to/parent/of/modules/
This amazing command, which is defined in a GDB Python script inside the Linux kernel source tree, automatically loads symbols for loaded modules present under the given directory recursively whenever GDB stops.
The best way to make that command available is to use:
gdb -ex add-auto-load-safe-path /full/path/to/linux/kernel
as explained at: GDB: lx-symbols undefined command
insmod the kernel module.
This must be done before setting breakpoints, because we don't know where the kernel will insert the module in memory beforehand.
lx-symbols automatically takes care of finding the module location (in host filesystem and guest memory!) for us.
Break GDB again with Ctrl + C, set breakpoints, and enjoy.
If were feeling hardcore, you could also drop lx-symbols entirely, and find the module location after insmod with:
cat /proc/modules
and then add the .ko manually with:
add-symbol-file path/to/mymodule.ko 0xfffffffa00000000
I have a application core file that was generated on an x86 target machine. All the libraries and binaries are stripped and so the back trace when running on the target is very minimal to nothing. i still do see the symbol name and back trace.
However, if i move the core file to a host machine and run the gdb for x86 on the host, provide it with sysroot , it complains with the following.
cannot load memory at address 0x104
Any ideas/suggestions why the host complains while the target is ok decoding the back trace?
BTW, all my code has been compiled with -fPIC.
i do the following: gdb-7.2 path_to_biniary , then set sysroot
Try this instead:
gdb /path/to/binary
(gdb) set sysroot /path/to/buildroot
(gdb) core /path/to/core_file
Setting sysroot after the core has already been loaded (as I believe happens in your invocation) is too late.
I want to debug pthreads on my custom linux distribution but I am missing something. My host is Ubuntu 12.04, my target is an i486 custom embedded Linux built with a crosstool-NG cross compiler toolset, the rest of the OS is made with Buildroot.
I will list the facts:
I can run multi-threaded applications on my target
Google Breakpad fails to create a crash report when I run a multi-threaded application on the target. The exact same application with the exact same build of Breakpad libraries will succeed when I run it on my host.
GDB fails to debug multithreaded applications on my target.
e.g.
$./gdb -n -ex "thread apply all backtrace" ./a.out --pid 716
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.
GNU gdb 6.8
I don't think ps_lgetfpregs is a problem because of this.
My crosstool build created the libthread_db.so file, and I put it on the target.
My crosstool build created the gdb for my target, so it should have been linked against the same libraries that I run on the target.
If I run gdb on my host, against my test app, I get a backtrace of each running thread.
I suspect the problem with Breakpad is related to the problem with GDB, but I cannot substantiate this. The only commonality is lack of multithreaded debug.
There is some crucial difference between my host and target that stops me from being able to debug pthreads on the target.
Does anyone know what it is?
EDIT:
Denys Dmytriyenko from TI says:
Normally, GDB is not very picky and you can mix and match different
versions of gdb and gdbserver. But, unfortunately, if you need to
debug multi-threaded apps, there are some dependencies for specific
APIs...
For example, this is one of the messages you may see if you didn't
build GDB properly for the thread support:
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1:
undefined symbol: ps_lgetfpregs GDB will not be able to debug
pthreads.
Note that this error is the same as the one that I get but he doesn't go in to detail about how to build GDB "properly".
and the GDB FAQ says:
(Q) GDB does not see any threads besides the one in which crash occurred;
or SIGTRAP kills my program when I set a breakpoint.
(A) This frequently
happen on Linux, especially on embedded targets. There are two common
causes:
you are using glibc, and you have stripped libpthread.so.0
mismatch between libpthread.so.0 and libthread_db.so.1
GDB itself does
not know how to decode "thread control blocks" maintained by glibc and
considered to be glibc private implementation detail. It uses
libthread_db.so.1 (part of glibc) to help it do so. Therefore,
libthread_db.so.1 and libpthread.so.0 must match in version and
compilation flags. In addition, libthread_db.so.1 requires certain
non-global symbols to be present in libpthread.so.0.
Solution: use
strip --strip-debug libpthread.so.0 instead of strip libpthread.so.0.
I tried a non-stripped libpthread.so.0 but it didn't make a difference. I will investigate any mismatch between pthread and thread_db.
This:
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.
means that the libthread_db.so.1 library was not able to find the symbol ps_lgetfpregs in gdb.
Why?
Because I built gdb using Crosstoolg-NG with the "Build a static native gdb" option and this adds the -static option to gcc.
The native gdb is built with the -rdynamic option and this populates the .dynsym symbol table in the ELF file with all symbols, even unused ones. libread_db uses this symbol table to find ps_lgetfpregs from gdb.
But -static strips the .dynsym table from the ELF file.
At this point there are two options:
Don't build a static native gdb if you want to debug threads.
Build a static gdb and a static libthread_db (not tested)
Edit:
By the way, this does not explain why Breakpad in unable to debug multithreaded applications on my target.
Just a though... To use the gdb debugger, you need to compile your code with -g option. For instance, gcc -g -c *.c.
I am using gdbserver on target and CodeSourcery IDE. My hardware is a gumstix with a omap3530.
I can step through code in my main application but if I attempt to step into a function in a shared library I get memory address and a debugger terminates.
This is my library that is compiled and copied to the /lib folder on the target system.(it does have debug symbols) I have attempted to use the .gbdinit file to set solib-absolute-prefix /lib
Here are the warnings from the gdb trace:
903,056 13-gdb-set sysroot-on-target /lib
903,065 13^done
903,065 (gdb)
903,065 14-target-select remote 192.168.1.101:2345
903,114 =thread-group-started,id="i1",pid="42000"
903,114 =thread-created,id="1",group-id="i1"
903,115 15-list-thread-groups --available
903,120 16-list-thread-groups
903,128 &"warning: Unable to find dynamic linker breakpoint function.\nGDB will be unable to debug shared library initializers\nand track explicitly loaded dynamic code."
903,128 &"\n"
Which leads to
903,395 &"Error while mapping shared library sections:\n"
903,397 &"/lib/libCoreLib.so: Invalid argument.\n"
903,399 =library-loaded,id="/lib/libCoreLib.so",target-name="/lib/libCoreLib.so",hostname="/lib/libCoreLib.so",low-address="0x0",high-address="0x0",symbols-loaded="0",thread-group="i1"
You can debug with the library installed on your host, provided the debugging machine is also the development machine. In that case, you use set sysroot instead of set sysroot-on-target. For example :
set sysroot /home/username/.../rootfs/
where /home/username/.../rootfs/ contains a copy of your target filesystem
I think you should also specify / instead of /lib
Target with debug symbols
This is the simplest method to get working, and it is specially useful when you are developing one particular shared library.
First copy the test executable and shared library to the target with debug information:
check with readelf ----debug-dump=decodedline libmyib.so: How can I tell if a library was compiled with -g?
I recommend using an NFS server on host, so that the compiled output gets automatically uploaded after compilation
Then on target:
gdbserver --multi :1234 ./executable_name
Host:
arm-linux-gnueabihf-gdb -q -nh \
-ex "target extended-remote target-hostname-or-ip:1234" \
-ex "file ./executable_name" \
-ex 'tb main' \
-ex 'c' \
-ex 'set solib-search-path .'
sharedlibrary libmylib.so also works.
The problem I had was that gdbserver stops at the dynamic loader, before main, and the dynamic libraries are not yet loaded at that point, and so GDB does not know where the symbols will go in memory yet.
GDB appears to have some mechanisms to automatically load shared library symbols, and if I compile for host, and run gdbserver locally, running to main is not needed. But on the ARM target, that is the most reliable thing to do.
Target gdbserver 7.12-6, host arm-linux-gnueabihf-gdb 7.6.1 from Linaro.
Target libraries without debug symbols
It is common to strip target libraries before deployment on embedded targets, since debug information makes them way larger.
For example, Buildroot does that by default, but you can disable it with BR2_STRIP_none=y.
You can identify this scenario by running:
info shared
Which shows something like:
From To Syms Read Shared Object Library
0x00007ffff7df7f90 0x00007ffff7dfcdd7 Yes (*) target:/lib/ld64-uClibc.so.0
0x00007ffff7b3a9b0 0x00007ffff7bbe05d Yes (*) target:/lib/libc.so.0
(*): Shared library is missing debugging information.
so there are asterisks (*) for both of the libraries which says that debug information is missing.
If that is the case, then you have to tell GDB to use the shared libraries on host before they were stripped.
Buildroot for example makes that easy for us, as it maintains the staging directory which contains the shared libraries before they were stripped and in the same relative paths as in the target:
set sysroot buildroot/output/staging/
When this option is set, gdb immediately searches for libraries in the host instead of target, and finds /lib/libc.so.0 at the path buildroot/output/staging/ + /lib/libc.so.0:
Reading symbols from buildroot/output/staging/lib/ld64-uClibc.so.0...done.
Reading symbols from buildroot/output/staging/lib/libc.so.0...done.
TODO: I don't think you can set more than one sysroot, so all your shared libraries must be placed in their correct relative paths as in the target image.
If you check the bad default sysroot, you will see:
show sysroot
give:
target:
which means that gdb searches for shared libraries on the target root / by default.
Similar issue was encountered while debugging. The debug was hanging up. Configuration is as follows
Host: Ubuntu 12.04LTS
IDE: Eclipse Kepler
Target: Beaglebone Black / ARM A8
OS: Angstrom
Solution
Update libraries and includes
Select properties for project in Eclipse
C/C++ General > Paths and Symbols > (include TAB) GNU C > Add > Files
systems > / > usr Change from /usr/lib/gcc/i686-linux-gnu/4/6/include
to /usr/arm-linux-gnueabi/include
C/C++ General > Paths and Symbols > (Include TAB) GNU C++> Add >
Files systems > / > usr /usr/arm-linux-gnueabi/include/c++/4.6.3/arm-linux-gnueabi
C/C++ General > Paths and Symbols > (Library Paths TAB) > Add > Files
systems > / > usr /usr/arm-linux-gnueabi/lib
Good day,
If the 'debug-file-directory' variable in GDB is set incorrectly,
then the reported error messages contains:
warning: Unable to find dynamic linker breakpoint function.
The root filesystem of the target is located on my host PC at
/opt/arm-linux-gnueabihf-rootfs
The following two commands helped me to get remote debugging working
via gdbserver using GDB (v7.11.1):
set debug-file-directory /opt/arm-linux-gnueabihf-rootfs/usr/lib/debug
set sysroot /opt/arm-linux-gnueabihf-rootfs
I've noticed that if 'sysroot' has a trailing slash in the path,
then GDB fails to use it.You will see this (incorrect output) after connecting to the remote target:
Reading /lib/ld-linux-armhf.so.3 from remote target...
or
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/lib/ld-linux-
armhf.so.3...(no debugging symbols found)...done
instead of the correct output:
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/lib/ld-linux-
armhf.so.3...
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/usr/lib/debug/
lib/arm-linux-gnueabihf/ld-2.23.so...done.
Regards,
Frikkie Thirion
After a Xen guest domain hang, I took a dump using xm core-dump . Following the sparse documentation I found, I tried using the crash utility to analyze the dump.
Unfortunately, the kernel image (Debian lenny) is stripped, so I am forced to make use of the map file.
However,
crash
/boot/System.map-2.6.26-2-xen-amd64
vmlinux-2.6.26-2-xen-amd64
/mnt/my-core-file
(with vmlinux-2.6.26-2-xen-amd64 being the gunzip'ed vmlinuz image) fails:
crash: vmlinux-2.6.26-2-xen-amd64: no
debugging data available
Then I read that current Xen versions produce ELF-compatible dumps for guest domains. Indeed, this seems to be the case:
~$ sudo file my-core-dump
my-core-dump: ELF 64-bit LSB core file x86-64, version 1
However, gdb vmlinux-2.6.26-2-xen-amd64 my-core-dump fails, too:
...is not a core dump: File format not
recognized
Any hints?
Have you tried attaching to the domU console ?
xm create domU.conf -c
On the subject of the core-dump file, I found this:
http://lists.xensource.com/archives/html/xen-devel/2006-12/msg00456.html
I just want to check that you aren't under the impression that 'xm
dump-core' emits an Elf core file. It doesn't -- the format is custom and as
far as I know is only interpreted by a set of gdbserver patches that we ship
in our repository. Does the crash utility really support this special
format?
Edit: This might help to debug the core-dump: http://os-drive.com/files/docbook/xen-faq.html#setup_gdb