I have a application core file that was generated on an x86 target machine. All the libraries and binaries are stripped and so the back trace when running on the target is very minimal to nothing. i still do see the symbol name and back trace.
However, if i move the core file to a host machine and run the gdb for x86 on the host, provide it with sysroot , it complains with the following.
cannot load memory at address 0x104
Any ideas/suggestions why the host complains while the target is ok decoding the back trace?
BTW, all my code has been compiled with -fPIC.
i do the following: gdb-7.2 path_to_biniary , then set sysroot
Try this instead:
gdb /path/to/binary
(gdb) set sysroot /path/to/buildroot
(gdb) core /path/to/core_file
Setting sysroot after the core has already been loaded (as I believe happens in your invocation) is too late.
Related
I have setup Linux Kernel debug environment with VMware Workstation. But When I tried to connect with gdb that connects correctly but I can't set any breakpoint or examine any kernel symbol.
Target Machine (debugee) Ubuntu 18:
I have compiled linux kernel 5.0-0 with the following directives:
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_FS=y
# CONFIG_DEBUG_SECTION_MISMATCH is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
Also my VMX file configuration:
debugStub.listen.guest64 = "TRUE"
debugStub.listen.guest64.remote="TRUE"
After that I transfered vmlinux to debugger machine and use gdb:
bash$ gdb vmlinux
gdb-peda$ target remote 10.251.31.28:8864
Remote debugging using 10.251.31.28:8864
Warning: not running or target is remote
0xffffffff9c623f36 in ?? ()
gdb-peda$ disas sys_open
No symbol "do_sys_open" in current context.
First you need to install kernel-debug-devel, kernel-debuginfo, kernel-debuginfo-common for corresponding kernel version.
Then you can use crash utility to debug kernel, which internally uses gdb
The symbol name you're looking for is sometimes not exactly what you expect it to be. You can use readelf or other similar tools to find the full name of the symbol in the kernel image. These names sometimes differ from the names in the code because of various architecture level differences and their related header and C definitions in kernel code. For example you might be able to disassemble the open() system call by using:
disas __x64_do_sys_open
if you've compiled it for x86-64 architecture.
Also keep in mind that these naming conventions are subject to change in different versions of kernel.
I am attempting to carry out postmortem analysis of a crashed binary, "TestApp", on a linux system.
I have a copy of the binaries and and shared objects that are copied onto the device in a path:
/usr/public/target
This folder contains all the binaries in question in the directory structure used on the system under test, ie:
/usr/public/target/sbin/TestApp
/usr/public/target/lib/TestAppLib.so
/usr/public/target/usr/lib/TestAppAPILib.so
The automated build process strips the debug information from the binaries, and stores them in external, symbol files, all under:
/usr/public/target_external_symbols
So the symbol information for the above binaries would exist in files named:
/usr/public/target_external_symbols/sbin/TestApp.sym
/usr/public/target_external_symbols/lib/TestAppLib.so.sym
/usr/public/target_external_symbols/usr/lib/TestAppAPILib.so.sym
How do I get GDB to be aware of the existence of these external symbols and to load them?
I typically invoke GDB via:
gdb TestApp TestApp.core
I've referred to other articles on creating a test .gdbinit file and passing it to GDB via the -command argument, but it doesn't appear to work. Every time I attempt to get a backtrace from my core file, I get an indication from GDB that it cannot open the debug symbols. Any help in resolving this is appreciated.
(gdb) info shared
From To Syms Read Shared Object Library
0x78000000 0x780061e8 Yes (*) /usr/public/target/lib/TestAppLib.so
0x78010000 0x7806e60c Yes (*) /usr/public/target/usr/lib/TestAppAPILib.so
0x78070000 0x78091d2c Yes (*) /usr/public/target/lib/libm.so.2
(*): Shared library is missing debugging information.
Thank you.
There are commands set debug-file-directory, symbol-file and add-symbol-file to load debugging symbols from within a gdb session. The latter one might require the address where the shared library was loaded into memory.
Maybe during your build process 'gnu debuglinks' have been added to your binaries. This would mean that in the executables there's a path coded that directs gdb where to look for debug symbols. More can be found here.
I want to debug pthreads on my custom linux distribution but I am missing something. My host is Ubuntu 12.04, my target is an i486 custom embedded Linux built with a crosstool-NG cross compiler toolset, the rest of the OS is made with Buildroot.
I will list the facts:
I can run multi-threaded applications on my target
Google Breakpad fails to create a crash report when I run a multi-threaded application on the target. The exact same application with the exact same build of Breakpad libraries will succeed when I run it on my host.
GDB fails to debug multithreaded applications on my target.
e.g.
$./gdb -n -ex "thread apply all backtrace" ./a.out --pid 716
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.
GNU gdb 6.8
I don't think ps_lgetfpregs is a problem because of this.
My crosstool build created the libthread_db.so file, and I put it on the target.
My crosstool build created the gdb for my target, so it should have been linked against the same libraries that I run on the target.
If I run gdb on my host, against my test app, I get a backtrace of each running thread.
I suspect the problem with Breakpad is related to the problem with GDB, but I cannot substantiate this. The only commonality is lack of multithreaded debug.
There is some crucial difference between my host and target that stops me from being able to debug pthreads on the target.
Does anyone know what it is?
EDIT:
Denys Dmytriyenko from TI says:
Normally, GDB is not very picky and you can mix and match different
versions of gdb and gdbserver. But, unfortunately, if you need to
debug multi-threaded apps, there are some dependencies for specific
APIs...
For example, this is one of the messages you may see if you didn't
build GDB properly for the thread support:
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1:
undefined symbol: ps_lgetfpregs GDB will not be able to debug
pthreads.
Note that this error is the same as the one that I get but he doesn't go in to detail about how to build GDB "properly".
and the GDB FAQ says:
(Q) GDB does not see any threads besides the one in which crash occurred;
or SIGTRAP kills my program when I set a breakpoint.
(A) This frequently
happen on Linux, especially on embedded targets. There are two common
causes:
you are using glibc, and you have stripped libpthread.so.0
mismatch between libpthread.so.0 and libthread_db.so.1
GDB itself does
not know how to decode "thread control blocks" maintained by glibc and
considered to be glibc private implementation detail. It uses
libthread_db.so.1 (part of glibc) to help it do so. Therefore,
libthread_db.so.1 and libpthread.so.0 must match in version and
compilation flags. In addition, libthread_db.so.1 requires certain
non-global symbols to be present in libpthread.so.0.
Solution: use
strip --strip-debug libpthread.so.0 instead of strip libpthread.so.0.
I tried a non-stripped libpthread.so.0 but it didn't make a difference. I will investigate any mismatch between pthread and thread_db.
This:
dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.
means that the libthread_db.so.1 library was not able to find the symbol ps_lgetfpregs in gdb.
Why?
Because I built gdb using Crosstoolg-NG with the "Build a static native gdb" option and this adds the -static option to gcc.
The native gdb is built with the -rdynamic option and this populates the .dynsym symbol table in the ELF file with all symbols, even unused ones. libread_db uses this symbol table to find ps_lgetfpregs from gdb.
But -static strips the .dynsym table from the ELF file.
At this point there are two options:
Don't build a static native gdb if you want to debug threads.
Build a static gdb and a static libthread_db (not tested)
Edit:
By the way, this does not explain why Breakpad in unable to debug multithreaded applications on my target.
Just a though... To use the gdb debugger, you need to compile your code with -g option. For instance, gcc -g -c *.c.
I' ve a cross-gdb configured with --host=i686-pc-linux-gnu --target=powerpc-e300c3-linux-gnu. i can correctly debug an application on a remote board with gdbserver but i get an error about a version mismatch between libthread_db and libpthread so i can't debug threads correctly (gdb recognizes only one thread instead of three threads). Maybe it's is due to a different version of libc: on host machine i've libc2.15 and on target machine lib2.5. I tried to rebuid libc2.5 for host in order to link gdb against to it but it's an hell. Before i get crazy to rebuild it, could someone confirm that it's a libc problem?
I tried to rebuid libc2.5 for host in order to link gdb against to it
That's not what you need.
What you need, is for gdb to find and load libthread_db.so.1, that matches your target libpthread.so.0.
For this, you need to
build libc-2.5 for host, and
set GDB's libthread-db-search-path such that it finds the libthread_db.so.1 built in step 1.
You don't actually need to build the entire libc in step 1. Something like this should suffice:
mkdir build && cd build
../configure --prefix=/usr
make -C ../nptl_db objdir=`pwd`
Update:
i have GDB 6.6 and there isn't libthread-db-search-path. What is another way to specify that path?
That GDB will just dlopen("libthread_db.so.1", ...). So to make it find the right libthread_db.so.1, you need to adjust LD_LIBRARY_PATH. Using bash:
LD_LIBRARY_PATH=/tmp/glibc-2.5/build/nptl_db gdb /path/to/target/a.out
I am using gdbserver on target and CodeSourcery IDE. My hardware is a gumstix with a omap3530.
I can step through code in my main application but if I attempt to step into a function in a shared library I get memory address and a debugger terminates.
This is my library that is compiled and copied to the /lib folder on the target system.(it does have debug symbols) I have attempted to use the .gbdinit file to set solib-absolute-prefix /lib
Here are the warnings from the gdb trace:
903,056 13-gdb-set sysroot-on-target /lib
903,065 13^done
903,065 (gdb)
903,065 14-target-select remote 192.168.1.101:2345
903,114 =thread-group-started,id="i1",pid="42000"
903,114 =thread-created,id="1",group-id="i1"
903,115 15-list-thread-groups --available
903,120 16-list-thread-groups
903,128 &"warning: Unable to find dynamic linker breakpoint function.\nGDB will be unable to debug shared library initializers\nand track explicitly loaded dynamic code."
903,128 &"\n"
Which leads to
903,395 &"Error while mapping shared library sections:\n"
903,397 &"/lib/libCoreLib.so: Invalid argument.\n"
903,399 =library-loaded,id="/lib/libCoreLib.so",target-name="/lib/libCoreLib.so",hostname="/lib/libCoreLib.so",low-address="0x0",high-address="0x0",symbols-loaded="0",thread-group="i1"
You can debug with the library installed on your host, provided the debugging machine is also the development machine. In that case, you use set sysroot instead of set sysroot-on-target. For example :
set sysroot /home/username/.../rootfs/
where /home/username/.../rootfs/ contains a copy of your target filesystem
I think you should also specify / instead of /lib
Target with debug symbols
This is the simplest method to get working, and it is specially useful when you are developing one particular shared library.
First copy the test executable and shared library to the target with debug information:
check with readelf ----debug-dump=decodedline libmyib.so: How can I tell if a library was compiled with -g?
I recommend using an NFS server on host, so that the compiled output gets automatically uploaded after compilation
Then on target:
gdbserver --multi :1234 ./executable_name
Host:
arm-linux-gnueabihf-gdb -q -nh \
-ex "target extended-remote target-hostname-or-ip:1234" \
-ex "file ./executable_name" \
-ex 'tb main' \
-ex 'c' \
-ex 'set solib-search-path .'
sharedlibrary libmylib.so also works.
The problem I had was that gdbserver stops at the dynamic loader, before main, and the dynamic libraries are not yet loaded at that point, and so GDB does not know where the symbols will go in memory yet.
GDB appears to have some mechanisms to automatically load shared library symbols, and if I compile for host, and run gdbserver locally, running to main is not needed. But on the ARM target, that is the most reliable thing to do.
Target gdbserver 7.12-6, host arm-linux-gnueabihf-gdb 7.6.1 from Linaro.
Target libraries without debug symbols
It is common to strip target libraries before deployment on embedded targets, since debug information makes them way larger.
For example, Buildroot does that by default, but you can disable it with BR2_STRIP_none=y.
You can identify this scenario by running:
info shared
Which shows something like:
From To Syms Read Shared Object Library
0x00007ffff7df7f90 0x00007ffff7dfcdd7 Yes (*) target:/lib/ld64-uClibc.so.0
0x00007ffff7b3a9b0 0x00007ffff7bbe05d Yes (*) target:/lib/libc.so.0
(*): Shared library is missing debugging information.
so there are asterisks (*) for both of the libraries which says that debug information is missing.
If that is the case, then you have to tell GDB to use the shared libraries on host before they were stripped.
Buildroot for example makes that easy for us, as it maintains the staging directory which contains the shared libraries before they were stripped and in the same relative paths as in the target:
set sysroot buildroot/output/staging/
When this option is set, gdb immediately searches for libraries in the host instead of target, and finds /lib/libc.so.0 at the path buildroot/output/staging/ + /lib/libc.so.0:
Reading symbols from buildroot/output/staging/lib/ld64-uClibc.so.0...done.
Reading symbols from buildroot/output/staging/lib/libc.so.0...done.
TODO: I don't think you can set more than one sysroot, so all your shared libraries must be placed in their correct relative paths as in the target image.
If you check the bad default sysroot, you will see:
show sysroot
give:
target:
which means that gdb searches for shared libraries on the target root / by default.
Similar issue was encountered while debugging. The debug was hanging up. Configuration is as follows
Host: Ubuntu 12.04LTS
IDE: Eclipse Kepler
Target: Beaglebone Black / ARM A8
OS: Angstrom
Solution
Update libraries and includes
Select properties for project in Eclipse
C/C++ General > Paths and Symbols > (include TAB) GNU C > Add > Files
systems > / > usr Change from /usr/lib/gcc/i686-linux-gnu/4/6/include
to /usr/arm-linux-gnueabi/include
C/C++ General > Paths and Symbols > (Include TAB) GNU C++> Add >
Files systems > / > usr /usr/arm-linux-gnueabi/include/c++/4.6.3/arm-linux-gnueabi
C/C++ General > Paths and Symbols > (Library Paths TAB) > Add > Files
systems > / > usr /usr/arm-linux-gnueabi/lib
Good day,
If the 'debug-file-directory' variable in GDB is set incorrectly,
then the reported error messages contains:
warning: Unable to find dynamic linker breakpoint function.
The root filesystem of the target is located on my host PC at
/opt/arm-linux-gnueabihf-rootfs
The following two commands helped me to get remote debugging working
via gdbserver using GDB (v7.11.1):
set debug-file-directory /opt/arm-linux-gnueabihf-rootfs/usr/lib/debug
set sysroot /opt/arm-linux-gnueabihf-rootfs
I've noticed that if 'sysroot' has a trailing slash in the path,
then GDB fails to use it.You will see this (incorrect output) after connecting to the remote target:
Reading /lib/ld-linux-armhf.so.3 from remote target...
or
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/lib/ld-linux-
armhf.so.3...(no debugging symbols found)...done
instead of the correct output:
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/lib/ld-linux-
armhf.so.3...
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/usr/lib/debug/
lib/arm-linux-gnueabihf/ld-2.23.so...done.
Regards,
Frikkie Thirion