I am attempting to carry out postmortem analysis of a crashed binary, "TestApp", on a linux system.
I have a copy of the binaries and and shared objects that are copied onto the device in a path:
/usr/public/target
This folder contains all the binaries in question in the directory structure used on the system under test, ie:
/usr/public/target/sbin/TestApp
/usr/public/target/lib/TestAppLib.so
/usr/public/target/usr/lib/TestAppAPILib.so
The automated build process strips the debug information from the binaries, and stores them in external, symbol files, all under:
/usr/public/target_external_symbols
So the symbol information for the above binaries would exist in files named:
/usr/public/target_external_symbols/sbin/TestApp.sym
/usr/public/target_external_symbols/lib/TestAppLib.so.sym
/usr/public/target_external_symbols/usr/lib/TestAppAPILib.so.sym
How do I get GDB to be aware of the existence of these external symbols and to load them?
I typically invoke GDB via:
gdb TestApp TestApp.core
I've referred to other articles on creating a test .gdbinit file and passing it to GDB via the -command argument, but it doesn't appear to work. Every time I attempt to get a backtrace from my core file, I get an indication from GDB that it cannot open the debug symbols. Any help in resolving this is appreciated.
(gdb) info shared
From To Syms Read Shared Object Library
0x78000000 0x780061e8 Yes (*) /usr/public/target/lib/TestAppLib.so
0x78010000 0x7806e60c Yes (*) /usr/public/target/usr/lib/TestAppAPILib.so
0x78070000 0x78091d2c Yes (*) /usr/public/target/lib/libm.so.2
(*): Shared library is missing debugging information.
Thank you.
There are commands set debug-file-directory, symbol-file and add-symbol-file to load debugging symbols from within a gdb session. The latter one might require the address where the shared library was loaded into memory.
Maybe during your build process 'gnu debuglinks' have been added to your binaries. This would mean that in the executables there's a path coded that directs gdb where to look for debug symbols. More can be found here.
Related
I realize a question with the exact title has already been answered, but the steps there requires running the compiler and linker manually, whereas I want to use cmake.
I am trying to debug a C program with WinDbg. But I'm getting this error:
*** WARNING: Unable to verify checksum for main.exe
Reading a mailing list thread1, I'm guessing I need to add a few flags, namely '/Zi' and '/Release'. But I'm building my project with cmake, and I don't know how to add those flags properly so that I can build my program using GNU toolchain with debug symbols too.
My CMakeLists.txt:
cmake_minimum_required(VERSION 3.00)
project(Hello LANGUAGES C)
add_executable(main src/main.c)
With the above cmake file, my program is built properly. Even a pdb file is generated, which is read by WinDbg no problem. But I can't see the line information with .lines and no source file is shown when debugging the EXE; only assembly commands are shown.
After the reading the mail thread (mentioned above), I checked the checksum value of my EXE. It's zero. Now I need to know how to set up a cmake file so it produces EXE with debug symbols with proper checksum.
The checksum-verification warning turned out not to be the issue (it was just a warning after all, not an error). WinDbg didn't load line information. Either it's the default (although I don't know why that would be) or I mistakenly turned it off myself. Whatever the case, here is how you turn it on:
.lines -e
After that, WinDbg was able to bring up the source window by its own accord when I started debugging.
When I attach to process with GDB I saw that warning:
Reading symbols from /CloneFileSystem/lib/ld-uClibc.so.0...
(No debugging symbols found in /CloneFileSystem/lib/ld-uClibc.so.0)
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Even I use set sysroot /CloneFileSystem with all the libraries there, that not help.
My process call to function in shared library and that function call to another function in another shared library.
For example ProcessFunc-> Lib1Func -> Lib2Func
I want to debug Lib2Func but when I run command
b Lib2Func
gdb can't set a breakpoint there event that library is already loaded
file myexe -> interpreter /lib/ld-uClibc.so.0
When I use info sharedlibrary in GDB I can't the value in From and To (address of loaded are empty)
You might want to try recompiling the library with a -g option. It's saying it can't find the debug info. -g provides for debugging info if you are using gcc. This is the first thing i'd check. If the library is big this may take awhile.
I got a core while executing an application and I saved the executable, the corefile and a shared library which the application use in /tmp to check them later. I then modified the library, rebuilt it and started the executable again.
Now when I am trying to debug the core, gdb is loading the shared library from its original path and not from the directory /tmp where I saved the original library.
For example, the original path was /opt/mydir/lib/libmylib.so.0.
gdb is loading this shared library, while I want it to load /tmp/libmylib.so.0.
The application also uses some standard libraries which are in /usr/lib and /lib directories so I don't want these paths to get changed. Just want to change /opt/mydir/lib/ -> /tmp. How can I do that?
The simplest solution is to temporarily restore /opt/mydir/lib/libmylib.so.0 to the copy that was used at crash time (i.e. the one now in /tmp), analyse the core, then restore back the new version.
If you don't want to do that, set solib-search-path and set sysroot are your friends.
Note that you must set both before loading the core. This sequence should work:
(gdb) set sysroot /no/such/file
(gdb) set solib-search-path /tmp:/usr/lib:/lib
(gdb) core /tmp/core
I am using gdbserver on target and CodeSourcery IDE. My hardware is a gumstix with a omap3530.
I can step through code in my main application but if I attempt to step into a function in a shared library I get memory address and a debugger terminates.
This is my library that is compiled and copied to the /lib folder on the target system.(it does have debug symbols) I have attempted to use the .gbdinit file to set solib-absolute-prefix /lib
Here are the warnings from the gdb trace:
903,056 13-gdb-set sysroot-on-target /lib
903,065 13^done
903,065 (gdb)
903,065 14-target-select remote 192.168.1.101:2345
903,114 =thread-group-started,id="i1",pid="42000"
903,114 =thread-created,id="1",group-id="i1"
903,115 15-list-thread-groups --available
903,120 16-list-thread-groups
903,128 &"warning: Unable to find dynamic linker breakpoint function.\nGDB will be unable to debug shared library initializers\nand track explicitly loaded dynamic code."
903,128 &"\n"
Which leads to
903,395 &"Error while mapping shared library sections:\n"
903,397 &"/lib/libCoreLib.so: Invalid argument.\n"
903,399 =library-loaded,id="/lib/libCoreLib.so",target-name="/lib/libCoreLib.so",hostname="/lib/libCoreLib.so",low-address="0x0",high-address="0x0",symbols-loaded="0",thread-group="i1"
You can debug with the library installed on your host, provided the debugging machine is also the development machine. In that case, you use set sysroot instead of set sysroot-on-target. For example :
set sysroot /home/username/.../rootfs/
where /home/username/.../rootfs/ contains a copy of your target filesystem
I think you should also specify / instead of /lib
Target with debug symbols
This is the simplest method to get working, and it is specially useful when you are developing one particular shared library.
First copy the test executable and shared library to the target with debug information:
check with readelf ----debug-dump=decodedline libmyib.so: How can I tell if a library was compiled with -g?
I recommend using an NFS server on host, so that the compiled output gets automatically uploaded after compilation
Then on target:
gdbserver --multi :1234 ./executable_name
Host:
arm-linux-gnueabihf-gdb -q -nh \
-ex "target extended-remote target-hostname-or-ip:1234" \
-ex "file ./executable_name" \
-ex 'tb main' \
-ex 'c' \
-ex 'set solib-search-path .'
sharedlibrary libmylib.so also works.
The problem I had was that gdbserver stops at the dynamic loader, before main, and the dynamic libraries are not yet loaded at that point, and so GDB does not know where the symbols will go in memory yet.
GDB appears to have some mechanisms to automatically load shared library symbols, and if I compile for host, and run gdbserver locally, running to main is not needed. But on the ARM target, that is the most reliable thing to do.
Target gdbserver 7.12-6, host arm-linux-gnueabihf-gdb 7.6.1 from Linaro.
Target libraries without debug symbols
It is common to strip target libraries before deployment on embedded targets, since debug information makes them way larger.
For example, Buildroot does that by default, but you can disable it with BR2_STRIP_none=y.
You can identify this scenario by running:
info shared
Which shows something like:
From To Syms Read Shared Object Library
0x00007ffff7df7f90 0x00007ffff7dfcdd7 Yes (*) target:/lib/ld64-uClibc.so.0
0x00007ffff7b3a9b0 0x00007ffff7bbe05d Yes (*) target:/lib/libc.so.0
(*): Shared library is missing debugging information.
so there are asterisks (*) for both of the libraries which says that debug information is missing.
If that is the case, then you have to tell GDB to use the shared libraries on host before they were stripped.
Buildroot for example makes that easy for us, as it maintains the staging directory which contains the shared libraries before they were stripped and in the same relative paths as in the target:
set sysroot buildroot/output/staging/
When this option is set, gdb immediately searches for libraries in the host instead of target, and finds /lib/libc.so.0 at the path buildroot/output/staging/ + /lib/libc.so.0:
Reading symbols from buildroot/output/staging/lib/ld64-uClibc.so.0...done.
Reading symbols from buildroot/output/staging/lib/libc.so.0...done.
TODO: I don't think you can set more than one sysroot, so all your shared libraries must be placed in their correct relative paths as in the target image.
If you check the bad default sysroot, you will see:
show sysroot
give:
target:
which means that gdb searches for shared libraries on the target root / by default.
Similar issue was encountered while debugging. The debug was hanging up. Configuration is as follows
Host: Ubuntu 12.04LTS
IDE: Eclipse Kepler
Target: Beaglebone Black / ARM A8
OS: Angstrom
Solution
Update libraries and includes
Select properties for project in Eclipse
C/C++ General > Paths and Symbols > (include TAB) GNU C > Add > Files
systems > / > usr Change from /usr/lib/gcc/i686-linux-gnu/4/6/include
to /usr/arm-linux-gnueabi/include
C/C++ General > Paths and Symbols > (Include TAB) GNU C++> Add >
Files systems > / > usr /usr/arm-linux-gnueabi/include/c++/4.6.3/arm-linux-gnueabi
C/C++ General > Paths and Symbols > (Library Paths TAB) > Add > Files
systems > / > usr /usr/arm-linux-gnueabi/lib
Good day,
If the 'debug-file-directory' variable in GDB is set incorrectly,
then the reported error messages contains:
warning: Unable to find dynamic linker breakpoint function.
The root filesystem of the target is located on my host PC at
/opt/arm-linux-gnueabihf-rootfs
The following two commands helped me to get remote debugging working
via gdbserver using GDB (v7.11.1):
set debug-file-directory /opt/arm-linux-gnueabihf-rootfs/usr/lib/debug
set sysroot /opt/arm-linux-gnueabihf-rootfs
I've noticed that if 'sysroot' has a trailing slash in the path,
then GDB fails to use it.You will see this (incorrect output) after connecting to the remote target:
Reading /lib/ld-linux-armhf.so.3 from remote target...
or
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/lib/ld-linux-
armhf.so.3...(no debugging symbols found)...done
instead of the correct output:
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/lib/ld-linux-
armhf.so.3...
Reading symbols from /opt/arm-linux-gnueabihf-rootfs/usr/lib/debug/
lib/arm-linux-gnueabihf/ld-2.23.so...done.
Regards,
Frikkie Thirion
I have a core file generated on a remote system that I don't have direct access to. I also have local copies of the library files from the remote system, and the executable file for the crashing program.
I'd like to analyse this core dump in gdb.
For example:
gdb path/to/executable path/to/corefile
My libraries are in the current directory.
In the past I've seen debuggers implement this by supplying the option "-p ." or "-p /=."; so my question is:
How can I specify that libraries be loaded first from paths relative to my current directory when analysing a corefile in gdb?
Start gdb without specifying the executable or core file, then type the following commands:
set solib-absolute-prefix ./usr
file path/to/executable
core-file path/to/corefile
You will need to make sure to mirror your library path exactly from the target system. The above is meant for debugging targets that don't match your host, that is why it's important to replicate your root filesystem structure containing your libraries.
If you are remote debugging a server that is the same architecture and Linux/glibc version as your host, then you can do as fd suggested:
set solib-search-path <path>
If you are trying to override some of the libraries, but not all then you can copy the target library directory structure into a temporary place and use the solib-absolute-prefix solution described above.
I'm not sure this is possible at all within gdb but then I'm no expert.
However I can comment on the Linux dynamic linker. The following should print the path of all resolved shared libraries and the unresolved ones.
ldd path/to/executable
We need to know how your shared libraries were linked with your executable. To do this, use the following command:
readelf -d path/to/executable | grep RPATH
Should the command print nothing, the dynamic linker will use standard locations plus the LD_LIBRARY_PATH environment variable to find the shared libraries.
If the command prints some lines, the dynamic linker will ignore LD_LIBRARY_PATH and use the hardcoded rpaths instead.
If the listed rpaths are absolute, the only solution I know is to copy (or symlink) your libraries to the listed locations.
If the listed rpaths are relative, they will contain a $ORIGIN which will be replaced at run time by the path of the executable. Move either the executable or the libraries to match.
For further informations, you could start with:
man ld.so
I found this excerpt on developer.apple.com
set solib-search-path path
If this variable is set, path is a
colon-separated list of directories to
search for shared libraries.
solib-search-path' is used after
solib-absolute-prefix' fails to
locate the library, or if the path to
the library is relative instead of
absolute. If you want to use
solib-search-path' instead of
solib-absolute-prefix', be sure to
set `solib-absolute-prefix' to a
nonexistant directory to prevent GDB
from finding your host's libraries.
EDIT:
I don't think using the above setting prepends the directories I added, but it does seem to append them, so files missing from my current system are picked up in the paths I added. I guess setting the solib-absolute-prefix to something bogus and adding directories in the solib-search-path in the order I need might be a full solution.
You can also just set LD_PRELOAD to each of the libraries or LD_LIBRARY_PATH to the current directory when invoking gdb. This will only cause problems if gdb itself tries to use any of the libraries you're preloading.
One important note:
if you're doing a cross compiling and trying to debug with gdb, then
after you've done file ECECUTABLE_NAME if you see smth. like :
Using host libthread_db library "/lib/libthread_db.so.1"
then check whether you have libthread_db for your target system. I found a lot of similar problems on the web. Such problem cannot be solved just using "set solib-", you has to build libthread_db using your cross-compiler as well.