I have a core dump that shows a thread dying from a SIGBUS signal while executing mov %r15d,0xa0(%rsp). That seems to tell me that it died because it ran out of thread stack.
But how can I prove it? I cannot seem to find a GDB command to display thread information besides thread backtraces. In this case there is no backtrace. It shows the current function and then 0x0000000000000000. Yet another indication of stack corruption, I think.
I don't have a copy of /proc/[pid]/maps from when the program died. Is there anything in GDB or in the core file I can look at to find the base of each thread stack?
That seems to tell me that it died because it ran out of thread stack.
Very likely
But how can I prove it?
(gdb) p/x $rsp
$1 = 0x7fffc5791000
(gdb) info target
Symbols from "a.out".
Local core dump file:
`core', file type elf64-x86-64.
0x0000000000400000 - 0x0000000000401000 is load1
...
0x00007faaf2240000 - 0x00007faaf2241000 is load14
0x00007fffc5791000 - 0x00007fffc5f91000 is load15
0x00007fffc5faf000 - 0x00007fffc5fb0000 is load16
0xffffffffff600000 - 0xffffffffff600000 is load17
Local exec file:
...
Note how $rsp is at the (low) end of the load15 segment, and there is no mapping that "covers" $rsp-8
Related
This question already has answers here:
Debugging core files generated on a Customer's box
(5 answers)
Closed 2 years ago.
So I have my core dump after setting the ulimit: (ulimit -c unlimited)
The core dump comes from another system that is experiencing some issues.
I have copied the core over to my dev system to examine it.
I go into gdb:
$ gdb -c core
...
Core was generated by `./ovcc'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fedd95678a9 in ?? ()
[Current thread is 1 (LWP 15155)]
(gdb) symbol-file ./ovcc
Reading symbols from ./ovcc...
(gdb) bt
#0 0x00007fedd95678a9 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x000055e01cd5e7e0 in ?? ()
#3 0x00007fedd21e9e00 in ?? ()
#4 0x0000000000000201 in ?? ()
#5 0x000055e01cd5e7e0 in ?? ()
#6 0x0000000000000201 in ?? ()
#7 0x0000000000000000 in ?? ()
(gdb)
I check the compile and link commands and they both have "-g" and I can visually step through the program with the codium debugger!
So why can't I see where the executable is crashing?
What have I missed?
Is the problem the fact that the core was created on another system?
Is the problem the fact that the core was created on another system?
Yes, exactly.
See this answer for possible solutions.
Update:
So does this mean I can only debug the program on the system where it is both built and crashes?
It is certainly not true that you can only debug a core on the system where the binary was both built and crashed -- I debug core dumps from different systems every day, and in my case the build host, the host where the program crashed, and the host on which I debug are all separate.
One thing I just noticed: your style of loading the core: gdb -c core followed by symbol-file, doesn't work for PIE executables (at least when using GDB 10.0) -- this may be a bug in GDB.
The "regular" way of loading the core is:
gdb ./ovcc core
See if that gives you better results. (You still need to arrange for matching DSOs, as linked answer shows how to do.)
If I load the crashing program and the core dump into gdb, it shows me a stack trace and crash point as below.
Core was generated by `./cut --output-d=: -b1,1234567890- /dev/fd/63'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 is_printable_field (i=1234567890) at src/cut.c:266
266 return (printable_field[n] >> (i % CHAR_BIT)) & 1;
(gdb) bt
#0 is_printable_field (i=1234567890) at src/cut.c:266
#1 set_fields (fieldstr=0x7ffccb0561c4 "") at src/cut.c:533
#2 main (argc=4, argv=0x7ffccb055cf8) at src/cut.c:865
Is there any means to know the exact assembly instruction that caused the segfault?
One possibility is to set:
(gdb)layout asm
When GDB stops the corresponding assembly line is pointed.
Example:
│0x7ffff7aa441d <strtok+45> je 0x7ffff7aa44d6 <strtok+230> │
│0x7ffff7aa4423 <strtok+51> mov %rsi,%rax │
>│0x7ffff7aa4426 <strtok+54> mov (%rax),%cl │
│0x7ffff7aa4428 <strtok+56> test %cl,%cl │
│0x7ffff7aa442a <strtok+58> je 0x7ffff7aa4454 <strtok+100>
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7aa4426 in strtok () from /lib64/libc.so.6
(gdb)
You could use the disassemble GDB command. Also perhaps use x/i on $rip (the program counter on x86-64)
However, in your case, assuming the code is in C (not C++ with some operator []), the only possible culprits are the printable_field pointer, or the n index.
Consider also using valgrind and/or compiling (in addition of -g -Wall options to a recent GCC compiler) with -fsanitize=... options, notably -fsanitize=address or -fsanitize=undefined...
I'm running a Qt5.4.0 application on my embedded Linux system (TI AM335x based) and it's stopping to run and I'm having a hard time debugging this. This is a QtWebKit QML example (youtubeview) but other QtWebKit examples are preforming the same for me so it's something WebKit based on my system.
When I run the application, it runs for a second or so, then ends with no messages. There is nothing reported to the syslog or dmesg either. When I kick it off with strace I can see this futex message:
futex(0x2d990, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x2d9ac, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
+++ exited with 1 +++
Then it stops. Not very helpful... My next though was to debug this with GDB, however GDB crashes when I try to run this:
-sh-4.2# gdb youtubeview
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
...
(gdb) run
Starting program: /usr/share/qt5/examples/webkitqml/youtubeview/youtubeview
/home/mike/ulf_qt_450/ulf/build-ulf/out/work/armv7ahf-vfp-neon-linux-gnueabihf/gdb/7.5-r0.0/gdb-7.5/gdb/utils.c:1081: internal-error: virtual memory exhausted: can't allocate 64652911 bytes.
A problem internal to GDB has been detected,
This issue occurs even if I set a break point at main first, just as soon as it starts running it will get stuck and run out of memory.
Are there other tools or techniques that can be used here to help isolate the issue?
Perhaps arguments to GDB to limit memory use or give some more information about why this Qt program made it crash?
Perhaps some FDs or system variables I could use to figure out why the FUTEX is being held and failing?
I'm not sure where to take this problem right now.
The Qt code itself is pretty simple, and I don't anticipate any issues in here:
#include <QGuiApplication>
#include <QQuickView>
int main(int argc, char* argv[])
{
QGuiApplication app(argc,argv);
QQuickView view;
view.setSource(QUrl("qrc:///" QWEBKIT_EXAMPLE_NAME ".qml"));
view.setResizeMode(QQuickView::SizeRootObjectToView);
view.show();
return app.exec();
return 0;
}
Running gdb on the device, especially with a huge library such as WebKit, is bound to get you out of memory errors.
Instead, run gdbserver on the device, and connect to it via gdb running on the host machine, using the toolchain's cross-gdb for that. In that scenario, the gdb on the host loads all the debug information, while the gdbserver on the device needs almost no memory.
It is even possible to have the separate debug information available on the host and stripped libraries on the device.
Please note that parts of WebKit are always built in release mode, even if the rest of Qt was built in debug mode, if you are going to debug into WebKit you might want to change that in the build system.
Here is a minimal example:
Device:
# gdbserver 192.168.1.2:12345 myapp
Process myapp created; pid = 989
Listening on port 12345
Host:
# arm-none-linux-gnueabi-gdb myapp
GNU gdb (Sourcery G++ Lite 2009q1-203) 6.8.50.20081022-cvs
(gdb) set solib-absolute-prefix /opt/targetroot
(gdb) target remote 192.168.1.42:12345
Remote debugging using 192.168.1.42:12345
(gdb) start
The "remote" target does not support "run". Try "help target" or "continue".
(gdb) break main
Breakpoint 1 at 0x1ab9c: file myapp/main.cpp, line 12.
(gdb) cont
Continuing.
Breakpoint 1, main (argc=1, argv=0xbecfedb4) at myapp/main.cpp:12
12 QApplication app(argc, argv, QApplication::GuiServer);
And you are right that it looks like a problem in QtWebKit itself, not in your application. Good luck!
I just know when gdb attach to a process, I can use p $_siginfo._sifields._sigfault.si_addr to show what address caused a segfault.
But, how to do in a core dump file?
I try it in a core dump file:
(gdb) p $_siginfo._sifields._sigfault.si_addr
Unable to read siginfo
when loading a core dump into GDB the reason why it crashed automatically is displayed. For example
Program terminated with signal 11, Segmentation fault.
Is there any way to get the information again?
The thing is, that I'm writing a script which needs this information. But if the signal is only available after loading the core dump amd I can't access the information later on.
Is there really no command for such an important feature?
To print the information about the last signal execute
p $_siginfo
If you know what the core file name is, you can issue the target core command which respecifies the target core file:
(gdb) target core core.8577
[New LWP 8577]
Core was generated by `./fault'.
Program terminated with signal 11, Segmentation fault.
#0 0x080483d5 in main () at fault.c:10
10 *ptr = '\123';
(gdb)
As for the implied question, what is the info last signal command?, I don't know. There does not seem to be one.
The core file's name can be obtained from the command info target:
(gdb) info target
Symbols from "/home/wally/.bin/fault".
Local core dump file:
`/home/wally/.bin/core.8577', file type elf32-i386.
0x00da1000 - 0x00da2000 is load1
0x08048000 - 0x08049000 is load2
...
0xbfe8d000 - 0xbfeaf000 is load14
Local exec file:
`/home/wally/.bin/fault', file type elf32-i386.
Entry point: 0x8048300
0x08048134 - 0x08048147 is .interp
0x08048148 - 0x08048168 is .note.ABI-tag
0x08048168 - 0x0804818c is .note.gnu.build-id
0x0804818c - 0x080481ac is .gnu.hash
0x080481ac - 0x080481fc is .dynsym
0x080481fc - 0x08048246 is .dynstr
...