I was trying to debug a deadlock issue in my code using gdb. When the program locked up, I attached to it with gdb to see what the threads were doing. For some reason, I cannot obtain backtraces for threads other than the current one:
(gdb) bt
#0 0xb774dcb0 in ?? ()
#1 0x8420e178 in TableCollect
#2 0x84200c12 in main
(gdb) thread apply all bt
Thread 3 (Thread 0xb6becb40 (LWP 18937)):
#0 0xb774dcb0 in ?? ()
#1 0x000d6ee6 in ?? ()
Thread 2 (Thread 0xb1f5eb40 (LWP 18939)):
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x5:
#0 0xb774dcb0 in ?? ()
Cannot access memory at address 0x5
(gdb) info threads
Id Target Id Frame
3 Thread 0xb6becb40 (LWP 18937) "loader.elf" 0xb774dcb0 in ?? ()
2 Thread 0xb1f5eb40 (LWP 18939) "Tuner" 0xb774dcb0 in ?? ()
* 1 Thread 0xb6bed700 (LWP 18936) "loader.elf" 0xb774dcb0 in ?? ()
(gdb) thread 2
[Switching to thread 2 (Thread 0xb1f5eb40 (LWP 18939))]
#0 0xb774dcb0 in ?? ()
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x5:
#0 0xb774dcb0 in ?? ()
Cannot access memory at address 0x5
(gdb) thread 3
[Switching to thread 3 (Thread 0xb6becb40 (LWP 18937))]
#0 0xb774dcb0 in ?? ()
(gdb) bt
#0 0xb774dcb0 in ?? ()
#1 0x000d6ee6 in ?? ()
ninja#vm:build$ cat /proc/`pidof loader.elf`/maps | grep b774d
b774b000-b774d000 r--p 00000000 00:00 0 [vvar]
b774d000-b774f000 r-xp 00000000 00:00 0 [vdso]
So it looks like all threads are inside a system call? That's likely since I'm expecting them all to be stuck on ptread_mutex_lock, but why don't I have valid stack frames below that? Why the 0x5 address for thread 2? And why does thread 3 have 0xd6ee6 on the stack, which isn't even mentioned in the memory map?
Related
I have a pintool that has calls to PIN_Backtrace(). It seems that libunwind does not support some registers in x86_64 and this leads to the following error message and an abnormal exit:
libunwind: uint64_t libunwind::Registers_x86_64::getRegister(int) const Source/bionic/libunwind/src/Registers.hpp:342 - unsupported x86_64 register
Aborted
I want the program to continue even with these errors. What is the most efficient way to handle this situation?
I debugged my pintool using gdb. Here is the backtrace when abort() is received by the debugger:
(gdb) bt
#0 0x00007ffff726c22b in ?? ()
#1 0x00007fffeca88730 in ?? ()
#2 0x00007ffff726c363 in ?? ()
#3 0x0000000000005386 in ?? ()
#4 0x0000000000000001 in ?? ()
#5 0x0000000000000000 in ?? ()
info proc mapping shows the address ranges as follows:
...
0x7fffeea3a000 0x7ffff71fb000 0x87c1000 0x0
0x7ffff71fb000 0x7ffff7283000 0x88000 0x0 /home/ahmad/Programs/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libc-dynamic.so
0x7ffff7283000 0x7ffff7284000 0x1000 0x0
0x7ffff7284000 0x7ffff7286000 0x2000 0x88000 /home/ahmad/Programs/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libc-dynamic.so
...
and
...
0x7fffe1687000 0x7fffec287000 0xac00000 0x0
0x7fffec287000 0x7fffed28e000 0x1007000 0x0
0x7fffed28e000 0x7fffedb6a000 0x8dc000 0x0
...
It seems that the libunwind library used by Pin is a special one exlusive to PinCRT. This can be seen in the info /proc/<PID>/maps output for the pintool:
...
7ffff7e33000-7ffff7e45000 r-xp 00000000 08:13 466277 /home/ahmad/Programs/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libunwind-dynamic.so
7ffff7e45000-7ffff7e46000 r--p 00011000 08:13 466277 /home/ahmad/Programs/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libunwind-dynamic.so
7ffff7e46000-7ffff7e47000 rw-p 00012000 08:13 466277 /home/ahmad/Programs/pin-3.7-97619-g0d0c92f4f-gcc-linux/intel64/runtime/pincrt/libunwind-dynamic.so
...
I have some complicated problem (by complicated I mean that I couldn't find a solution within a few hours of researching) and the problem is:
I submitted large amount of scripts to run (on an SGE cluster), some of those scripts generated core.# files (core dump files). I figured I could open those files with gdb, now when I simply open the core.# file I see this:
(the last few lines of the gdb output)
Core was generated by `/tools/graphmap/bin/Linux-x64/graphmap align --overlappe'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004a554b in ?? ()
"/4thExp/core.82912" is a core file.
Please specify an executable to debug.
Now to my question - I need to find what was the arguments to the program that caused the crash.
The output mentioned above shows only the beginning of the command that caused the crash: "Core was generated by `/groups/nshomron/artemd/tools/graphmap/bin/Linux-x64/graphmap align --overlappe'."
If I could see the rest of the command I would solve my problem, but after hours of searching online and checking gdb manual I couldn't find anything useful. I tried loading gdb with the program that caused the crash "gdb ..../graphmap core.#" but still I got only the beginning of the faulty command and couldn't get anything else.
Any help suggestion would be very appreciated.
Update: As the user #ks1322 suggested - I looked closely at the threads.
First I executed "info threads" and got the output:
(gdb) info threads
24 Thread 0x2b29409bd700 (LWP 82927) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
23 Thread 0x2b29401b9700 (LWP 82923) 0x00000031d00f820e in __lll_lock_wait_private () from /lib64/libc.so.6
* 22 Thread 0x2b29403ba700 (LWP 82924) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
21 Thread 0x2b29413c2700 (LWP 82932) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
20 Thread 0x2b293fbb6700 (LWP 82920) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
19 Thread 0x2b293fdb7700 (LWP 82921) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
18 Thread 0x2b2940bbe700 (LWP 82928) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
17 Thread 0x2b293f3b2700 (LWP 82916) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
16 Thread 0x2b29411c1700 (LWP 82931) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
15 Thread 0x2b2940dbf700 (LWP 82929) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
14 Thread 0x2b29419c5700 (LWP 82935) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
13 Thread 0x2b293efb0700 (LWP 82914) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
12 Thread 0x2b293f7b4700 (LWP 82918) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
11 Thread 0x2b29407bc700 (LWP 82926) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
10 Thread 0x2b293f9b5700 (LWP 82919) 0x00000031d00f820e in __lll_lock_wait_private () from /lib64/libc.so.6
9 Thread 0x2b29415c3700 (LWP 82933) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
8 Thread 0x2b29405bb700 (LWP 82925) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
7 Thread 0x2b292ea08be0 (LWP 82912) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
6 Thread 0x2b293ffb8700 (LWP 82922) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
5 Thread 0x2b293edaf700 (LWP 82913) 0x00000031d0045063 in vfprintf () from /lib64/libc.so.6
4 Thread 0x2b2940fc0700 (LWP 82930) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
3 Thread 0x2b293f1b1700 (LWP 82915) 0x00000031d00ac6aa in times () from /lib64/libc.so.6
2 Thread 0x2b29417c4700 (LWP 82934) 0x0000000000412bd6 in obtainAlignment(unsigned char const*, unsigned char const*, int, unsigned char const*, unsigned char const*, int, int, int, unsigned char**, int*) ()
1 Thread 0x2b293f5b3700 (LWP 82917) 0x00000000004a554b in GraphMap::ProcessKmerCacheFriendly_(signed char*, long, ScoreRegistry*, MappingData*, Index*, SingleSequence const*, ProgramParameters const*) ()
It didn't tell me very much so I continued to look for a "main thread". I switched to each thread, one by one, and executed "info stack". The only thread containing something relevant was thread 7. the info stack output:
(gdb) thread 7
[Switching to thread 7 (Thread 0x2b292ea08be0 (LWP 82912))]#0 0x00000031d00ac6aa in times () from /lib64/libc.so.6
(gdb) info stack
#0 0x00000031d00ac6aa in times () from /lib64/libc.so.6
#1 0x00000031d009bcba in clock () from /lib64/libc.so.6
#2 0x00000000004ccaed in GraphMap::RegionSelectionNoCopy_(long, MappingData*, std::vector<Index*, std::allocator<Index*> >, SingleSequence const*, ProgramParameters const*) ()
#3 0x00000000004c3544 in GraphMap::ProcessRead(MappingData*, std::vector<Index*, std::allocator<Index*> >, SingleSequence const*, ProgramParameters const*, EValueParams const*) ()
#4 0x00000000004adf43 in GraphMap::ProcessSequenceFileInParallel ()
#5 0x00002b292e7f096f in GOMP_parallel () from /share/apps/gcc/gcc530/lib64/libgomp.so.1
#6 0x00000000004b0b08 in GraphMap::ProcessSequenceFileInParallel(ProgramParameters*, SequenceFile*, long*, _IO_FILE*, long*, long*) ()
#7 0x00000000004b1796 in GraphMap::ProcessReadsFromSingleFile(ProgramParameters&, _IO_FILE*) ()
#8 0x00000000004b281e in GraphMap::Run(ProgramParameters&) ()
#9 0x00000000005087fe in main ()
But I got stuck again from here (short reminder: my goal is to find the full command that crushed the execution, the beginning of which is displayed on the first page of gdb like this:
Core was generated by `/tools/graphmap/bin/Linux-x64/graphmap align
--overlappe'.
Any help from here would be very appreciated.
Final Update, problem solved: I followed #ks1322 advice and went to this stack overflow thread and then I repeated what was described in the first answer and was able to get the arguments.
(short overview of what I understood with my limited knowledge of working with gdb: First you should check what threads were running the task, you can do it with "info threads" then you need to check which thread has "main" in it's stack, I did it by switching threads one by one with "thread #" and printing the stack with "info stack" until I found the thread that had main in it. in my case it was shown like this in the "info stack" #9 0x00000000005087fe in main ()". Then according to the instructions in the linked thread, I executed "set backtrace past-main" then "bt" and then changed frame to the frame containing "in _start ()" with the command "frame #". Almost done, now I ran the command "x/8gx $rsp+8" with showed few four lines with 2 addressees in each line. In my case the second line looked like this "0x7ffe38f872d8: 0x00007ffe38f88c35 0x00007ffe38f88c73" and now if everything was right this address can contain one of the arguments of the command that caused the crush, you can check it with "x/s" command like so: "x/s 0x00007ffe38f88c35" and it prints the argument. Important note: I had a lot of arguments so I needed to go to later addressees that did not show in the "x/8gx $rsp+8" command, I noticed that the addresses are incremented by constant value (3 in hex) so I kept manually in a calculator adding "3" to the address and checking the address with x/s until I got to my wanted argument)
Very messy solution and I hope someone could find an easier solution but that is all I could find.
Big thanks to #ks1322 who cleared up things for me and guided me to the solution.
You can load core dump with matching binary (the one for which core dump was generated) and print argv values in the frame where main function resides.
Something like this:
gdb /tools/graphmap/bin/Linux-x64/graphmap /4thExp/core.82912
Then go up in stack trace to initial frame where int main(int argc, char *argv[]) resides. Now you can print the number of arguments and their values from gdb prompt.
Update:
It appears that your binary is multithreaded and crash happened in some auxiliary thread. You should therefore find main thread and switch to it. Here is an example of how to do it for Firefox with many threads:
(gdb) t a a bt -1
Thread 59 (Thread 0x7f691deff700 (LWP 25924)):
#12 0x00007f69dce93f6f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
..........
..........
many threads are listed here
..........
..........
Thread 1 (Thread 0x7f69de01a740 (LWP 4143)):
#17 0x000056374cb38817 in main ()
(gdb) t 1
[Switching to thread 1 (Thread 0x7f69de01a740 (LWP 4143))]
#0 0x00007f69dce8800d in poll () at ../sysdeps/unix/syscall-template.S:84
84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
Now gdb is switched to main thread (Thread 1).
My OS is RHEL 7, and I run a simple Go program:
package main
import (
"time"
)
func main() {
time.Sleep(1000 * time.Second)
}
During its running, I check the thread count of process:
# cat /proc/13858/status | grep Thread
Threads: 5
While using the pstack command shipped on RHEL, it only prints one thread's stack:
# pstack 13858
Thread 1 (process 13858):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:307
#1 0x0000000000422580 in runtime.futexsleep (addr=0x4c7af8 <runtime.timers+24>, val=0, ns=999999997446) at /usr/local/go/src/runtime/os1_linux.go:57
#2 0x000000000040b07b in runtime.notetsleep_internal (n=0x4c7af8 <runtime.timers+24>, ns=999999997446, ~r2=255) at /usr/local/go/src/runtime/lock_futex.go:174
#3 0x000000000040b1e6 in runtime.notetsleepg (n=0x4c7af8 <runtime.timers+24>, ns=999999997446, ~r2=false) at /usr/local/go/src/runtime/lock_futex.go:206
#4 0x000000000043e5de in runtime.timerproc () at /usr/local/go/src/runtime/time.go:209
#5 0x0000000000451001 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1998
#6 0x0000000000000000 in ?? ()
Why does pstack only print one thread's content?
P.S.: The pstack script is here:
#!/bin/sh
if test $# -ne 1; then
echo "Usage: `basename $0 .sh` <process-id>" 1>&2
exit 1
fi
if test ! -r /proc/$1; then
echo "Process $1 not found." 1>&2
exit 1
fi
# GDB doesn't allow "thread apply all bt" when the process isn't
# threaded; need to peek at the process to determine if that or the
# simpler "bt" should be used.
backtrace="bt"
if test -d /proc/$1/task ; then
# Newer kernel; has a task/ directory.
if test `/bin/ls /proc/$1/task | /usr/bin/wc -l` -gt 1 2>/dev/null ; then
backtrace="thread apply all bt"
fi
elif test -f /proc/$1/maps ; then
# Older kernel; go by it loading libpthread.
if /bin/grep -e libpthread /proc/$1/maps > /dev/null 2>&1 ; then
backtrace="thread apply all bt"
fi
fi
GDB=${GDB:-/usr/bin/gdb}
# Run GDB, strip out unwanted noise.
# --readnever is no longer used since .gdb_index is now in use.
$GDB --quiet -nx $GDBARGS /proc/$1/exe $1 <<EOF 2>&1 |
set width 0
set height 0
set pagination no
$backtrace
EOF
/bin/sed -n \
-e 's/^\((gdb) \)*//' \
-e '/^#/p' \
-e '/^Thread/p'
pstack uses gdb. This is a quote from golang doc (https://golang.org/doc/gdb):
GDB does not understand Go programs well. The stack management,
threading, and runtime contain aspects that differ enough from the
execution model GDB expects that they can confuse the debugger, even
when the program is compiled with gccgo. As a consequence, although
GDB can be useful in some situations, it is not a reliable debugger
for Go programs, particularly heavily concurrent ones.
4 out of 5 threads that you see in /proc are created even before you program enters main. I assume that golang runtime creates them.
Why does pstack only print one thread's content?
Judging from output of strace for gdb I see that gdb actually tries to attach to them but after something goes wrong and gdb does not try to inspect these threads. These are syscalls that gdb issued for these runtime threads but due to unknown reason decided stopped investigating them immediately:
5072 ptrace(PTRACE_ATTACH, 5023, 0, 0) = 0
5072 --- SIGCHLD (Child exited) # 0 (0) ---
5072 rt_sigreturn(0x11) = 0
5072 ptrace(PTRACE_ATTACH, 5024, 0, 0) = 0
5072 --- SIGCHLD (Child exited) # 0 (0) ---
5072 rt_sigreturn(0x11) = 0
5072 ptrace(PTRACE_ATTACH, 5025, 0, 0) = 0
5072 --- SIGCHLD (Child exited) # 0 (0) ---
5072 rt_sigreturn(0x11) = 0
However you can inspect them yourself. It seems that these threads belong to golang runtime
$ pstack 5094
Thread 1 (process 5094):
#0 0x0000000000459243 in runtime.futex ()
#1 0x00000000004271e0 in runtime.futexsleep ()
#2 0x000000000040d55b in runtime.notetsleep_internal ()
#3 0x000000000040d64b in runtime.notetsleep ()
#4 0x0000000000435677 in runtime.sysmon ()
#5 0x000000000042e6cc in runtime.mstart1 ()
#6 0x000000000042e5d2 in runtime.mstart ()
#7 0x00000000004592b7 in runtime.clone ()
#8 0x0000000000000000 in ?? ()
$ pstack 5095
Thread 1 (process 5095):
#0 0x0000000000459243 in runtime.futex ()
#1 0x0000000000427143 in runtime.futexsleep ()
#2 0x000000000040d3f4 in runtime.notesleep ()
#3 0x000000000042f6eb in runtime.stopm ()
#4 0x0000000000430a79 in runtime.findrunnable ()
#5 0x00000000004310ff in runtime.schedule ()
#6 0x000000000043139b in runtime.park_m ()
#7 0x0000000000455acb in runtime.mcall ()
#8 0x000000c820021500 in ?? ()
#9 0x0000000000000000 in ?? ()
$ pstack 5096
Thread 1 (process 5096):
#0 0x0000000000459243 in runtime.futex ()
#1 0x0000000000427143 in runtime.futexsleep ()
#2 0x000000000040d3f4 in runtime.notesleep ()
#3 0x000000000042f6eb in runtime.stopm ()
#4 0x000000000042fff7 in runtime.startlockedm ()
#5 0x0000000000431147 in runtime.schedule ()
#6 0x000000000043139b in runtime.park_m ()
#7 0x0000000000455acb in runtime.mcall ()
#8 0x000000c820020000 in ?? ()
Update for gdb 8.0
pstack that uses gdb 8.0 correctly prints backtraces for all threas. The command looks like:
$ GDB=$HOME/bin/gdb pstack $(pidof main)
And here is its output (shortened):
$ GDB=$HOME/bin/gdb pstack $(pidof main) | egrep "^Thread"
Thread 4 (LWP 18335):
Thread 3 (LWP 18334):
Thread 2 (LWP 18333):
Thread 1 (LWP 18332):
When you're passing LWP/thread id to pstack you get a stack of that thread only. Try to pass a PID of the process to pstack and you'll get stacks of all its threads. You may get a PID or Tgid (thread group id) of the process: cat /proc/13858/status | grep Tgid. To get all LWPs created by your process you may run ps -L <PID>
Is it possible for me to track what function/code a thread is executing when debugging with gdb.I do the following
$gdb
$gdb attach pid
$info threads
27 Thread 0x7f0d3a0fe700 (LWP 3418) "BgHangManager" 0x00007f0d4fd9b0af in pthread_cond_wait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
26 Thread 0x7f0d389ff700 (LWP 3419) "gdbus" 0x00007f0d4f098b3d in poll () from /lib64/libc.so.6
25 Thread 0x7f0d381fe700 (LWP 3420) "gmain" 0x00007f0d4f098b3d in poll () from /lib64/libc.so.6
24 Thread 0x7f0d36fff700 (LWP 3422) "Timer" 0x00007f0d4fd9b458 in pthread_cond_timedwait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
23 Thread 0x7f0d3ee52700 (LWP 3423) "Cache2 I/O" 0x00007f0d4fd9b0af in pthread_cond_wait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
$bt
Only gives me the stack-trace of the currently executing code while I would like to
for know what code is exuting in another thread for instance thread 26.
Worked around it by switching to the "wanted" thread and printing the stack-trace
all
I am tring to use gdb to debug a program with 24 threads in Fedora, and I have the following output of GDB. When I want to switch to the specific thread that a segmentation fault happens, I cannot switch to that thread with the thread command (last 4 lines in the GDB output). Could you please help me with it? And do you know what error it is?
The GDB output is as follows:
[root#localhost nameComponentEncoding]# gdb NCE_david
GNU gdb (GDB) Fedora (7.2.90.20110429-36.fc15)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /mnt/disk2/experiments_BLOODMOON/two_stage_bloom_filter/programs/nameComponentEncoding/NCE_david
(gdb) r
Starting program:
[Thread debugging using libthread_db enabled]
[New Thread 0x7fffd2bf5700 (LWP 11538)]
[New Thread 0x7fffd23f4700 (LWP 11539)]
[New Thread 0x7fffd1bf3700 (LWP 11540)]
[New Thread 0x7fffd13f2700 (LWP 11541)]
[New Thread 0x7fffd0bf1700 (LWP 11542)]
[New Thread 0x7fffd03f0700 (LWP 11543)]
[New Thread 0x7fffcfbef700 (LWP 11544)]
[New Thread 0x7fffcf3ee700 (LWP 11545)]
[New Thread 0x7fffcebed700 (LWP 11546)]
[New Thread 0x7fffce3ec700 (LWP 11547)]
[New Thread 0x7fffcdbeb700 (LWP 11548)]
[New Thread 0x7fffcd3ea700 (LWP 11549)]
[New Thread 0x7fffccbe9700 (LWP 11550)]
[New Thread 0x7fffcc3e8700 (LWP 11551)]
[New Thread 0x7fffcbbe7700 (LWP 11552)]
[New Thread 0x7fffcb3e6700 (LWP 11553)]
[New Thread 0x7fffcabe5700 (LWP 11554)]
[New Thread 0x7fffca3e4700 (LWP 11555)]
[New Thread 0x7fffc9be3700 (LWP 11556)]
[New Thread 0x7fffc93e2700 (LWP 11557)]
[New Thread 0x7fffc8be1700 (LWP 11558)]
[New Thread 0x7fffc83e0700 (LWP 11559)]
[New Thread 0x7fffc7bdf700 (LWP 11560)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd2bf5700 (LWP 11538)]
compare (__str="art", this=0xbf934d0) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/
2175 const size_type __size = this->size();
(gdb) thread 11538
Thread ID 11538 not known.
(gdb) thread 0x7fffd2bf5700
Thread ID -759212288 not known.
GDB has its own numbering for threads. Do something like:
(gdb) info threads
Id Target Id Frame
<... snip ...>
2 Thread 0x7ffff7fc1700 (LWP 11538) ...
* 1 Thread 0x7ffff7fc2740 (LWP 11537) ...
# thread with PID 11538 is GDB thread number 2
(gdb) thread 2
From
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd2bf5700 (LWP 11538)]
compare (__str="art", this=0xbf934d0) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/
2175 const size_type __size = this->size();
It does seem that the segfault is caused by thread 11538. Using pthread_setname_np() to name your threads would make debugging in GDB easier.
My guess would be that this, the C++ instance variable, was NULL at that point (easily verifiable) and your case is a "use after free" error.