I've been tracking a bug triggered at the launch of my program. Here is the backtrace provided by gdb:
(gdb) bt
#0 0xb753f571 in llvm::cl::parser<llvm::FunctionPass* (*)()>::getOption(unsigned int) const ()
from ./libgdl.so
#1 0xb79aeab4 in llvm::cl::generic_parser_base::findOption(char const*) ()
from ./libgdl.so
#2 0xb753f679 in llvm::RegisterPassParser<llvm::RegisterRegAlloc>::NotifyRemove(char const*) ()
from ./libgdl.so
#3 0xaf35f0b6 in llvm::MachinePassRegistry::Add(llvm::MachinePassRegistryNode*) () from /usr/lib
/i386-linux-gnu/libLLVM-3.1.so.1
#4 0xaef42b16 in ?? () from /usr/lib/i386-linux-gnu/libLLVM-3.1.so.1
#5 0xb7fece9b in ?? () from /lib/ld-linux.so.2
In fact, the crash is due to the system using LLVM 3.1 ( for graphics related task ) while I'm using LLVM 3.0, which is embedded in my program ( libgdl.so ):
When libLLVM-3.1.so.1 wants to call the NotifyRemove function, the call is forwarded to my version of LLVM in libgdl.so and it lead to the crash as the version are incompatible.
Is there any way to prevent such a mess?
Related
I have an application which was running fine for last 15 days and below functions were getting called multiple times, but it crashed today in fopen. I have pasted the bt below, can someone please advice what might have happened wrong, from backtrace it doesn't seems to be a memory corruption as all thread data and stack variables look good. Can it be related to some bug in RHEL 5.x
>>(gdb) bt
>>#0 0x00fe4410 in __kernel_vsyscall ()
>>#1 0x0057ab10 in raise () from /lib/libc.so.6
>>#2 0x0057c421 in abort () from /lib/libc.so.6
>>#3 0x005b367b in __libc_message () from /lib/libc.so.6
>>#4 0x005bc8bd in _int_malloc () from /lib/libc.so.6
>>#5 0x005be247 in malloc () from /lib/libc.so.6
>>#6 0x005aa8ef in __fopen_internal () from /lib/libc.so.6
>>#7 0x005aa9bc in fopen##GLIBC_2.1 () from /lib/libc.so.6
>>#8 0x0811cbff in file_timer_expiry (p_mod_ctx=0xb07e4c8, p_timer_ctx=0x7ce78368)
>>#9 0x08117c33 in timer_handler (timerId=0xad54aa50, p_timer_info=0x7ce78368, p_module_context=0xb07e4c8)
>>#10 0x08397b43 in ProcessTimerTable (vc=0xae6edb8, nw=0xa89fd380)
>>#11 0x0839974c in Schedule (nw=0xa89fd380, f=0x832027e <BaseUpdate>, ctxt=0x9955e98)
>>#12 0x080730a1 in DriverWhile (p_info=0x95f68c8, W=0x84a698c, policy=2 '\002')
>>#13 0x080732e1 in start_id (args=0x95f68c8)
>>#14 0x006e7912 in start_thread () from /lib/libpthread.so.0
>>#15 0x0062747e in clone () from /lib/libc.so.6
>>#16 0x00000000 in ?? ()
A crash inside malloc implementation is (in 99.99% of cases) a result of heap corruption.
It is likely that your program has printed a message, similar to
glibc detected ./a.out: double free or corruption (!prev): 0x0000000000c6ed50
to the terminal on which it ran.
To find heap corruption, use Valgrind or (better) Address Sanitizer (supported by recent versions of GCC and Clang).
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
I got a problem when starting Matlab 2011b on ubuntu platform running a server with Intel Xeon processor.
I installed this version of matlab with network lincese, and installation process was quite well done.(I hope so, but there is no extra warning).
When I start matlab with binary in
/usr/local/MATLAB/R2011b/bin/glnx86 name MATLAB
program dump a core-dump and received SIGSEGV.
I got a backtrace using gdb as follow.
#0 0xb7feb2b6 in ?? () from /lib/ld-linux.so.2
#1 0xb7ff0dba in ?? () from /lib/ld-linux.so.2
#2 0xb7feccbf in ?? () from /lib/ld-linux.so.2
#3 0xb7ff07e4 in ?? () from /lib/ld-linux.so.2
#4 0xb70e6be9 in ?? () from /lib/i386-linux-gnu/libdl.so.2
#5 0xb7feccbf in ?? () from /lib/ld-linux.so.2
#6 0xb70e733a in ?? () from /lib/i386-linux-gnu/libdl.so.2
#7 0xb70e6c97 in dlopen () from /lib/i386-linux-gnu/libdl.so.2
#8 0xb7f330e6 in utLoadLibrary () from /usr/local/MATLAB/R2011b/bin/glnx86/libut.so
#9 0xb2b2a1bc in ?? () from /usr/local/MATLAB/R2011b/bin/glnx86/libmwbinder.so
#10 0xb2b2a412 in Binder::_load_libs(std::vector<std::string,
std::allocator<std::string> > const&)
() from /usr/local/MATLAB/R2011b/bin/glnx86/libmwbinder.so
#11 0xb2b2bd48 in Binder::_load_and_resolve() ()
from /usr/local/MATLAB/R2011b/bin/glnx86/libmwbinder.so
#12 0xb2abf356 in ?? () from /usr/local/MATLAB/R2011b/bin/glnx86/libmwblas.so
#13 0xb2abf484 in ?? () from /usr/local/MATLAB/R2011b/bin/glnx86/libmwblas.so
#14 0xb2abfd5d in zdotu_ () from /usr/local/MATLAB/R2011b/bin/glnx86/libmwblas.so
#15 0xb2ac23d0 in ?? () from /usr/local/MATLAB/R2011b/bin/glnx86/libmwblas.so
#16 0xb2ab82d8 in _init () from /usr/local/MATLAB/R2011b/bin/glnx86/libmwblas.so
#17 0xb7fece39 in ?? () from /lib/ld-linux.so.2
#18 0xb7fecf84 in ?? () from /lib/ld-linux.so.2
#19 0xb7fdf20f in ?? () from /lib/ld-linux.so.2
=========================================================
Any comment? or help?
Any word with your attention will be appreciated. Thanks.
You shouldn't start MATLAB directly from your architecture-specific directory. Try running /usr/local/MATLAB/R2011b/bin/matlab instead.The script performs some initializations and is platform-aware. The initialization process is needed because MATLAB uses quite specific (older) versions of some libraries.
If your problem persists, though, I'd contact customer service of The Mathworks.
Im building a shared library on linux. the library ".so" was sucessfully created, but when I tried to link it to a test application (with an empty main) and run the executable I got a segmentation error : "Segmentation error (cure dumped)"
when I tried to debug it with gdb and check the backtrace I got this output:
Program received signal SIGSEGV, Segmentation fault.
0x0073d5df in std::_Rb_tree_decrement(std::_Rb_tree_node_base*) () from /usr/lib/libstdc++.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12.1-4.i686 libgcc-4.4.5-2.fc13.i686 libstdc++-4.4.5-2.fc13.i686 zlib-1.2.3-23.fc12.i686
(gdb) backtrace
#0 0x0073d5df in std::_Rb_tree_decrement(std::_Rb_tree_node_base*) () from /usr/lib/libstdc++.so.6
#1 0x0012d70c in ?? () from /opt/cuda/lib/libcudart.so.3
#2 0x0012df0c in ?? () from /opt/cuda/lib/libcudart.so.3
#3 0x0012c88a in ?? () from /opt/cuda/lib/libcudart.so.3
#4 0x00121435 in __cudaRegisterFatBinary () from /opt/cuda/lib/libcudart.so.3
#5 0x005d7bfd in __sti____cudaRegisterAll_55_tmpxft_00000fe6_00000000_26_MonteCarloPaeo_SM10_cpp1_ii_3a8af011()
() from libsharedCUFP.so
#6 0x005db40d in __do_global_ctors_aux () from libsharedCUFP.so
#7 0x005a8748 in _init () from libsharedCUFP.so
#8 0x008abd00 in _dl_init_internal () from /lib/ld-linux.so.2
#9 0x0089d88f in _dl_start_user () from /lib/ld-linux.so.2
Im not familiar with gdb debugging, and it's the first time Im trying to build a shared library on Linux, but it seems to me that it has something to do with the library dynamic linking.
If anyone had any idea about this error and could help me, I would be grateful.
It doesn't have anything to do with dynamic linking or shared libraries - one of the constructors in libsharedCUFP.so (I assume this is your shared library) is most probably passing an illegal address to a function in libcudart.so which crashes.
You simply need to debug your code.
I am working on multhithreading application when ever the process dump it always generates core as shown below i am not able to understand where it is actually dumping.
GNU gdb Red Hat Linux (6.5-25.el5rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: exec file is newer than core file.
Core was generated by `multithreadprocess '.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000448f7a in std::ostream::operator<< ()
(gdb) where
0x000000000044bd32 in std::ostream::operator<< ()
#1 0x0000000000450b21 in std::ostream::operator<< ()
#2 0x000000000042eda9 in std::string::operator= ()
#3 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#4 0x00000030576ce3bd in clone () from /lib64/libc.so.6
(gdb)thread apply all bt
Thread 6 (process 11674):
#0 0x000000305820a687 in pthread_cond_timedwait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000431140 in std::string::operator= ()
#2 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000030576ce3bd in clone () from /lib64/libc.so.6
Thread 5 (process 11683):
#0 0x000000305820cbfb in write () from /lib64/libpthread.so.0
#1 0x0000000000449151 in std::ostream::operator<< ()
#2 0x000000000043b74a in std::string::operator= ()
#3 0x000000000046c3f4 in std::string::substr ()
#4 0x000000000046e3c1 in std::string::substr ()
#5 0x00000000004305a4 in std::string::operator= ()
#6 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#7 0x00000030576ce3bd in clone () from /lib64/libc.so.6
Thread 4 (process 11744):
#0 0x00000030576c5896 in poll () from /lib64/libc.so.6
#1 0x0000000000474f1c in std::string::substr ()
#2 0x000000000043b889 in std::string::operator= ()
#3 0x0000000000474dbc in std::string::substr ()
#4 0x00000000004306a5 in std::string::operator= ()
#5 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#6 0x00000030576ce3bd in clone () from /lib64/libc.so.6
Thread 3 (process 11864):
#0 0x000000305820a687 in pthread_cond_timedwait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000431140 in std::string::operator= ()
#2 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000030576ce3bd in clone () from /lib64/libc.so.6
Thread 2 (process 11866):
#0 0x000000305820a687 in pthread_cond_timedwait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000431140 in std::string::operator= ()
#2 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000030576ce3bd in clone () from /lib64/libc.so.6
Thread 1 (process 11865):
#0 0x000000000044bd32 in std::ostream::operator<< ()
#1 0x0000000000450b21 in std::ostream::operator<< ()
#2 0x000000000042eda9 in std::string::operator= ()
#3 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
#4 0x00000030576ce3bd in clone () from /lib64/libc.so.6
If i give bt full it is showing like this
(gdb) bt full
#0 0x000000000044bd32 in std::ostream::operator<< ()
No symbol table info available.
#1 0x0000000000450b21 in std::ostream::operator<< ()
No symbol table info available.
#2 0x000000000042eda9 in std::string::operator= ()
No symbol table info available.
#3 0x00000030582062e7 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x00000030576ce3bd in clone () from /lib64/libc.so.6
No symbol table info available.
GDB 6.5 is quite old. You will likely get significantly better stack traces from (current) GDB 7.0.1.
You also appear to be trying to debug optimized code, built without -g flag, and you may not be debugging the right executable (GDB warns that your executable is newer than your core).
Make sure that your executable and all the libraries listed in info shared GDB output exactly match between the system where your core was produced and the system on which you are analyzing the core (if they are not the same) -- this is paramount -- if there is a mismatch, you'll likely get bogus stack traces (and the stack traces you've posted do look completely bogus to me).
Looks to me like you're using iostream inside a multithreaded application without the appropriate flags. See this. In particular, note that it says
When you build an application that
uses the iostream classes of the libC
library to run in a multithreaded
environment, compile and link the
source code of the application using
the -mt option. This option passes
-D_REENTRANT to the preprocessor and -lthread to the linker.
This is for a particular platform; your requirements may vary.
This is on a Redhat EL5 machine w/ a 2.6.18-164.2.1.el5 x86_64 kernel using gcc 4.1.2 and gdb 7.0.
When I run my application with gdb and break in while it's running, several of my threads show the following call stack when I do a backtrace:
#0 0x000000000051d7da in pthread_cond_wait ()
#1 0x0000000100000000 in ?? ()
#2 0x0000000000c1c3b0 in ?? ()
#3 0x0000000000c1c448 in ?? ()
#4 0x00000000000007dd in ?? ()
#5 0x000000000051d630 in ?? ()
#6 0x00007fffffffdc90 in ?? ()
#7 0x000000003b1ae84b in ?? ()
#8 0x00007fffffffdd50 in ?? ()
#9 0x0000000000000000 in ?? ()
Is this a symptom of a common problem?
Is there a known issue with viewing the call stack while waiting on a condition?
The problem is that pthread_cond_wait is written in hand-coded assembly, and apparently doesn't have proper unwind descriptor (required on x86_64 to unwind the stack) in your build of glibc. This problem may have recently been fixed here.
You can try to build and install the latest glibc (note: if you screw up installation, your machine will likely become unbootable; approach with extreme caution!), or just live with "bogus" stack traces from pthread_cond_wait.
Generally, synchronization is required when multiple threads share a single resource.
In such a case, when you interrupt the program, you'll see only 1 thread is running (i.e., accessing the resource) and other threads are waiting within pthread_cond_wait().
So I don't think pthread_cond_wait() itself is problematic.
If your program hangs with deadlock or performance doesn't scale, it might be caused by pthread_cond_wait().
That looks like a corrupt stack trace to me
for example:
#9 0x0000000000000000 in ?? ()
There shouldn't be code at NULL