I have the following program:
#pragma omp parallel num_threads(cfg.max_parallel_threads) private(pos,row,info)
{
int threadID = omp_get_thread_num();
#pragma omp for
for (pos = 0;pos < (end - start);pos++)
{
/*some code to define row,B,P diffts and X, L_sim,work*/
dggglm_(&row, &col, &row, B[threadID], &row, P[threadID], &row, diffts[threadID], X[threadID], L_sim[threadID], work[threadID], &lwork, &info);
}
}
dggglm_ is a function of LAPACK. This program occasionally gets stuck (probably when there is a high cpu or memory usage, but not always). For the same data, it sometimes runs OK but gets stuck on another run (randomly).
Here is the output of pstack on a child process which seems to be the problem:
Thread 115 (Thread 0x7fcd81bdf700 (LWP 91874)):
#0 0x00007fcd8a078fa6 in ATL_dscal_xp1yp0aXbX () from /usr/lib64/atlas/libatlas.so.3
#1 0x00007fcd8ae22666 in dlarfp_ () from /usr/lib64/atlas/liblapack.so.3
#2 0x00007fcd8adb1a61 in dgeqr2_ () from /usr/lib64/atlas/liblapack.so.3
#3 0x00007fcd8adb1e06 in dgeqrf_ () from /usr/lib64/atlas/liblapack.so.3
#4 0x00007fcd8add9055 in dggqrf_ () from /usr/lib64/atlas/liblapack.so.3
#5 0x00007fcd8add71ae in dggglm_ () from /usr/lib64/atlas/liblapack.so.3
#6 0x000000000041d4ae in sbas::sbas_step2_sbas_linear_new ()
#7 0x0000003b0160e0d5 in ?? () from /usr/lib64/libgomp.so.1
#8 0x0000003af3207aa1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003af2ee8c4d in clone () from /lib64/libc.so.6
The output of ps -eLo pid,lwp,pcpu | grep 91757
91757 91869 10.7
91757 91870 9.7
91757 91871 9.0
91757 91872 12.0
91757 91873 9.2
91757 91874 41.6
91757 91875 17.7
91757 91876 9.2
91757 91877 8.7
91757 91878 12.5
91757 91880 9.0
You can see the child process 91874 takes a long time and is still running. It seem that lapack went into an endless loop. Can someone suggest a way to debug this?
Thanks.
Related
stack trace of the thread show nothing except __nanosleep_nocancel from the core dump'ed using gdb on Debian. This is been observed when analyzing the threads stack trace from the coredump generated by the kernel which is triggered from the application when anomaly found
Thread 5 (Thread 0x7f8b307bf700 (LWP 27000)):
#0 ......Application function .....
#1......Application function .....
#2 ......Application function .....
#3 ......Application function .....
#4 0x00007f8b303c9494 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007f8b2f666aff in __libc_ifunc_impl_list () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f8b30685700 (LWP 27025)):
#0 0x00007f8b303d27dd in __nanosleep_nocancel () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f8b2eb31700 (LWP 27032)):
#0 0x00007f8b303d27dd in __nanosleep_nocancel () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f8b306c3700 (LWP 27022)):
#0 0x00007f8b303d2f9f in raise () from /lib/x86_64-linux- gnu/libpthread.so.0
Here thread 2 and 3's stack trace showing __nanosleep_nocancel , where I expect stack trace be like thread 5.
any leads on this would be greatly appreciated.
I'm debugging a OpenCV app compiled with C++11 (I use OpenCV 2.4.10). The app has two threads that do some image processing on the CPU (no GPU functions used but I also included libopencv_gpu.so in the linked libraries).
Using gdb I noticed that instead of just two threads (the main process thread and another thread created by the main process thread) I found 3 threads running:
(gdb) info threads
Id Target Id Frame
78 Thread 0x7fffe2ff5700 (LWP 20531) "app_name" 0x00007ffff5bb2f3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
2 Thread 0x7fffe3c42700 (LWP 20454) "app_name" 0x00007ffff5bdf12d in poll () at ../sysdeps/unix/syscall-template.S:81
* 1 Thread 0x7ffff7fab800 (LWP 20450) "app_name" 0x00007ffff5bb2f3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
Thread 1 and 78 (using gdb ID) are executing my code. I added a sleep call in each one so I can make sure that those are my threads.
Thread 2 (using gdb ID) is created before entering the main function of the main process I believe. As far as I could debug this, thread with ID 2 just calls poll() function all the time.
I'm new to gdb and maybe you can tell me how to find out who creates this thread and what is it's purpose? Is this OpenCV related or C++11? When I compile the same app using Opencv4Tegra and run it on a Tegra K1 board, thread number 2 does not exist.
EDIT
This is the backtrace when creating thread number 2. It seems that libusb creates this but I don't know why yet:
(gdb) backtrace
#0 __pthread_create_2_1 (newthread=0x7fffea79c438, attr=0x0, start_routine=0x7fffea5941c0, arg=0x0) at pthread_create.c:466
#1 0x00007fffea5943df in ?? () from /lib/x86_64-linux-gnu/libusb-1.0.so.0
#2 0x00007fffea5926a5 in ?? () from /lib/x86_64-linux-gnu/libusb-1.0.so.0
#3 0x00007fffea58b715 in libusb_init () from /lib/x86_64-linux-gnu/libusb-1.0.so.0
#4 0x00007ffff2f06a0e in ?? () from /usr/lib/x86_64-linux-gnu/libdc1394.so.22
#5 0x00007ffff2ef5465 in dc1394_new () from /usr/lib/x86_64-linux-gnu/libdc1394.so.22
#6 0x00007ffff6f615e9 in CvDC1394::CvDC1394() () from /usr/local/lib/libopencv_highgui.so.2.4
#7 0x00007ffff6f373f0 in _GLOBAL__sub_I_cap_dc1394_v2.cpp () from /usr/local/lib/libopencv_highgui.so.2.4
#8 0x00007ffff7dea13a in call_init (l=<optimized out>, argc=argc#entry=3, argv=argv#entry=0x7fffffffdcd8, env=env#entry=0x7fffffffdcf8) at dl-init.c:78
#9 0x00007ffff7dea223 in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:36
#10 _dl_init (main_map=0x7ffff7ffe1c8, argc=3, argv=0x7fffffffdcd8, env=0x7fffffffdcf8) at dl-init.c:126
#11 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
(gdb) quit
I'm currently debugging kernel code using KGDB.
Whenever I break in I naturally jump to the interrupt handler for kgdb.
Under GDB I ran the following command.
info threads
and the output would be
7 Thread 7 (rcu_sched) 0x0000000000000000 in irq_stack_union ()
6 Thread 5 (kworker/0:0H) 0x0000000000000000 in irq_stack_union ()
5 Thread 3 (ksoftirqd/0) 0x0000000000000000 in irq_stack_union ()
4 Thread 2 (kthreadd) 0x0000000000000000 in irq_stack_union ()
3 Thread 1 (init) 0x0000000000000000 in irq_stack_union ()
2 Thread 3754 (Xorg) 0x0000000000000000 in irq_stack_union ()
* 1 Thread 4294967294 (shadowCPU0) kgdb_breakpoint ()
at kernel/debug/debug_core.c:1042
I would then jump through the code expecting to end up in a different thread (I'm interested in Xorg) however after i step through the code the next executing thread becomes cpu idle.
info thread
* 1 Thread 4294967294 (shadowCPU0) cpu_idle_loop () at kernel/cpu/idle.c:116
How can I switch my debug context to Xorg or any other thread, additionally what does irq_stack_union () mean. Thread is idle pending interrupts?
According to the offical documentation is is just thread threadno
https://sourceware.org/gdb/onlinedocs/gdb/Threads.html
I have a multithreaded program that I'm trying to debug. When I run info thread in GDB, I get the following:
(gdb) info thread
Id Target Id Frame
8 Thread 0x7fffe77fd700 (LWP 17425) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
7 Thread 0x7fffe73fc700 (LWP 17426) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
6 Thread 0x7fffe7fff700 (LWP 17423) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
5 Thread 0x7fffe7bfe700 (LWP 17424) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
* 4 Thread 0x7ffff4810700 (LWP 17422) "unittest" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
3 Thread 0x7ffff4c11700 (LWP 17421) "receiver" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
2 Thread 0x7ffff5a3b700 (LWP 17420) "unittest" 0x00007ffff634e553 in select () from /usr/lib/libc.so.6
1 Thread 0x7ffff7fc9780 (LWP 17419) "unittest" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
It would be excellent if I could make GDB display the parent/child relationships between the threads, something like the following:
(gdb) info thread
Id Target Id Frame
1 Thread 0x7ffff7fc9780 (LWP 17419) "unittest" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
3 Thread 0x7ffff4c11700 (LWP 17421) "receiver" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
8 Thread 0x7fffe77fd700 (LWP 17425) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
6 Thread 0x7fffe7fff700 (LWP 17423) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
2 Thread 0x7ffff5a3b700 (LWP 17420) "unittest" 0x00007ffff634e553 in select () from /usr/lib/libc.so.6
5 Thread 0x7fffe7bfe700 (LWP 17424) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
* 4 Thread 0x7ffff4810700 (LWP 17422) "unittest" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
7 Thread 0x7fffe73fc700 (LWP 17426) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
For example, thread 3 is the parent of threads 8, 6, and 2, and thread 1 is the parent of everything.
Does such functionality exist? I have not seen reference to it, if it does.
gdb doesn't print this information because it doesn't exist in your program -- there is no way for gdb to discover it once the threads have been created.
There are maybe two ways it could be done.
First, you could set a breakpoint on the thread-creation function and record the information. This is readily done from Python. Then you can write a new command, also in Python, to format the output the way you like.
The problem with this approach is that it won't work if you "attach" to a running program. It will be too late to capture the information.
Another method is if you have extra information available in your program that describes the hierarchy. Then you can write a new command in Python that extracts this information to display things as you like.
I am in a gdb session trying to debug a core dump with more than 200 threads.
When I do thread apply all bt in gbd, I have to press Enter key repeatedly for more threads. Its quite annoying. Is there a way I can specify in my command to do it without pressing Enter?
Thanks for any info.
EDIT:
Here is a sample output:
(gdb) thread apply all bt
Thread 409 (Thread 7505):
#0 0x00007ffff1d6961c in ?? ()
#1 0x0000000000000000 in ?? ()
...
...
...
...
<snipping out 20 some backtraces>
...
...
...
...
Thread 390 (Thread 10529):
#0 0x00007ffff1d6961c in ?? ()
#1 0x0000001300000000 in ?? ()
#2 0x00007fffe860bd50 in ?? ()
#3 0x00007fffe8464690 in ?? ()
#4 0x0000000000000014 in ?? ()
---Type <return> to continue, or q <return> to quit---
Disable the pager by using:
set height 0