I'm currently debugging kernel code using KGDB.
Whenever I break in I naturally jump to the interrupt handler for kgdb.
Under GDB I ran the following command.
info threads
and the output would be
7 Thread 7 (rcu_sched) 0x0000000000000000 in irq_stack_union ()
6 Thread 5 (kworker/0:0H) 0x0000000000000000 in irq_stack_union ()
5 Thread 3 (ksoftirqd/0) 0x0000000000000000 in irq_stack_union ()
4 Thread 2 (kthreadd) 0x0000000000000000 in irq_stack_union ()
3 Thread 1 (init) 0x0000000000000000 in irq_stack_union ()
2 Thread 3754 (Xorg) 0x0000000000000000 in irq_stack_union ()
* 1 Thread 4294967294 (shadowCPU0) kgdb_breakpoint ()
at kernel/debug/debug_core.c:1042
I would then jump through the code expecting to end up in a different thread (I'm interested in Xorg) however after i step through the code the next executing thread becomes cpu idle.
info thread
* 1 Thread 4294967294 (shadowCPU0) cpu_idle_loop () at kernel/cpu/idle.c:116
How can I switch my debug context to Xorg or any other thread, additionally what does irq_stack_union () mean. Thread is idle pending interrupts?
According to the offical documentation is is just thread threadno
https://sourceware.org/gdb/onlinedocs/gdb/Threads.html
Related
i am trying to find oldest thread in Linux core dump. So i need get start time of each thread that are running. So is there any way to get the start time of thread.
i tried to some command in gdb core file but that gives each call stack of threads but not start time.
Regards
Rabi
i am trying to find oldest thread in Linux core dump. So i need get start time of each thread that are running.
Your conclusion does not follow from your previous statement.
So is there any way to get the start time of thread.
No: that time is not recorded anywhere in the thread structure.
However, in general Linux allocates thread ids in increasing order, so sorting by thread id should give you the order you desire. For example:
(gdb) info thread
Id Target Id Frame
* 1 Thread 0x7fa91e769740 (LWP 2128629) __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
2 Thread 0x7fa91d766700 (LWP 2128632) 0x0000557aa5787165 in fn ()
3 Thread 0x7fa91af61700 (LWP 2128637) 0x0000557aa5787165 in fn ()
4 Thread 0x7fa91df67700 (LWP 2128631) 0x0000557aa5787165 in fn ()
5 Thread 0x7fa91cf65700 (LWP 2128633) 0x0000557aa5787165 in fn ()
6 Thread 0x7fa91bf63700 (LWP 2128635) 0x0000557aa5787165 in fn ()
7 Thread 0x7fa91c764700 (LWP 2128634) 0x0000557aa5787165 in fn ()
8 Thread 0x7fa91a760700 (LWP 2128638) 0x0000557aa5787165 in fn ()
9 Thread 0x7fa91b762700 (LWP 2128636) 0x0000557aa5787165 in fn ()
10 Thread 0x7fa919f5f700 (LWP 2128639) 0x0000557aa5787165 in fn ()
Here you can see that thread 1 was created first, followed by thread 4, 2, 5, 7, etc (LWP is the Linux thread id here).
stack trace of the thread show nothing except __nanosleep_nocancel from the core dump'ed using gdb on Debian. This is been observed when analyzing the threads stack trace from the coredump generated by the kernel which is triggered from the application when anomaly found
Thread 5 (Thread 0x7f8b307bf700 (LWP 27000)):
#0 ......Application function .....
#1......Application function .....
#2 ......Application function .....
#3 ......Application function .....
#4 0x00007f8b303c9494 in start_thread () from /lib/x86_64-linux- gnu/libpthread.so.0
#5 0x00007f8b2f666aff in __libc_ifunc_impl_list () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f8b30685700 (LWP 27025)):
#0 0x00007f8b303d27dd in __nanosleep_nocancel () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f8b2eb31700 (LWP 27032)):
#0 0x00007f8b303d27dd in __nanosleep_nocancel () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f8b306c3700 (LWP 27022)):
#0 0x00007f8b303d2f9f in raise () from /lib/x86_64-linux- gnu/libpthread.so.0
Here thread 2 and 3's stack trace showing __nanosleep_nocancel , where I expect stack trace be like thread 5.
any leads on this would be greatly appreciated.
I'm debugging a OpenCV app compiled with C++11 (I use OpenCV 2.4.10). The app has two threads that do some image processing on the CPU (no GPU functions used but I also included libopencv_gpu.so in the linked libraries).
Using gdb I noticed that instead of just two threads (the main process thread and another thread created by the main process thread) I found 3 threads running:
(gdb) info threads
Id Target Id Frame
78 Thread 0x7fffe2ff5700 (LWP 20531) "app_name" 0x00007ffff5bb2f3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
2 Thread 0x7fffe3c42700 (LWP 20454) "app_name" 0x00007ffff5bdf12d in poll () at ../sysdeps/unix/syscall-template.S:81
* 1 Thread 0x7ffff7fab800 (LWP 20450) "app_name" 0x00007ffff5bb2f3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
Thread 1 and 78 (using gdb ID) are executing my code. I added a sleep call in each one so I can make sure that those are my threads.
Thread 2 (using gdb ID) is created before entering the main function of the main process I believe. As far as I could debug this, thread with ID 2 just calls poll() function all the time.
I'm new to gdb and maybe you can tell me how to find out who creates this thread and what is it's purpose? Is this OpenCV related or C++11? When I compile the same app using Opencv4Tegra and run it on a Tegra K1 board, thread number 2 does not exist.
EDIT
This is the backtrace when creating thread number 2. It seems that libusb creates this but I don't know why yet:
(gdb) backtrace
#0 __pthread_create_2_1 (newthread=0x7fffea79c438, attr=0x0, start_routine=0x7fffea5941c0, arg=0x0) at pthread_create.c:466
#1 0x00007fffea5943df in ?? () from /lib/x86_64-linux-gnu/libusb-1.0.so.0
#2 0x00007fffea5926a5 in ?? () from /lib/x86_64-linux-gnu/libusb-1.0.so.0
#3 0x00007fffea58b715 in libusb_init () from /lib/x86_64-linux-gnu/libusb-1.0.so.0
#4 0x00007ffff2f06a0e in ?? () from /usr/lib/x86_64-linux-gnu/libdc1394.so.22
#5 0x00007ffff2ef5465 in dc1394_new () from /usr/lib/x86_64-linux-gnu/libdc1394.so.22
#6 0x00007ffff6f615e9 in CvDC1394::CvDC1394() () from /usr/local/lib/libopencv_highgui.so.2.4
#7 0x00007ffff6f373f0 in _GLOBAL__sub_I_cap_dc1394_v2.cpp () from /usr/local/lib/libopencv_highgui.so.2.4
#8 0x00007ffff7dea13a in call_init (l=<optimized out>, argc=argc#entry=3, argv=argv#entry=0x7fffffffdcd8, env=env#entry=0x7fffffffdcf8) at dl-init.c:78
#9 0x00007ffff7dea223 in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:36
#10 _dl_init (main_map=0x7ffff7ffe1c8, argc=3, argv=0x7fffffffdcd8, env=0x7fffffffdcf8) at dl-init.c:126
#11 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
(gdb) quit
I have a multithreaded program that I'm trying to debug. When I run info thread in GDB, I get the following:
(gdb) info thread
Id Target Id Frame
8 Thread 0x7fffe77fd700 (LWP 17425) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
7 Thread 0x7fffe73fc700 (LWP 17426) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
6 Thread 0x7fffe7fff700 (LWP 17423) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
5 Thread 0x7fffe7bfe700 (LWP 17424) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
* 4 Thread 0x7ffff4810700 (LWP 17422) "unittest" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
3 Thread 0x7ffff4c11700 (LWP 17421) "receiver" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
2 Thread 0x7ffff5a3b700 (LWP 17420) "unittest" 0x00007ffff634e553 in select () from /usr/lib/libc.so.6
1 Thread 0x7ffff7fc9780 (LWP 17419) "unittest" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
It would be excellent if I could make GDB display the parent/child relationships between the threads, something like the following:
(gdb) info thread
Id Target Id Frame
1 Thread 0x7ffff7fc9780 (LWP 17419) "unittest" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
3 Thread 0x7ffff4c11700 (LWP 17421) "receiver" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
8 Thread 0x7fffe77fd700 (LWP 17425) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
6 Thread 0x7fffe7fff700 (LWP 17423) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
2 Thread 0x7ffff5a3b700 (LWP 17420) "unittest" 0x00007ffff634e553 in select () from /usr/lib/libc.so.6
5 Thread 0x7fffe7bfe700 (LWP 17424) "SocketReader" 0x00007ffff7bcc66d in read () from /usr/lib/libpthread.so.0
* 4 Thread 0x7ffff4810700 (LWP 17422) "unittest" 0x00007ffff7bcc38c in __lll_lock_wait () from /usr/lib/libpthread.so.0
7 Thread 0x7fffe73fc700 (LWP 17426) "SocketWriter" 0x00007ffff7bc9b2f in pthread_cond_wait##GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
For example, thread 3 is the parent of threads 8, 6, and 2, and thread 1 is the parent of everything.
Does such functionality exist? I have not seen reference to it, if it does.
gdb doesn't print this information because it doesn't exist in your program -- there is no way for gdb to discover it once the threads have been created.
There are maybe two ways it could be done.
First, you could set a breakpoint on the thread-creation function and record the information. This is readily done from Python. Then you can write a new command, also in Python, to format the output the way you like.
The problem with this approach is that it won't work if you "attach" to a running program. It will be too late to capture the information.
Another method is if you have extra information available in your program that describes the hierarchy. Then you can write a new command in Python that extracts this information to display things as you like.
EX: Thread A creates Thread B.
i wish to block thread A when it starts to create Thread B.
but i do not know how to do it with GDB.
Any help would be appreciated！
Thanks
Use the GDB non-stop mode and put a breakpoint at the line after your program calls pthread_create().
Reading symbols from bad_thread...done.
(gdb) set target-async 1
(gdb) set pagination off
(gdb) set non-stop on
(gdb) b pthread_create
Breakpoint 1 at 0x400570
(gdb) r
Starting program: bad_thread
[Thread debugging using libthread_db enabled]
Breakpoint 1, 0x0000000000400570 in pthread_create#plt ()
(gdb) bt
#0 0x0000000000400570 in pthread_create#plt ()
#1 0x00000000004006d8 in main () at bad_thread.c:21
(gdb) list
12 pthread_mutex_unlock(&x_mutex);
13
14 pthread_exit(NULL);
15 }
16
17 int
18 main () {
19 pthread_t tid1, tid2;
20
21 pthread_create(&tid1, NULL, add_thread, NULL);
(gdb) b bad_thread.c:22
Breakpoint 2 at 0x4006d8: file bad_thread.c, line 22.
(gdb) b add_thread
Breakpoint 3 at 0x400694: file bad_thread.c, line 10.
(gdb) c
Continuing.
[New Thread 0x40a00960 (LWP 26829)]
Breakpoint 2, main () at bad_thread.c:22
22 pthread_create(&tid2, NULL, add_thread, NULL);
(gdb)
Breakpoint 3, add_thread (arg=0x0) at bad_thread.c:10
10 pthread_mutex_lock(&x_mutex);
bt
#0 main () at bad_thread.c:22
(gdb)
To explain the above:
Turn on non-stop mode as per GDB docs
put a breakpoint at pthread_create() to determine where the program calls this
run and hit the breakpoint
backtrace and list to find the next line of code
put a breakpoint at the next line of code and at the thread start function (add_thread, in this case)
run
Notice we hit the breakpoint at the next line of code in the original thread. The new thread continues in the background and then hits the breakpoint at the thread start function.