Now I know that gdb enables us to switch between threads executing.
But, but, for greater convenience, I'd like to know if it's possible to have as many terminal emulators open as threads in the application, and have a gdb instance in each of these emulators, each bound to a particular thread?
and have a gdb instance in each of these emulators, each bound to a
particular thread
You can't attach multiple gdb instances to the same process. This is a limitation of ptrace syscall used by gdb. From man ptrace:
EPERM The specified process cannot be traced. This could be because
the tracer has insufficient privileges (the required
capability is CAP_SYS_PTRACE); unprivileged processes cannot
trace processes that they cannot send signals to or those
running set-user-ID/set-group-ID programs, for obvious
reasons. Alternatively, the process may already be being
traced, or (on kernels before 2.6.26) be init(1) (PID 1).
Related
How can I trace the threads / systems calls in a process if the process id is known?
For system calls, you can use strace(1) and it supports attaching to live processes. You just need to figure out the process/thread id.
I am trying to create an application in userspace that sets affinity of processes. I would like the program to be triggered immediately every time a new pid/tid is spawned by the kernel. I am attempting to write to a file node under /proc from the do_fork() method in the kernel but I feel that it may have too much overhead.
Does anyone know any alternatives to detect a new process creation immediately after it is spawned?
If monitoring do_fork() is the way to go, would a call back to an userspace program via a system call be faster that using a fs node to communicate?
Forkstat is a program that logs process fork() [among other things]
Install it:
$ sudo apt-get install forkstat
Use it to log "fork" events:
$ forkstat -e fork
Use a socket with NETLINK_CONNECTOR. The kernel will tell you about process events, including fork()s and exec()s. You must have CONFIG_CONNECTOR and CONFIG_PROC_EVENTS enabled in your kernel.
Here's a related question with more details:
Detect launching of programs on Linux platform
For a complete socket NETLINK_CONNECTOR example, see:
http://bewareofgeek.livejournal.com/2945.html
As an aside, Inotify doesn't work. It will not work on /proc/ to detect new processes:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/454722
execsnoop can be a good alternative to show new processes and arguments.
From Wikipedia it says:
A kernel thread is the "lightest" unit of kernel scheduling. At least one kernel thread exists within each process.
I've learned that a process is a container that houses memory space, file handles, device handles, system resources, etc... and the thread is the one that really gets scheduled by the kernel.
So in single-threaded applications, is that one thread(main thread i believe) a kernel thread?
I assume you are talking about this article:
http://en.wikipedia.org/wiki/Kernel_thread
According to that article, in a single threaded application, since you have only one thread by definition, it has to be a kernel thread, otherwise it will not get scheduled and will not run.
If you had more than one thread in your application, then it would depend on how user mode multi threading is implemented (kernel threads, fibers, etc ...).
It's important to note however it would be a kernel thread running in user mode, when executing the application code (unless you make a system call). Any attempt to execute a protected instruction when running in user mode would cause a fault that will eventually lead to the process being terminated.
So kernel thread here not to be confused with supervisor/privileged mode and kernel code.
You can execute kernel code, but you have to go through a system call gate first.
No. In modern operating systems applications and the kernel run at different processor protection levels (often called rings). For example, Intel CPUs have four protection levels. Kernel code runs at Ring 0 (kernel mode) and is able to execute the most privileged processor instructions, whereas application code runs at Ring 3 (user mode) and is not allowed to execute certain operations. See http://en.wikipedia.org/wiki/Ring_(computer_security)
Is there a simple way of setting the affinity of the application I'm debugging without locking gdb to the same core? The reason why I'm asking is that the application is running with real time priority and it needs to run on a single core. At the moment I use this command line
taskset -c 3 gdbserver :1234 ./app.out
but the application stops responding and freezes the gdb server, making debugging impossible. I suspect that the real time priority of the application prevents gdb from executing. If I start the application and then start gdb without affinity setting, then I can attach and debug the application without gdb freezing.
Is there a simple way to start gdb and the application with different affinities? Or preferably: Is there a gdb command to set affinity of the child process?
I found a solution: Use the --wrapper argument.
http://sourceware.org/gdb/onlinedocs/gdb/Server.html
gdbserver --wrapper taskset -c 3 -- :1234 ./app.out
I had a similar problem and found a solution for myself, drawing in fact from your question for inspiration. As you suspect, it is possible that your gdbserver freezes when running on the same core because one of your application threads is using all the core's cycles and gdbserver is not allowed to run because its priority is too low.
For my particular needs, I am using gdb for an application employing realtime scheduling running on the local machine, and I don't mind if gdb runs on the same core, but I want to be able to debug my program with all the priorities of the application threads respected. For me, what made things work was expanding the gdb command to this more complex construction:
taskset -c 3 chrt 99 gdb
The chrt command added to your taskset command switches to the SCHED_RR policy and runs gdb at the specified priority. My threads being debugged run themselves with lesser priority, and so I assume they only run when gdb is not running.
I was having a problem before. I think, when I requested gdb to resume execution after gdb had suspended execution at a breakpoint or so, that one thread would start running before a higher-priority thread was resumed and thus it was not always the thread that I expected to be running that was in fact running. For me, the above command seems to fix everything -- I assume because the application threads can only run when gdb has finished everything it needs to do in order to resume the program. So, I guess the command line that would be applicable in your case if you wanted to try this out would be:
taskset -c 3 chrt 99 gdbserver :1234 ./app.out
Note: so this would lock gdbserver to a certain core, but likely your real time threads would (or probably could) be running at a lesser priority and so gdbserver would not freeze on you.
I'm trying to understand how Linux handles process scheduling and thread scheduling. I read that Linux can schedule both processes and threads.
Does Linux have a thread scheduler AND a process scheduler? If yes, how do they cooperate?
The Linux kernel scheduler is actually scheduling tasks, and these are either threads or (single-threaded) processes.
So a task (a task_struct inside the kernel), in the context of the scheduler, is the thing being scheduled, and can be some kernel thread like kworker or kswapd, some user thread of a multi-threaded process (like firefox), or the single-thread of a single-threaded process (like bash), identified with that single-threaded process.
A process is a non-empty finite set (sometimes a singleton) of threads sharing the same virtual address space (and other things like file descriptors, working directory, etc etc...). See also credentials(7), capabilities(7) etc....
Threads on Linux are kernel threads (in the sense of being managed by the kernel, which also creates its own threads), created by the Linux specific clone syscall (which can also be used to create processes on Linux). The pthread_create function is probably built (on Linux) above clone inside NPTL and Gnu Libc (which integrated NPTL on Linux) and musl-libc.
Kernel threads under Linux are implemented as processes that share resources. The scheduler does not differentiate between a thread and a process
See here for more information:
http://www.linuxquestions.org/linux/articles/Technical/Linux_Kernel_Thread
Under LINUX there is no concept of threads,to make LINUX POSIX complaint thread is nothing but another process.when you try to get a process id it would display the leader process id under any thread. For more details try to refer this book "Understanding LINUX kernel".Hope