I execute the following strace command with the intention of getting data about PID 13221
strace -fF -tT -all -o abc.txt -p 13221
However when the command executes and finishes I get output like below :
Process 13221 attached with 12 threads - interrupt to quit
Process 13252 attached
Process 13253 attached (waiting for parent)
Process 13253 resumed (parent 13252 ready)
Process 13252 suspended
Process 13252 resumed
Process 13253 detached
Process 13252 detached
Process 13232 detached
Process 13228 detached
Process 13225 detached
Process 13222 detached
Process 13221 detached
What are these extra PID's ? Are these the children of 13221 ? Who is creating them ?
Thanks.
What are these extra PID's ? Are these the children of 13221 ?
It must have been threads of your program. You have used "-f" in strace and this is why threads are also monitored.
How to know which of these are threads
If you run ls /proc/<PID>/task for your process you will get PIDs of all threads in your process (including a PID of the main thread).
It is simpler to do when you need to get thread PIDs comparing with running pstack for the same process. pstack is actually a gdb script, it stops a process when attaches. So it is simpler just to run ls /proc/<PID>/task
Related
I have the following process tree
test1.sh
\- test2.sh
\- sleep 600
Normally If I kill the test1.sh process, the child processes test2.sh and sleep 600 will continue running. But If I suspend the sleep 600 process through send signal (SIGSTOP or SIGTSTP), and then kill the test1.sh process, the child test2.sh and sleep 600 will exit. Why?
Here is my test program:
test1.sh
#!/bin/sh
./test2.sh
test2.sh
#!/bin/sh
sleep 600
Test steps:
run test1.sh
$ ./test1.sh
open new console and suspend the child process.
$ kill -19 < sleep pid > or kill -20 < sleep pid >
kill the parent process test1.sh
$ kill < test1.sh pid >
You will find the after step3, the test2.sh and sleep 600 exited.
Bug if I only run step1 and step3, ignore step2, the test2.sh and sleep 600 process will not exit.
Can anyone explain this? Many thanks.
When you are killing process test1.sh, you leave test2.sh orphan so you need to know what happens with orphan processes in your Operating System.
When process test2.sh is running and his parent dies, the OS moves it to the init process and keeps its execution. So the result is both, test2.sh and sleep processes are still up even if you have killed test1.sh.
When process sleep is stopped (signal 20) and his parent dies, the OS tries to move it to the init process. However, since the process is stopped and there will no longer be any tty capable of resuming it (since its parent has died), the OS may decide to do other things with the process. In your case, it dies with SIGKILL to avoid the problem of many stopped, orphaned processes lying around the system. Since the sleep process have exited, the test2.sh process ends too.
From the GNU man page:
While a process is stopped, no more signals can be delivered to it
until it is continued, except SIGKILL signals and (obviously) SIGCONT
signals. The signals are marked as pending, but not delivered until
the process is continued. The SIGKILL signal always causes termination
of the process and can’t be blocked, handled or ignored. You can
ignore SIGCONT, but it always causes the process to be continued
anyway if it is stopped. Sending a SIGCONT signal to a process causes
any pending stop signals for that process to be discarded. Likewise,
any pending SIGCONT signals for a process are discarded when it
receives a stop signal.
When a process in an orphaned process group (see Orphaned Process
Groups) receives a SIGTSTP, SIGTTIN, or SIGTTOU signal and does not
handle it, the process does not stop. Stopping the process would
probably not be very useful, since there is no shell program that will
notice it stop and allow the user to continue it. What happens instead
depends on the operating system you are using. Some systems may do
nothing; others may deliver another signal instead, such as SIGKILL or
SIGHUP. On GNU/Hurd systems, the process dies with SIGKILL; this
avoids the problem of many stopped, orphaned processes lying around
the system.
By the way, if you are willing to kill them always you can add a trap on the main process to capture signals and exit the children properly.
Consider the following, which runs sleep 60 in the background and then exits:
$ cat run.sh
sleep 60&
ps
echo Goodbye!!!
$ docker run --rm -v $(pwd)/run.sh:/run.sh ubuntu:16.04 bash /run.sh
PID TTY TIME CMD
1 ? 00:00:00 bash
5 ? 00:00:00 sleep
6 ? 00:00:00 ps
Goodbye!!!
This will start a Docker container, with bash as PID1. It then fork/execs a sleep process, and then bash exits. When the Docker container dies, the sleep process somehow dies too.
My question is: what is the mechanism by which the sleep process is killed? I tried trapping SIGTERM in a child process, and that appears to not get tripped. My presumption is that something (either Docker or the Linux kernel) is sending SIGKILL when shutting down the cgroup the container is using, but I've found no documentation anywhere clarifying this.
EDIT The closest I've come to an explanation is the following quote from baseimage-docker:
If your init process is your app, then it'll probably only shut down itself, not all the other processes in the container. The kernel will then forcefully kill those other processes, not giving them a chance to gracefully shut down, potentially resulting in file corruption, stale temporary files, etc. You really want to shut down all your processes gracefully.
So at least according to this, the implication is that when the container exits, the kernel will sending a SIGKILL to all remaining processes. But I'd still like clarity on how it decides to do that (i.e., is it a feature of cgroups?), and ideally a more authoritative source would be nice.
OK, I seem to have come up with some more solid evidence that this is, in fact, the Linux kernel doing the terminating. In the clone(2) man page, there's this useful section:
CLONE_NEWPID (since Linux 2.6.24)
The first process created in a new namespace (i.e., the process
created using the CLONE_NEWPID flag) has the PID 1, and is the
"init" process for the namespace. Children that are orphaned
within the namespace will be reparented to this process rather than
init(8). Unlike the traditional init process, the "init" process of a
PID namespace can terminate, and if it does, all of the processes in
the namespace are terminated.
Unfortunately this is still vague on how exactly the processes in the namespace are terminated, but perhaps that's because, unlike a normal process exit, no entry is left in the process table. Whatever the case is, it seems clear that:
The kernel itself is killing the other processes
They are not killed in a way that allows them any chance to do cleanup, making it (almost?) identical to a SIGKILL
Tried with some examples like ps and ps -ef,after killing a process by using kill-9 PID in Linux,how to verify weather the process is killed or not?
Just run ps aux stat,pid again, and you will see the process with this pid is either a zombie ('Z' in the first column) or dead.
Edit:
Thanks, Mark B, for pointing me about zombies.
Ater the kill, check the PID of the process:
$ pidof PROCESS
You should see no output if the process is gone.
Another similar way:
$ ps aux | grep PROCESS
Notes:
You can kill your own process, but only root user can kill system process or another user process.
After you KILL the process you can get a zombie, you can still see it in the process list but with process STATE Z (wich means Zombie). Zombies cant be killed, they are already dead, so to "kill" zombies you need to kill the zombie's parents. That said, in general you don’t need to get rid of zombie processes unless you have a large amount of them.
I am writing one shell script in which I have parent process and it has child processes which are created by sleep & command. Now I wish to kill the parent process so that the child process will be also killed. I was able to do that this with below command:
trap "kill $$" SIGINT
trap 'kill -HUP 0' EXIT
trap 'kill $(jobs -p)' EXIT
These commands are working with kill [parent_process_ID] commands but if I use kill -9 [parent_process_ID] then only the parent process will be killed.
Please guide me further to achieve this functionality so that when I kill parent process with any command then child process should be also killed.
When you kill a process alone, it will not kill the children.
You have to send the signal to the process group if you want all processes for a given group to receive the signal.
kill -9 -parentpid
Otherwise, orphans will be linked to init.
Child can ask kernel to deliver SIGHUP (or other signal) when parent dies by specifying option PR_SET_PDEATHSIG in prctl() syscall like this:
prctl(PR_SET_PDEATHSIG, SIGHUP);
See man 2 prctl for details.
Sending the -9 signal (SIGKILL) to a program gives no chance for it to execute its own signal handlers (e.g., your trap statements). That is why the children don't get killed automatically. (In general, -9 gives no chance for the app to clean up after itself.) You have to use a weaker signal to kill it (such as SIGTERM.)
See man 7 signal for details.
If I want to strace a multi-threaded process (of all of its threads), how should I do it?
I know that one can do strace -f to follow forked process? But how about attaching to a process which is already multi-threaded when I start stracing? Is a way to tell strace to trace all of system calls of all the threads which belong to this process?
2021 update
strace -fp PID just does the right thing on my system (Ubuntu 20.04.1 LTS). The strace manual page points this out:
-f Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system
calls. Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with thread_id = PID.
Looks like this text was added back in 2013. If -f had this behavior on my system at the time, I didn't realize it. It does now, though!
Original 2013 answer
I just did this in a kludgy way, by listing each tid to be traced.
You can find them through ps:
$ ps auxw -T | fgrep program_to_trace
me pid tid1 ...
me pid tid2 ...
me pid tid3 ...
me pid tid4 ...
and then, according to man strace, you can attach to multiple pids at once:
-p pid Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt
signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Mul‐
tiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is
given).
It says pid, but iirc on Linux the pid and tid share the same namespace, and this appeared to work:
$ strace -f -p tid1 -p tid2 -p tid3 -p tid4
I think that might be the best you can do for now. But I suppose someone could extend strace with a flag for expanding tids. There would probably still be a race between finding the processes and attaching to them in which a freshly started one would be missed. It'd fit in with the existing caveat about strace -f:
-f Trace child processes as they are created by currently traced processes as a result of the fork(2) system call.
On non-Linux platforms the new process is attached to as soon as its pid is known (through the return value of fork(2) in the par‐
ent process). This means that such children may run uncontrolled for a while (especially in the case of a vfork(2)), until the par‐
ent is scheduled again to complete its (v)fork(2) call. On Linux the child is traced from its first instruction with no delay. If
the parent process decides to wait(2) for a child that is currently being traced, it is suspended until an appropriate child
process either terminates or incurs a signal that would cause it to terminate (as determined from the child's current signal dispo‐
sition).
On SunOS 4.x the tracing of vforks is accomplished with some dynamic linking trickery.
As answered in multiple comments, strace -fp <pid> will show the trace of all threads owned by that process - even ones that process already has spawned before strace begins.