I need to watch for a process with a known PID in Linux. Once it is terminated want to execute a command with reason of termination.
Questions
How to subscribe to process health rather than polling (e.g. watch
command)?
Where to inject the event handler in OS' user space?
How to detect the termination/failure reason inside handler?
Note
The process I intend to keep a tab on is not forked as a child process of some parent through which it can be monitored.
The process type is generic (good number of them are daemons)
The only way to get that kind of control over another process, is to use ptrace(2) to trace the target process. You would use ptrace(PTRACE_ATTACH, pid) to attach to the process, after which you effectively become the target process's parent (and can use wait or more ptrace calls to figure out what the process is doing).
Related
Background processes don't belong to a user and a terminal, nor do daemon processes. What is the main difference between the two? If I were to write a server program, should I run it as a background process or a daemon?
When one says 'background process', it's usually in the context of a shell (like bash), which implements job control.
When a process (or a process group) is put into the background, it's still part of the session created by the shell and will still have an association with the shell's controlling terminal. The standard input/output of a background process will still be linked to the terminal (unless explicitly changed). Also, depending on how the shell exits, it may send a SIGHUP signal to all the background processes (See this answer to know exactly when). Until the shell terminates, it remains the parent of the background process.
A daemon on the other hand does not have a controlling terminal and is usually explicitly made to be a child of the init process. The standard input/output of a dare usually redirected to /dev/null
A Background process usually refers to a process which:
Another process is its parent; eg, a shell;
It has standard streams (input, output, error) connected to that parent
The most common type is when you run a shell program with a trailing &. It generally shares the shell’s output streams, but will get a signal and stop if it tries to read from its input stream.
More significantly (usually), a background process like this is still parented, so signals to that process group will continue to it. If the parent process terminates, the children will receive signals that will most likely terminate them, as well. (This is probably the biggest difference between the two for most users.)
A Daemon process is one that:
Has no parent, ie, its parent process is the system (or container) initial thread, commonly systemd (Linux), init (other Unix), or launchd? (MacOS);
Typically has its output disconnected, or connected to a log file;
Typically has its input disconnected.
Daemons are usually also written to accept the “user hung up” signal (SIGHUP), which would terminate a program if not handled, as a special instruction to re-read their configuration files and continue working.
Most often, these are processes created by some system-level facility that continue to operate completely independently of user activity (logins, logouts, and the like). Things that, themselves, handle logins (getty or gdm and the like), as well as other network-facing services (web servers, mail servers, etc) may be daemons, as well as self-monitoring services like cron, or smartd.
Suppose I have a parent process and a child process (started with e.g. fork() or clone()) running on Linux. Further suppose that there is some shared memory that both the parent and the child can modify.
Within the context of the parent process, I would like to stop the child process and know that it has actually stopped, and moreover that any shared memory writes made by the child are visible to the parent (including whatever synchronization or cache flushes that may require in a multi-processor system).
This answer, which speaks of using kill(SIGSTOP) to stop a child process, contains an interesting tidbit:
When the first kill() call succeeds, you can safely assume that the child has stopped.
Is this statement actually true, and if so, can anyone expound on it, or point me to some more detailed documentation (e.g. a Linux manpage)? Otherwise, is there another mechanism that I can use to ensure that the child process is completely stopped and is not going to be doing any more writes to the shared memory?
I'm imagining something along the lines of:
the parent sends a different signal (e.g. SIGUSR1), which the child can handle
the child handles the SIGUSR1 and does something like a pthread_cond_wait() in the signal handler to safely "stop" (though still running from the kernel perspective) -- this is not fully fleshed out in my mind yet, just an idea
I'd like to avoid reinventing the wheel if there's already an established solution to this problem. Note that the child process needs to be stopped preemptively; adding some kind of active polling to the child process is not an option in this case.
If it only existed on Linux, pthread_suspend() would be perfect ...
It definitely sounds like you should be using a custom signal with a handler, and not sigstop.
It's rare not to care about the state of the child at all, e.g. being fine with it having stored 32bits out of a single non-atomic 64bit write, or logically caught between two dependent writes.
Even if you are, POSIX allows the OS to not make shared writes immediately visible to other processes, so the child should have a chance to call msync for portability, to ensure that writes are completely synced.
The POSIX documentation on Signal Concepts strongly suggests, but does not explicitly say, that the targeted process will be STOPped by the time kill() returns:
A signal is said to be "generated" for (or sent to) a process or thread when the event that causes the signal first occurs... Examples of such events include ... invocations of the kill() and sigqueue() functions.
The documentation is at pains to distinguish signal generation from delivery (when the signal action takes effect) or acceptance. Unfortunately, it sometimes mentions actions taken in response to a stop signal upon generation, and sometimes upon delivery. Given that something must happen upon generation per se, I'd agree that the target process must be STOPped by the time your call returns.
However, at the cost of another syscall, you can be sure. Since you have a parent/child relationship in your design, you can waitpid()/WUNTRACED to receive notification that your child process has, indeed, STOPped.
Edit
See the other answer from that other guy [sic] for reasons why you might not want to do this.
In am writing an SDK in Go that integrators will communicate with via a local socket connection.
From the integrating application I need a way to start the SDK as a process but more importantly, I need to be able to cancel that process when the main application is closing too.
This question is language agnostic (I think) as I think the challenge is linux related. i.e. How to start a program and cancel it at a later stage.
Some possible approaches:
I am thinking that it's a case of starting the program via exec, getting it's PID or some ID then using that to kill later. Sudo may be required to do this, which is not ideal. Also, not good practice as you will be effectively force closing the SDK, offering no time for cleanup.
Start the program via any means. Once ready to close, just send a "shutdown" command via the SDK API which will allow the SDK to cleanup, manage state then exit the application.
What is best practice for this please?
Assuming you're using Linux or similar Unix:
You are on the right track. You won't need sudo. The comments thus far are pointing in the right direction, but not spelling out the details.
See section 2 of the manual pages (man 2 ...) for details on the functions mentioned here. They are documented for calling from C. I don't have experience with Go to help determine how to use them there.
The integrator application will be called the "parent" process. The SDK-as-a-process will be called the "child" process. A process creates a child and becomes its parent by calling fork(). The new process is a duplicate of the parent, running the same code, and having for the most part all the same state (data in memory). But fork() returns different values to parent and child, so each can determine its role in the relationship. This includes informing the parent of the process identifier (pid) of the child. Hang on to this value. Then, the child uses exec() to execute a different program within the existing process, i.e. your SDK binary. An alternative to fork-then-exec is posix_spawn(), which has rather involved parameters (but gives greater control if you need it).
Designing the child to shutdown in response to a signal, rather than a command through the API, will allow processes other than the parent to initiate clean shutdown in standard fashion. For example, this might be useful for the administrator or user; it enables sending the shutdown signal from the shell command-line or script.
The child installs a signal handler function, that will be called when the child process receives a signal, by calling signal() (or the more complex sigaction() recommended for its portability). There are different signals that can be sent/received, identified by different integer values (and also given names like SIGTERM). You indicate which you're interested in receiving when calling signal(). When your signal handler function is invoked, you've received the signal, and can initiate clean shutdown.
When the parent wants the child to shut down cleanly, the parent sends a signal to the child using the unfortunately named kill(). Unfortunately named because signals can be used for other purposes. Anyway, you pass to kill() the pid (returned by fork()) and the specific signal (e.g. SIGTERM) you want to send.
The parent can also determine when the child has completely shut down by calling waitpid(), again passing the pid returned by fork(); or alternately by registering to receive signal SIGCHLD. Register to receive SIGCHLD before fork()/exec() or you might miss the signal.
Actually, it's important that you do call waitpid(), optionally after receiving SIGCHLD, in order to deallocate a resource holding the child process's exit status, so the OS can cleanup that last remnant of the process. Failing to do so keeps the child as a "zombie" process, unable to be fully reclaimed. Too many zombies and the OS will be unable to launch new processes.
If a process refuses to shut down cleanly or as quickly as you require, you may force it to quit (without executing its cleanup code) by sending the signal SIGKILL.
There are variants of exec(), waitpid() and posix_spawn(), with different names and behaviors, mentioned in their man pages.
I have a test case where there are threads spawned using CLONE_THREAD option in clone() .Here if i want to kill a particular thread I suppose we should be using SYS_tgkill in systemcall(). But will the kill actually affect a thread if it is waiting in kernel space(say a futex_wait)?
I tried killing a thread created in the above manner.But when SIGKILL is sent to the same the whole process is getting killed.Am i missing something in using syscall(SYS_tgkill,pid,tid,9) ?
SIGKILL always kills the target process. There is no way around this; it's unblockable, unignorable, and uncatchable.
You could try sending another signal (like SIGUSR1 or SIGHUP or SIGRTMIN) and having a signal handler installed that calls pthread_exit (but note that this function is not async-signal-safe, so you must ensure that the signal handler did not interrupt another async-signal-unsafe function) or use cancellation (pthread_cancel) to stop the blocked thread.
This should work for normal blocking operations (like waiting for data from a pipe or socket), but it will not help you if the thread is in an uninterruptable sleep state (like trying to read from a badly scratched CD or failing hard disk).
I want to terminate a process group by sending SIGTERM to processes within it. This can be accomplished via the kill command, but the manuals I found provide few details about how exactly it works:
int kill(pid_t pid, int sig);
...
If pid is less than -1, then sig is sent to every process in
the process group whose ID is -pid.
However, in which order will the signal be sent to the processes that form the group? Imagine the following situation: a pipe is set between master and slave processes in the group. If slave is killed during processing kill(-pid), while the master is still not, the master might report this as an internal failure (upon receiving notification that the child is dead). However, I want all processes to understand that such termination was caused by something external to their process group.
How can I avoid this confusion? Should I be doing something more than mere kill(-pid,SIGTERM)? Or it is resolved by underlying properties of the OS, about which I'm not aware?
Note that I can't modify the code of the processes in the group!
Try doing it as a three-step process:
kill(-pid, SIGSTOP);
kill(-pid, SIGTERM);
kill(-pid, SIGCONT);
The first SIGSTOP should put all the processes into a stopped state. They cannot catch this signal, so this should stop the entire process group.
The SIGTERM will be queued for the process but I don't believe it will be delivered, since the processes are stopped (this is from memory, and I can't currently find a reference but I believe it is true).
The SIGCONT will start the processes again, allowing the SIGTERM to be delivered. If the slave gets the SIGCONT first, the master may still be stopped so it will not notice the slave going away. When the master gets the SIGCONT, it will be followed by the SIGTERM, terminating it.
I don't know if this will actually work, and it may be implementation dependent on when all the signals are actually delivered (including the SIGCHLD to the master process), but it may be worth a try.
My understanding is that you cannot rely on any specific order of signal delivery.
You could avoid the issue if you send the TERM signal to the master process only, and then have the master kill its children.
Even if all the various varieties of UNIX would promise to deliver the signals in a particular order, the scheduler might still decide to run the critical child process code before the parent code.
Even your STOP/TERM/CONT sequence will be vulnerable to this.
I'm afraid you may need something more complicated. Perhaps the child process could catch the SIGTERM and then loop until its parent exits before it exits itself? Be sure and add a timeout if you do this.
Untested: Use shared memory and put in some kind of "we're dying" semaphore, which may be checked before I/O errors are treated as real errors. mmap() with MAP_ANONYMOUS|MAP_SHARED and make sure it survives your way of fork()ing processes.
Oh, and be sure to use the volatile keyword or your semaphore is optimized away.