I have a multi-process application that works like so...
There is a parent process. The parent process queries a database to find work, then forks children to process that work. The children communicate back to the parent via System V message queues to indicate they're done with their work. When the parent process picks up that message, it updates the database to indicate that the work is complete.
This works okay but I'm struggling with handling the parent process being killed.
What happens is the parent receives a SIGINT(from CTRL-C), and then sends SIGKILLs to each of the children. If a child is currently blocking on a Sys V message queue write when it receives that signal, the write is "interrupted" by the signal and the blocking canceled and the parent never learns that the child's work was done, and the database never gets updated.
That means that the next time I run the script, it will re-run any work that was blocking on the System V queue write.
I don't have a good idea for a solution for this yet. Ideally I would like to be able to force the queue write to remain blocking even when it receives that SIGKILL but I don't think such a thing is possible.
Well SIGKILL is, by definition, immediately fatal to the process which receives it and cannot be trapped or handled.
That is why you should only use it as a last resort, when the process does not respond to more polite requests to shut down. Your parent process should start off by sending something like SIGINT or SIGTERM to the children, and only reset to SIGKILL if they don't exit within a reasonable period of time.
Signals like SIGINT and SIGTERM may still cause the system call in the child to return, with EINTR, but you can handle that and retry the call and let it complete before exiting.
Related
I know that a child process whose parent has died becomes a zombie process but when that happens, does it continue execution normally ?
What I have read so far seems to suggest that yes but I have not found confirmation and my programming ventures seems to suggest otherwise.
Whether a child's parent has exited has no effect on whether it continues running. Assuming the child has access to the resources that it needs, it will continue to run normally.
This is important when writing a daemon, since typically the started process forks twice, and it is the grandchild that ultimately runs as a service.
Note that there are some reasons a child may end up exiting abnormally due to a parent exiting. For example, if the parent is an interactive shell and it exits, the terminal may disappear, and as a result the child may receive a SIGHUP. However, in that case, the reason the child will have exited is because it received a signal it didn't handle, and if it had set up a suitable handler, it would have continued running.
In am writing an SDK in Go that integrators will communicate with via a local socket connection.
From the integrating application I need a way to start the SDK as a process but more importantly, I need to be able to cancel that process when the main application is closing too.
This question is language agnostic (I think) as I think the challenge is linux related. i.e. How to start a program and cancel it at a later stage.
Some possible approaches:
I am thinking that it's a case of starting the program via exec, getting it's PID or some ID then using that to kill later. Sudo may be required to do this, which is not ideal. Also, not good practice as you will be effectively force closing the SDK, offering no time for cleanup.
Start the program via any means. Once ready to close, just send a "shutdown" command via the SDK API which will allow the SDK to cleanup, manage state then exit the application.
What is best practice for this please?
Assuming you're using Linux or similar Unix:
You are on the right track. You won't need sudo. The comments thus far are pointing in the right direction, but not spelling out the details.
See section 2 of the manual pages (man 2 ...) for details on the functions mentioned here. They are documented for calling from C. I don't have experience with Go to help determine how to use them there.
The integrator application will be called the "parent" process. The SDK-as-a-process will be called the "child" process. A process creates a child and becomes its parent by calling fork(). The new process is a duplicate of the parent, running the same code, and having for the most part all the same state (data in memory). But fork() returns different values to parent and child, so each can determine its role in the relationship. This includes informing the parent of the process identifier (pid) of the child. Hang on to this value. Then, the child uses exec() to execute a different program within the existing process, i.e. your SDK binary. An alternative to fork-then-exec is posix_spawn(), which has rather involved parameters (but gives greater control if you need it).
Designing the child to shutdown in response to a signal, rather than a command through the API, will allow processes other than the parent to initiate clean shutdown in standard fashion. For example, this might be useful for the administrator or user; it enables sending the shutdown signal from the shell command-line or script.
The child installs a signal handler function, that will be called when the child process receives a signal, by calling signal() (or the more complex sigaction() recommended for its portability). There are different signals that can be sent/received, identified by different integer values (and also given names like SIGTERM). You indicate which you're interested in receiving when calling signal(). When your signal handler function is invoked, you've received the signal, and can initiate clean shutdown.
When the parent wants the child to shut down cleanly, the parent sends a signal to the child using the unfortunately named kill(). Unfortunately named because signals can be used for other purposes. Anyway, you pass to kill() the pid (returned by fork()) and the specific signal (e.g. SIGTERM) you want to send.
The parent can also determine when the child has completely shut down by calling waitpid(), again passing the pid returned by fork(); or alternately by registering to receive signal SIGCHLD. Register to receive SIGCHLD before fork()/exec() or you might miss the signal.
Actually, it's important that you do call waitpid(), optionally after receiving SIGCHLD, in order to deallocate a resource holding the child process's exit status, so the OS can cleanup that last remnant of the process. Failing to do so keeps the child as a "zombie" process, unable to be fully reclaimed. Too many zombies and the OS will be unable to launch new processes.
If a process refuses to shut down cleanly or as quickly as you require, you may force it to quit (without executing its cleanup code) by sending the signal SIGKILL.
There are variants of exec(), waitpid() and posix_spawn(), with different names and behaviors, mentioned in their man pages.
Say I create a process with multiple child process and I call wait() in the main process. If one child terminates, its pid is returned. But what happens if a couple of child process terminate simultaneously? The call should return with one of them, and a second call should return with the other, right? Is there a particular order in which they will return (maybe there is a precedence to the child with lower pid)?
No.
SUSv4 leaves explicitly unspecified in which order (if any) child processes are reaped by one or several wait calls. There is also no "accidential" order that you could rely on, since different Linux kernel versions perform differently. (Source: M. Kerrisk, TLPI, 26.1.1, page 542).
Somewhat related trivia:
You might wonder why you can reliably wait on several child processes that terminate concurrently at all. If you think about how signals work, you might be inclined to believe that it is perfectly possible to lose child termination signals. Signals, a well-known fact, are not queued (except for realtime signals, but SIGCHLD isn't one!). Which means that if you go strictly by the letter of the book, then clearly several children terminating could cause some child termination signals becoming lost!
You can only call wait once at the same time, so you can at most consume one signal synchronously as it is generated, and have a second one made pending before your next call to wait. It appears that there is no way to account for any other signals that are generated in the mean time.
Luckily, that isn't the case. Waiting on child processes demonstrably works fine. Always, 100%. But why?
The reason is simple, the kernel is "cheating". If a child process exits while you are blocked in wait, there is no question as to what's happening. The signal is immediately delivered, the status information is being filled in, the parent unblocks, and the child process goes * poof *.
On the other hand, if a child is exiting and the parent isn't in a wait call (or if the parent is in a wait call reaping another child), the system converts the exiting process to a "zombie".
Now, when the parent process performs a wait, and there are any zombie processes, it won't block waiting for a signal to be generated (which may never happen if all the children have already exited!). Instead, it will just reap one of the zombies, pretending the signal was delivered just now.
I've got something that uses a bunch of async forks to do it's work (underneath a toolkit).
In a specific region of code, I'm forking, and then doing a blocking wait on the child process.
Is the SIGCHLD handler going to gobble the signal before the blocking wait will see it, leaving me potentially hung, or is the wait always going to get something back?
A SIGCHLD handler gets fired on the event, the edge, of a child process exiting. A blocking call to waitpid() will wait for the condition, the level, of that specific child process no longer existing.
When the process exits, a SIGCHLD will be delivered, and its handler will execute normally. If there was a waitpid() blocking on that process, it will then return as normal, regardless of the presence of a signal handler.
I want to terminate a process group by sending SIGTERM to processes within it. This can be accomplished via the kill command, but the manuals I found provide few details about how exactly it works:
int kill(pid_t pid, int sig);
...
If pid is less than -1, then sig is sent to every process in
the process group whose ID is -pid.
However, in which order will the signal be sent to the processes that form the group? Imagine the following situation: a pipe is set between master and slave processes in the group. If slave is killed during processing kill(-pid), while the master is still not, the master might report this as an internal failure (upon receiving notification that the child is dead). However, I want all processes to understand that such termination was caused by something external to their process group.
How can I avoid this confusion? Should I be doing something more than mere kill(-pid,SIGTERM)? Or it is resolved by underlying properties of the OS, about which I'm not aware?
Note that I can't modify the code of the processes in the group!
Try doing it as a three-step process:
kill(-pid, SIGSTOP);
kill(-pid, SIGTERM);
kill(-pid, SIGCONT);
The first SIGSTOP should put all the processes into a stopped state. They cannot catch this signal, so this should stop the entire process group.
The SIGTERM will be queued for the process but I don't believe it will be delivered, since the processes are stopped (this is from memory, and I can't currently find a reference but I believe it is true).
The SIGCONT will start the processes again, allowing the SIGTERM to be delivered. If the slave gets the SIGCONT first, the master may still be stopped so it will not notice the slave going away. When the master gets the SIGCONT, it will be followed by the SIGTERM, terminating it.
I don't know if this will actually work, and it may be implementation dependent on when all the signals are actually delivered (including the SIGCHLD to the master process), but it may be worth a try.
My understanding is that you cannot rely on any specific order of signal delivery.
You could avoid the issue if you send the TERM signal to the master process only, and then have the master kill its children.
Even if all the various varieties of UNIX would promise to deliver the signals in a particular order, the scheduler might still decide to run the critical child process code before the parent code.
Even your STOP/TERM/CONT sequence will be vulnerable to this.
I'm afraid you may need something more complicated. Perhaps the child process could catch the SIGTERM and then loop until its parent exits before it exits itself? Be sure and add a timeout if you do this.
Untested: Use shared memory and put in some kind of "we're dying" semaphore, which may be checked before I/O errors are treated as real errors. mmap() with MAP_ANONYMOUS|MAP_SHARED and make sure it survives your way of fork()ing processes.
Oh, and be sure to use the volatile keyword or your semaphore is optimized away.