I had a doubt on using fork on a multi-threaded process.
If a process has multiple threads (already created using pthread_create and did a pthread_join) and I call fork, will it copy the same functions assigned to the threads in the child process or create a space where we can reassign the functions?
Read carefully what POSIX says about fork() and threads. In particular:
A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.
The child process will have a single thread running in the context of the calling thread. Other parts of the original process may be tied up by threads that no longer exist (so mutexes may be locked, for example).
The rationale section (further down the linked page) says:
There are two reasons why POSIX programmers call fork(). One reason is to create a new thread of control within the same program (which was originally only possible in POSIX by creating a new process); the other is to create a new process running a different program. In the latter case, the call to fork() is soon followed by a call to one of the exec functions.
The general problem with making fork() work in a multi-threaded world is what to do with all of the threads. There are two alternatives. One is to copy all of the threads into the new process. This causes the programmer or implementation to deal with threads that are suspended on system calls or that might be about to execute system calls that should not be executed in the new process. The other alternative is to copy only the thread that calls fork(). This creates the difficulty that the state of process-local resources is usually held in process memory. If a thread that is not calling fork() holds a resource, that resource is never released in the child process because the thread whose job it is to release the resource does not exist in the child process.
When a programmer is writing a multi-threaded program, the first described use of fork(), creating new threads in the same program, is provided by the pthread_create() function. The fork() function is thus used only to run new programs, and the effects of calling functions that require certain resources between the call to fork() and the call to an exec function are undefined.
Related
I am taking an OS class and trying to wrap my head around this question, any help would be appreciated:
Consider a multi-threaded process with 10 threads. Thread 3 invokes an execlp() system call. Describe what will happen to each of the threads.
My understanding of exec() is that is replaces the current process with a new one, and it's main difference from fork() is that fork() creates a clone and you end up with duplicates.
So if exec() replaces the current process, would it kill the threads of the old process and replace them with the new one? Any help will be appreciated.
exec()...replaces the current process with a new one.
Actually, it's still the same process after calling exec, and that's important because the parent process may still need to communicate with it, signal it, etc. What exec does is, it guts the process—wipes out all of it's virtual memory, resets all of its signal handlers, unlocks locks, closes some open files, etc. (See here for more)—and then it loads a new program into the existing process and starts executing it.
would it kill the threads of the old process...?
man 2 execve says, "All threads other than the calling thread are destroyed during
an execve(). Mutexes, condition variables, and other pthreads
objects are not preserved."
In a multi-threaded application, if a thread calls fork(), it will copy the state of only that thread. So the child process created would be a single-thread process. If some other thread were to hold a lock required by the thread which called the fork(), that lock would never be released in the child process. This is a problem.
To counter this, we can modify the fork() in two ways. Either we can copy all the threads instead of only that single one. Or we can make sure that any lock held by the (other) non-copied threads will be released. So what will be the modified fork() system call in both these cases. And which of these two would be better, or what would be the advantages and disadvantages of either option?
This is a thorny question.
POSIX has pthread_atfork() to work through the mess of mixing forks and thread creation. The NOTES section of that man page discusses mutexes etc. However, it acknowledges that getting it right is hard.
The function isn't so much an alternative to fork() as it is a way to explain to the pthread library how your program needs to be prepared for the use of fork().
In general not trying to launch a thread from the child of fork but either exiting that child or calling exec asap, will minimize problems.
This post has a good discussion of pthread_atfork().
...Or we can make sure that any lock held by the (other) non-copied threads will be released.
That's going to be harder than you realize because a program can implement "locks" entirely in user-mode code, in which case, the OS would have no knowledge of them.
Even if you were careful only to use locks that were known to the OS you still have a more general problem: Creating a new process with just the one thread would effectively be no different from creating a new process with all of the threads and then immediately killing all but one of them.
Read about why we don't kill threads. In a nutshell: Locks aren't the only state that needs to be cleaned up. Any of the threads that existed in the parent but not in the child could, at the moment of the fork call, been in the middle of making a mess that needs to be cleaned up. If that thread doesn't exist in the child, then you've lost the knowledge of what needs to be cleaned up.
we can copy all the threads instead of only that single one...
That also is a potential problem. The one thread that calls fork() would know when and why fork() was called, and it would be prepared for the fork call. None of the other threads would have any warning. And, if any of those threads is interacting with something outside of the process (e.g., talking to a remote service) then,where you previously had one client talking to the service, you suddenly have two clients, talking to the same service, and they both think that they are the only one. That's not going to end well.
Don't call fork() from multi-threaded programs.
In one project I worked on: We had a big multi-threaded program that needed to spawn other processes. How we did it is, we had it spawn a simple, single-threaded "helper" program before it created any new threads. Then, whenever it needed to spawn another process, it sent a message to the helper, and the helper did it.
pthread_detach marks a thread so that when it terminates, its resources are automatically released without requiring the parent thread to call pthread_join. How can it do this? From the perspective of Linux in particular, there are two resources in particular I am curious about:
As an implementation detail, I would expect that if a wait system call is not performed on the terminated thread, then the thread would become a zombie. I assume that the pthread library's solution to this problem does not involve SIGCHLD, because (I think) it still works regardless of what action the program has specified to occur when SIGCHLD is received.
Threads are created using the clone system call. The caller must allocate memory to serve as the child thread's stack area before calling clone. Elsewhere on Stack Overflow, it was recommended that the caller use mmap to allocate the stack for the child. How can the stack be unmapped after the thread exits?
It seems to me that pthread_detach must somehow provide solutions to both of these problems, otherwise, a program that spawns and detaches many threads would eventually lose the ability to continue spawning new threads, even though the detached threads may have terminated already.
The pthreads library (on Linux, NPTL) provides a wrapper around lower-level primitives such as clone(2). When a thread is created with pthread_create, the function passed to clone is a wrapper function. That function allocates the stack and stores that information plus any other metadata into a structure, then calls the user-provided start function. When the user-provided start function returns, cleanup happens. Finally, an internal function called __exit_thread is called to make a system call to exit the thread.
When such a thread is detached, it still returns from the user-provided start function and calls the cleanup code as before, except the stack and metadata is freed as part of this since there is nobody waiting for this thread to complete. This would normally be handled by pthread_join.
If a thread is killed or exits without having run, then the cleanup is handled by the next pthread_create call, which will call any cleanup handlers yet to be run.
The reason a SIGCHLD is not sent to the parent nor is wait(2) required is because the CLONE_THREAD flag to clone(2) is used. The manual page says the following about this flag:
A new thread created with CLONE_THREAD has the same parent process as the process that made the clone call (i.e., like CLONE_PARENT), so that calls to getppid(2) return the same value for all of the threads in a thread group. When a CLONE_THREAD thread terminates, the thread that created it is not sent a SIGCHLD (or other termination) signal; nor can the status of such a thread be obtained using wait(2). (The thread is said to be detached.)
As you noted, this is required for the expected POSIX semantics to occur.
It is well-known that the default way to create a new process under POSIX is to use fork() (under Linux this internally maps to clone(...))
What I want to know is the following: It is well-known that when one calls fork() "The child process is created with a single thread--the one that called fork()"
(cf. https://linux.die.net/man/2/fork). This can of course cause problems if for example some other thread currently holds a lock. To me not also forking all the threads that exist in the process intuitively feels like a "leaky abstraction".
So I would like to know: What is the reason why only the thread calling fork() will exist in the child process instead of all threads of the process? Is there a good technical reason for this?
I know that on Multithreaded fork there is a related question, but the answers given there don't answer mine.
Of these two possibilities:
only the thread calling fork() continues running in the child process
Downside: if another thread was holding on to an internal resource such as a lock, it will not be released.
after fork(), all threads are duplicated into the child process
Downside: threads that were interacting with external resources continue running in parallel. If a thread was appending data to a file: now it happens twice.
Both are bad, but the first one choice only deadlocks the new child process, while the second choice results in corruption outside of the process. This could be described as "bad".
POSIX did standardize pthread_atfork to try to allow automatic cleanup in the first case, but it cannot possibly work.
tl;dr Don't use both threads and forks. Use posix_spawn if you have to.
If I fork a process in which one thread holds a mutex, am I relatively safe if I immediately exec in the child? What things are safe to do in the child before I exec?
If the thread that executes fork then the child goes on to release a mutex before calling exec will this cause problems? What happens if I try to acquire a mutex in a child that the parent process owned before fork (and may or may not still own)?
Are the answers different on different platforms? I'm primarily concerned with Unix variants, and in particular Linux. But I am curious about NT. Though, of course, NT doesn't (to my knowledge) have fork.
See pthread_atfork, especially RATIONALE section, for a discussion of problems related to fork in multithreaded environment. It also gives a hint on what supposed to be valid before and after fork within a child and a parent.
UPDATE: the RATIONALE section is non-normative, and it turned out to be in conflict with other parts of the standard. See this defect report by Dave Butenhof for more details.
Immediate exec after fork is supposed to be safe for any state of multithreaded program (that is, any threads holding any mutexes). As of the things possible between fork and exec, the situation is complicated:
The most important thing is that only one thread (that which called fork) is duplicated in the child process. Consequently, any mutex held by another thread at the moment of fork becomes locked forever. That is (assuming non-process-shared mutexes) its copy in the child process is locked forever, because there is no thread to unlock it.
Releasing mutex after fork is safe when it's possible, that is, if the forking thread owns the mutex in the first place. That's how pthread_atfork handlers usually work: locking mutexes before fork, unlocking in child and unlocking in parent.
As of acquiring a mutex that the process owned before fork (remember, we discuss a copy in the child's address space): if it was owned by a forking thread, it's recursive locking (works for PTHREAD_MUTEX_RECURSIVE); if it was owned by another thread, it remains locked forever and can't be reacquired.
By registering appropriate pthread_atfork handlers, third-party libraries can provide a guarantee of being safe to use between fork and exec. (I would expect it mostly from programming language runtimes, not for general purpose libraries).
After some more research, I would recommend to avoid relying in any way on pthread_atfork, and doing nothing but async-signal-safe calls between fork and exec (abandoning fork/exec for posix_spawn would be even better).
The problem is, fork itself can be invoked in signal handler. It precludes any nontrivial use of pthread_atfork, even if its RATIONALE explicitly mentions unlocking mutexes and recreating threads (!) in a child process.
I think that a "grey area" of different possible interpretations remains:
For pthread_atfork handlers in a program which is known to never call fork in a signal handler.
For non-pthread-atfork actions happening around the fork call which is not in a signal handler.
But it's crystal clear which reading is to be used for portable applications.
It is safe to exec() after fork() provided the mutex is owned by the program that will get replaced by the exec(). If the mutex is part of a library and is protecting a resource that must be accessed serially then it should call pthread_atfork() to register callbacks:
A function that gets called prior to the fork itself and in the context of the thread that makes the fork() system call. Typically this function will grab the lock on mutexes that protect critical sections, thereby guaranteeing that during the fork no thread is inside a critical section
A function to be called in the context of the thread invoking fork(), after the child process has been created but before the fork() system call returns. This function can then unlock the mutexes in the parent process.
A function to be called in the context of the thread of the child process if/when the process forks - it gets called after the child process has been created but before the fork() system call returns. The child process can then unlock its copy of the mutex.
The POSIX standard limits the type of system calls that are allowed after a fork() and before an exec() to so called async-signal-safe system calls. Creating a thread isn't explicitly listed as an async-signal-safe system call, therefore POSIX doesn't allow a child process to create threads after a fork() and before exec(). Ironically, unlocking a mutex isn't explicitly listed as an async-signal-safe system call either, and thereofre isn't strictly allowed after a fork() in the child process if the intention of the fork() is to then exec() - probably an oversight in the standard.