I understand that race condition is possible between threads because they are operating on the same Data.
So in my CS course I learned that the main difference between process and thread is that thread is lightweight Process in sense that the Data that is being operated on is not copied, but that the thread simply gets cpu time, but operates on the same Data as some other thread, which makes sense why race condition is possible, but how can race condition be possible with process, the course states that this is possible becauese of the shared Data, but how can Data be share between 2 processes if the process gets its own "Block" within which it operates, shared ressources would mean we have threads and not a process:
void forkexample{
pid_t pid = fork()
if (pid == 0){
printf ("Hello from Child! /n");
}
else
{
printf ("Hello from Parent! child has pid = %d /n", pid);
}
}
the child process that will be created is basically a deep copy of the parent Process so how can deep copy cause race conditions?
Parent Process:
------------------
CPU States |
Memory block |
other ressources |
------------------
Child Process completaly new block contaning same code and initial states,
but is completaly new Block in which it operates i.e Data is copied etc.
so how can race condition happen here?
------------------
CPU States |
Memory block |
other ressources |
------------------
When a fork happens, the standard allocated memory is not shared by default (operating systems typically use a copy-on-write approach in practice for the sake of performance). However, the memory can be manually/explicitly shared using the mmap system call. Because of this, race condition are still possible. Moreover, there are still several shared resources on mainstream machines that can generally be shared between processes by default. A good example is the file system : if two processes read/write the same file without any lock or synchronization, then there is likely a race condition.
Related
On a linux system, does the child process view the existing threads the same way as the parent process ?
int main() {
//create thread 1
int child_pid = fork();
if ( 0 == child_pid)
{
..
}
else
{
..
}
Since the whole address space is copied for the child process, what happens to the state of the threads. What if the thread 1 in the above segment is waiting on a conditional signal. Is it in the waiting state in child process as well ?
Threads on Linux nowadays try to stay POSIX compliant. Only the calling thread is replicated, not other threads (note that e.g. on Solaris you can choose what fork does depending on what library you link to)
From http://www.opengroup.org/onlinepubs/000095399/functions/fork.html (POSIX 2004):
A process shall be created with a
single thread. If a multi-threaded
process calls fork(), the new process
shall contain a replica of the calling
thread and its entire address space,
possibly including the states of
mutexes and other resources.
Consequently, to avoid errors, the
child process may only execute
async-signal-safe operations until
such time as one of the exec functions
is called. Fork
handlers may be established by means
of the pthread_atfork() function in
order to maintain application
invariants across fork() calls.
The POSIX 2018 specification of fork() is similar.
Threads are not inherited from a child process on a linux system using fork(). An in-depth source is here: http://linas.org/linux/threads-faq.html
in Linux threads are called light weight processes. Whether process or thread, they are implemented by task_struct data structure.
1> So, in that sense how kernel distinguishes between thread and process?
2> when context switching happens, how do threads get less overhead in context switching? because prior to this thread, another thread from another process may be running. So kernel should load all resources even if resources are shared between threads of a processes.
how kernel distinguishes between thread and process.
From http://www.kernel.org/doc/ols/2002/ols2002-pages-330-337.pdf and from Linux - Threads and Process
This was addressed during the 2.4 development cycle with the
addition of a concept called a ’thread group’. There is a linked
list of all tasks that are part of the thread group, and there is an
ID that represents the group, called the tgid. This ID is actually the
pid of the first task in the group (pid is the task ID assigned with a
Linux task), similar to the way sessions and process groups work. This
feature is enabled via a flag to clone().
and
In the kernel, each thread has it's own ID, called a PID (although it would possibly make more sense to call this a TID, or thread ID) and they also have a TGID (thread group ID) which is the PID of the thread that started the whole process.
Simplistically, when a new process is created, it appears as a thread
where both the PID and TGID are the same (new) number.
When a thread starts another thread, that started thread gets its own
PID (so the scheduler can schedule it independently) but it inherits
the TGID from the original thread.
So a main thread is a thread with the same PID and TGID and this PID is a process PID. A thread (but not a main thread) has different PID but the same TID.
Inside kernel, each process and threads have a unique id (even threads of same process) which is stored in pid variable and threads of same process also share a common id which is stored in tgid variable, and is returned to user when getpid() is invoked therefore allowing kernel to distinguish them as different entities which are schedulable in themselves.
When a thread is preempted by another thread of same process, since various segments such as .text, .bss, .data, file descriptors etc. are shared and hence allows a fast context switch compared to when different processes are context switched, or when threads of different processes are context switched.
It seems you mixed some concepts together, implemented by the same data structure does not mean they run in the same way.
you can read
what-is-the-difference-between-a-process-and-a-thread to clarify your comprehension about process and thread firstly.
I'm writing a basic UNIX program that involves processes sending messages to each other. My idea to synchronize the processes is to simply have an array of flags to indicate whether or not a process has reached a certain point in the code.
For example, I want all the processes to wait until they've all been created. I also want them to wait until they've all finished sending messages to each other before they begin reading their pipes.
I'm aware that a process performs a copy-on-write operation when it writes to a previously defined variable.
What I'm wondering is, if I make an array of flags, will the pointer to that array be copied, or will the entire array be copied (thus making my idea useless).
I'd also like any tips on inter-process communication and process synchronization.
EDIT: The processes are writing to each other process' pipe. Each process will send the following information:
typedef struct MessageCDT{
pid_t destination;
pid_t source;
int num;
} Message;
So, just the source of the message and some random number. Then each process will print out the message to stdout: Something along the lines of "process 20 received 5724244 from process 3".
Unix processes have independent address spaces. This means that the memory in one is totally separate from the memory in another. When you call fork(), you get a new copy of the process. Immediately on return from fork(), the only thing different between the two processes is fork()'s return value. All of the data in the two processes are the same, but they are copies. Updating memory in one cannot be known by the other, unless you take steps to share the memory.
There are many choices for interprocess communication (IPC) in Unix, including shared memory, semaphores, pipes (named and unnamed), sockets, message queues and signals. If you Google these things you will find lots to read.
In your particular case, trying to make several processes wait until they all reach a certain point, I might use a semaphore or shared memory, depending on whether there is some master process that started them all or not.
If there is a master process that launches the others, then the master could setup the semaphore with a count equal to the number of processes to synchronize and then launch them. Each child could then decrement the semaphore value and wait for the semaphore value to reach zero.
If there is no master process, then I might create a shared memory segment that contains a count of processes and a flag for each process. But when you have two or more processes using shared memory, then you also need some kind of locking mechanism (probably a semaphore again) to ensure that two processes do not try to update the shared memory simultaneously.
Keep in mind that reading a pipe that nobody is writing to will block the reader until data appears. I don't know what your processes do, but perhaps that is synchronization enough? One other thing to consider if you have multiple processes writing to a given pipe, their data may become interleaved if the writes are larger than PIPE_BUF. The value and location of this macro are system dependent.
-Kevin
The entire array of flags will seem to be copied. It will not actually be copied until one process or another writes to it of course. But that's an implementation detail and transparent to the individual processes. As far as each process is concerned, they each get a copy of the array.
There are ways to make this not happen. You can use mmap with the MAP_SHARED option for the memory used for your flags. Then each sub-process will share the same region of memory. There's also Posix shared memory (which I, BTW, think is an awful hack). To find out about Posix shared memory, look at the shm_overview(7) man page.
But using memory in this way isn't really a good idea. On multi-core systems it's not always the case that when one process (or thread) writes to an area of shared memory that all other processes will see the value written right away. Frequently the value will hang out for awhile in the L2 cache and not be immediately flushed.
If you want to communicate using shared memory, you will have to used mutexes or the C++11 atomic operations to ensure that writes are properly seen by the other processes.
Just a quick question, if I clone a process, the PID of the cloned process is the same, yes ? fork() creates a child process where the PID differs, but everything else is the same. Vfork() creates a child process with the same PID. Exec works to change a process currently in execution to something else.
Am I correct in all of these statements ?
Not quite. If you clone a process via fork/exec, or vfork/exec, you will get a new process id. fork() will give you the new process with a new process id, and exec() replaces that process with a new process, but maintaining the process id.
From here:
The vfork() function differs from
fork() only in that the child process
can share code and data with the
calling process (parent process). This
speeds cloning activity significantly
at a risk to the integrity of the
parent process if vfork() is misused.
Neither fork() nor vfork() keep the same PID although clone() can in one scenario (*a). They are all different ways to achieve roughly the same end, the creation of a distinct child.
clone() is like fork() but there are many things shared by the two processes and this is often used to enable threading.
vfork() is a variant of clone in which the parent is halted until the child process exits or executes another program. It's more efficient in those cases since it doesn't involve copying page tables and such. Basically, everything is shared between the two processes for as long as it takes the child to load another program.
Contrast that last option with the normal copy-on-write where memory itself is shared (until one of the processes writes to it) but the page tables that reference that memory are copied. In other words, vfork() is even more efficient than copy-on-write, at least for the fork-followed-by-immediate-exec use case.
But, in most cases, the child has a different process ID to the parent.
*a Things become tricky when you clone() with CLONE_THREAD. At that stage, the processes still have different identifiers but what constitutes the PID begins to blur. At the deepest level, the Linux scheduler doesn't care about processes, it schedules threads.
A thread has a thread ID (TID) and a thread group ID (TGID). The TGID is what you get from getpid().
When a thread is cloned without CLONE_THREAD, it's given a new TID and it also has its TGID set to that value (i.e., a brand new PID).
With CLONE_THREAD, it's given a new TID but the TGID (hence the reported process ID) remains the same as the parent so they really have the same PID. However, they can distinguish themselves by getting the TID from gettid().
There's quite a bit of trickery going on there with regard to parent process IDs and delivery of signals (both to the threads within a group and the SIGCHLD to the parent), all which can be examined from the clone() man page.
It deserves some explanation. And it's simple as rain.
Consider this. A program has to do some things at the same time. Say, your program is printing "hello world!", each second, until somebody enters "hello, Mike", then, each second, it prints that string, waiting for John to change that in the future.
How do you write this the standard way? In your program, that basically prints "hello," you must create another branch that is waiting for user input.
You create two processes, one outputting those strings, and another one, waiting the user input. And, the only way to create a new process in UNIX was calling the system call fork(), like this:
ret = fork();
if(ret > 0) /* parent, continue waiting */
else /* child */
This scheme posed numerous problems. The user enters "Mike" but you have no simple way to pass that string to the parent process so that it'd be able to print that, because +each+ process has its own view of memory that isn't shared with the child.
When the processes are created by fork(), each one receives a copy of the memory existing at that moment, and if that memory really changes later, the mapping that was identical for those memory segments will be chaged at once (it's called a copy-on-write mechanism).
Another thingies to share between the child and the parent are, for example, opened file descriptors, descriptors of the shared memory, input/outpue stuff, etc., that also wouldn't survive after fork().
So. The very fork() call had to be alleviated, to include shared memory/signals etc. But how? This was the idea behind clone(). That call takes a flag indicating what exatly would you share with the child. For example, the memory, the signal handlers, etc. And if you call this with flag=0, this will be identical to fork(), up to the args they take. And when POSIX pthreads are created, that flag will reflect the attributes you have indicated in pthread_attr.
From the kernel point of view, there's no difference between the processes created such way, and no special semantics to differentiate the "processess". The kernel does not even know, what that "thread" is, it creates a new process, but it simply combines it as belogning to that process group that had the parent who called it, taking care what that process may do. So, you have different procesess (that share the same pid) combined in a process group each assigned with a different "TID" (that starts from PID of the parent).
Care to explain that clone() does exactly that. You may pass this whaterver you need (as the matter of fact, the old vfork() call will do). Are you going to share memory? Hanlers? You may tune everything, just be sure you don't clash with the pthreads library written right away around this very call.
An important thing, the kernel vesion is quite outrageous, it expects just 2 out of 4 parameters to be passed, the user stack, and options.
Since PID is an unique identifier for a process, there's no way to have two distinct process with the same PID.
Threads (which have the same visible 'pid') are implemented with the clone() call. When the flag CLONE_THREAD is supplied then the new process (a 'thread') share the Thread Group Identifier (TGID) with its creator process. getpid actually returns the TGID.
See the clone manpage for more details.
In summary the real PID, as seen by the kernel is always different. The visible PID is the same for threads.
I am having some trouble understanding how to use Unix's fork(). I am used to, when in need of parallelization, spawining threads in my application. It's always something of the form
CreateNewThread(MyFunctionToRun());
void myFunctionToRun() { ... }
Now, when learning about Unix's fork(), I was given examples of the form:
fork();
printf("%d\n", 123);
in which the code after the fork is "split up". I can't understand how fork() can be useful. Why doesn't fork() have a similar syntax to the above CreateNewThread(), where you pass it the address of a function you want to run?
To accomplish something similar to CreateNewThread(), I'd have to be creative and do something like
//pseudo code
id = fork();
if (id == 0) { //im the child
FunctionToRun();
} else { //im the parent
wait();
}
Maybe the problem is that I am so used to spawning threads the .NET way that I can't think clearly about this. What am I missing here? What are the advantages of fork() over CreateNewThread()?
PS: I know fork() will spawn a new process, while CreateNewThread() will spawn a new thread.
Thanks
fork() says "copy the current process state into a new process and start it running from right here." Because the code is then running in two processes, it in fact returns twice: once in the parent process (where it returns the child process's process identifier) and once in the child (where it returns zero).
There are a lot of restrictions on what it is safe to call in the child process after fork() (see below). The expectation is that the fork() call was part one of spawning a new process running a new executable with its own state. Part two of this process is a call to execve() or one of its variants, which specifies the path to an executable to be loaded into the currently running process, the arguments to be provided to that process, and the environment variables to surround that process. (There is nothing to stop you from re-executing the currently running executable and providing a flag that will make it pick up where the parent left off, if that's what you really want.)
The UNIX fork()-exec() dance is roughly the equivalent of the Windows CreateProcess(). A newer function is even more like it: posix_spawn().
As a practical example of using fork(), consider a shell, such as bash. fork() is used all the time by a command shell. When you tell the shell to run a program (such as echo "hello world"), it forks itself and then execs that program. A pipeline is a collection of forked processes with stdout and stdin rigged up appropriately by the parent in between fork() and exec().
If you want to create a new thread, you should use the Posix threads library. You create a new Posix thread (pthread) using pthread_create(). Your CreateNewThread() example would look like this:
#include <pthread.h>
/* Pthread functions are expected to accept and return void *. */
void *MyFunctionToRun(void *dummy __unused);
pthread_t thread;
int error = pthread_create(&thread,
NULL/*use default thread attributes*/,
MyFunctionToRun,
(void *)NULL/*argument*/);
Before threads were available, fork() was the closest thing UNIX provided to multithreading. Now that threads are available, usage of fork() is almost entirely limited to spawning a new process to execute a different executable.
below: The restrictions are because fork() predates multithreading, so only the thread that calls fork() continues to execute in the child process. Per POSIX:
A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. [THR] [Option Start] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls. [Option End]
When the application calls fork() from a signal handler and any of the fork handlers registered by pthread_atfork() calls a function that is not asynch-signal-safe, the behavior is undefined.
Because any library function you call could have spawned a thread on your behalf, the paranoid assumption is that you are always limited to executing async-signal-safe operations in the child process between calling fork() and exec().
History aside, there are some fundamental differences with respect to ownership of resource and life time between processes and threads.
When you fork, the new process occupies a completely separate memory space. That's a very important distinction from creating a new thread. In multi-threaded applications you have to consider how you access and manipulate shared resources. Processed that have been forked have to explicitly share resources using inter-process means such as shared memory, pipes, remote procedure calls, semaphores, etc.
Another difference is that fork()'ed children can outlive their parent, where as all threads die when the process terminates.
In a client-server architecture where very, very long uptime is expected, using fork() rather than creating threads could be a valid strategy to combat memory leaks. Rather than worrying about cleaning up memory leaks in your threads, you just fork off a new child process to process each client request, then kill the child when it's done. The only source of memory leaks would then be the parent process that dispatches events.
An analogy: You can think of spawning threads as opening tabs inside a single browser window, while forking is like opening separate browser windows.
It would be more valid to ask why CreateNewThread doesn't just return a thread id like fork() does... after all fork() set a precedent. Your opinion's just coloured by you having seen one before the other. Take a step back and consider that fork() duplicates the process and continues execution... what better place than at the next instruction? Why complicate things by adding a function call into the bargain (and then one what only takes void*)?
Your comment to Mike says "I can't understand is in which contexts you'd want to use it.". Basically, you use it when you want to:
run another process using the exec family of functions
do some parallel processing independently (in terms of memory usage, signal handling, resources, security, robustness), for example:
each process may have intrusive limits of the number of file descriptors they can manage, or on a 32-bit system - the amount of memory: a second process can share the work while getting its own resources
web browsers tend to fork distinct processes because they can do some initialisation then call operating system functions to permanently reduce their privileges (e.g. change to a less-trusted user id, change the "root" directory under which they can access files, or make some memory pages read-only); most OSes don't allow the same extent of fine-grained permission-setting on a per-thread basis; another benefit is if a child process seg-faults or similar the parent process can handle that and continue, whereas similar faults in multi-threaded code raise questions about whether memory has been corrupted - or locks have been held - by the crashing thread such that remaining threads are compromised
BTW / using UNIX/Linux doesn't mean you have to give up threads for fork()ing processes... you can use pthread_create() and related functions if you're more comfortable with the threading paradigm.
Letting the difference between spawning a process and a thread set aside for a second: Basically, fork() is a more fundamental primitive. While SpawnNewThread has to do some background work to get the program counter in the right spot, fork does no such work, it just copies (or virtually copies) your program memory and continues the counter.
Fork has been with us for a very, very, long time. Fork was thought of before the idea of 'start a thread running a particular function' was a glimmer in anyone's eye.
People don't use fork because it's 'better,' we use it because it is the one and only unprivileged user-mode process creation function that works across all variations of Linux. If you want to create a process, you have to call fork. And, for some purposes, a process is what you need, not a thread.
You might consider researching the early papers on the subject.
It is worth noting that multi-processing not exactly the same as multi-threading. The new process created by fork share very little context with the old one, which is quite different from the case for threads.
So, lets look at the unixy thread system: pthread_create has semantics similar to CreateNewThread.
Or, to turn it around, lets look at the windows (or java or other system that makes its living with threads) way of spawning a process identical to the one you're currently running (which is what fork does on unix)...well, we could except that there isn't one: that just not part of the all-threads-all-the-time model. (Which is not a bad thing, mind you, just different).
You fork whenever you want to more than one thing at the same time. It’s called multitasking, and is really useful.
Here for example is a telnetish like program:
#!/usr/bin/perl
use strict;
use IO::Socket;
my ($host, $port, $kidpid, $handle, $line);
unless (#ARGV == 2) { die "usage: $0 host port" }
($host, $port) = #ARGV;
# create a tcp connection to the specified host and port
$handle = IO::Socket::INET->new(Proto => "tcp",
PeerAddr => $host,
PeerPort => $port)
or die "can't connect to port $port on $host: $!";
$handle->autoflush(1); # so output gets there right away
print STDERR "[Connected to $host:$port]\n";
# split the program into two processes, identical twins
die "can't fork: $!" unless defined($kidpid = fork());
if ($kidpid) {
# parent copies the socket to standard output
while (defined ($line = <$handle>)) {
print STDOUT $line;
}
kill("TERM" => $kidpid); # send SIGTERM to child
}
else {
# child copies standard input to the socket
while (defined ($line = <STDIN>)) {
print $handle $line;
}
}
exit;
See how easy that is?
Fork()'s most popular use is as a way to clone a server for each new client that connect()s (because the new process inherits all file descriptors in whatever state they exist).
But I've also used it to initiate a new (locally running) service on-demand from a client.
That scheme is best done with two calls to fork() - one stays in the parent session until the server is up and running and able to connect, the other (I fork it off from the child) becomes the server and departs the parent's session so it can no longer be reached by (say) SIGQUIT.