Almost all the code and tutorials that I have read online so far involve using mutexes and semaphores for synchronisation amongst threads. Can they be used to synchronise amongst processes?
I'd like to write code that looks like this:
void compute_and_print() {
// acquire mutex
// critical section
// release mutex
}
void main() {
int pid = fork();
if ( pid == 0 ) {
// do something
compute_and_print();
}
else {
// do something
compute_and_print();
}
}
Could someone point me towards similar code that does this?
I understand that different processes have different address spaces, but I wonder if the above would be different address spaces, however, wouldn't the mutex refer to the same kernel object?
Yes, it is possible. There are many ways to synchronize different processes. Perhaps the most popular solutions for mutual exclusion in this field are System V IPC semaphores and atomic operations on shared memory. I recommend you read chapter 5 of David A Ruslin's book called Interprocess Communication Mechanisms, or better yet - the whole book.
As for your second question, most modern operating systems on commodity hardware would place processes in different address spaces, though it is also possible for processes to share the same address space (see Virtual Memory, Memory Protection). Either way, if IPC mechanism is handled by the kernel, then two processes would refer to the same "kernel object", as you said. In cases where mutual exclusion is implemented (almost) without the kernel (like spin locks of some sort that use "shared memory"), both processes would refer to the same physical memory even though their virtual addresses for that memory might be different.
Hope it helps. Good Luck!
Just to be clear, POSIX (and linux) support two separate families of semaphores that have two different interfaces and methods of usage.
There is the older SysV semaphores consisting of semget, semop, semctl and (somewhat optionally) ftok.
The more modern "posix" semaphores consist of sem_open/sem_init, sem_wait, sem_post, sem_close and sem_unlink.
The setup/usage regimes and capabilities are different enough that it is worth familiarizing yourself with both to see what is better for your use case.
You can also use process shared mutexes from the pthreads package.
Sounds like you're looking for System V IPC You would probably use a semaphore to synchronize between processes.
Here is a nice introduction to Sys V IPC in Linux
You'll want named mutex/semaphore's. Take a look at this StackOverflow answer that describes some points. Here's an IBM describing the use of pthread_mutexattr_setname_np. One thing to note that named mutex's in Linux are not 100% portable (i.e. it MIGHT work on Ubuntu but NOT on CentOS, etc. etc.), so you'll need to be sure your platform supports it. If a named mutex is not available on your system, you could use named pipes with some wait conditions or even local sockets. As another answer points out is SysV IPC.
The biggest question you'll need to answer first is how 'cross-linux-platform' compatible do you want your app to be.
Related
From APUE
POSIX semaphores are available in two flavors: named and unnamed. They differ
in how they are created and destroyed, but otherwise work the same. Unnamed
semaphores exist in memory only and require that processes have access to the memory
to be able to use the semaphores. This means they can be used only by threads in the
same process or threads in different processes that have mapped the same memory
extent into their address spaces. Named semaphores, in contrast, are accessed by name
and can be used by threads in any processes that know their names.
Unnamed semaphores "can be used only by threads in the
same process or threads in different processes that have mapped the same memory
extent into their address spaces", because "Unnamed semaphores exist in memory only".
What is the reason that named semaphores are usable by threads in any processes that know their names?
Thanks.
From man page of sem_overview:
On Linux, named semaphores are created in a virtual file system,
normally mounted under /dev/shm, with names of the form sem.somename
So those are accessible for 'threads in any processes' similar way than normal files are.
pthread library may then map those files to memory.
You're thinking about this backwards. The question is: "if I need to synchronize use of a shared resource between teo unrelated processes, hiw do I do it?" And the answer is "you can give a semaphore a name, and then it's not restricted to use in processes which share memory."
Why is that even useful? Well, the use cases may not be common -- perhaps you've never run into one -- but they certainly exist. There are lots of resources which are shared between unrelated processes: databases, configuration files, serial ports, printer queues, and many more. You can mediate between shared uses of these resources with lock files, but it's clunky and you end up reinventing the wheel on every project. Semaphores, on the other hand, are easy to use and have well-defined documented semantics.
However, most uses of semaphores are indeed between related processes which share memory. And you wouldn't want to unnecessarily pay the overhead for maintaining a name in a filesystem.
So we end up with two kinds of semaphores: cheap low-overhead ones which serve the most frequent use cases, and heavier higher-overhead ones which can be used more generally. The nice thing is that the semantics and API are very similar, so you don't need to learn a whole new set of concepts when you start using named semaphores.
The title might not be clear enough because I don't know how to define my questions actually.
I understand Pthread is a thread library meeting POSIX standard (about POSIX, see wikipedia: http://en.wikipedia.org/wiki/Posix). It is available in Unix-like OS.
About thread, I read that there are three different models:
User level thread: the kernel does not know it. User himself creates/implements/destroy threads.
Kernel level thread: kernel directly supports multiple threads of control in a process.
Light weight process(LWP): scheduled by kernel but can be bounded with user threads.
Did you see my confusion? When I call pthread_create() to create a thread, did I create a user level thread? I guess so. So can I say, Pthread offers a user level solution for threads? It can not manipulate kernel/LWP?
#paulsm4 I am doubtful about your comment that kernel knows every thing. In this particular context of user level threads, the kernel is unaware of the fact that such a thing is happening. A user level thread's scheduling is maintained by the user himself (via the interface provided by a library) and the kernel ends up allotting just a single kernel thread to the whole process. Kernel would treat the process as a single threaded and any blocking call by one of the threads would end up blocking all the threads of that process.
Refer to http://www.personal.kent.edu/~rmuhamma/OpSystems/Myos/threads.htm
In Linux, pthread is implemented as a lightweight process. Kernel (v2.6+) is actually implemented with NPTL. Let me quote the wiki content:
NPTL is a so-called 1×1 threads library, in that threads created by the user (via the pthread_create() library function) are in 1-1 correspondence with schedulable entities in the kernel (tasks, in the Linux case). This is the simplest possible threading implementation.
So pthread in linux kernel is actually implemented as kernel thread.
pthreads, per se, isn't really a threading library. pthreads is the interface which a specific threading library implements, using the concurrency resources available on that platform. So there's a pthreads implementation on linux, on bsd, on solaris, etc., and while the interface (the header files and the meaning of the calls) is the same, the implementation of each is different.
So what pthread_create actually does, in terms of kernel thread objects, varies between OSes and pthread library implementations. At a first approximation, you don't need to know (that's stuff that the pthread abstraction allows you to not need to know about). Eventually you might need to see "behind the curtain", but for most pthread users that's not necessary.
If you want to know what a /specific/ pthread implementation does, on a specific OS, you'll need to clarify your question. What Solaris and Linux do, for example, is very different.
Q: I understand Pthread is a thread library meeting POSIX standard
A: Yes. Actually, "Pthreads" stands for "Posix threads":
http://en.wikipedia.org/wiki/Pthreads
Q: It is available in Unix-like OS.
A: Actually, it's available for many different OSs ... including Windows, MacOS ... and, of course, Linux, BSD and Solaris.
Q: About thread, I read that there are three different models
Now you're getting fuzzy. "Threads" is a very generic term. There are many, many different models. And many, many different ways you can characterize and/or implement "threads". Including stuff like the Java threading model, or the Ada threading model.
Q: When I call pthread_create() to create a thread, did I create a
user level thread?
A: Yes: Just about everything you do in user space is "protected" in your own, private "user space".
Q: User level thread: the kernel does not know it.
A: No. The kernel knows everything :)
Q: Kernel level thread: kernel directly supports multiple threads of
control in a process.
A: Yes, there is such a thing as "kernel threads".
And, as it happens, Linux makes EXTENSIVE use of kernel threads. For example, every single process in a Linux system is a "kernel thread". And every user-created pthread is ALSO implemented as a new "kernel thread". As are "worker threads" (which are completely invisible to any user-level process).
But this is an advanced topic you do NOT need to understand in order to effectively use pthreads. Here's a great book that discussed this - and many other topics - in detail:
Linux Kernel Development, Robert Love
Remember: "Pthreads" is an interface. How it's implemented depends on the platform. Linux uses kernel threads; Windows uses Win32 threads, etc.
===========================================================================
ADDENDUM:
Since people still seem to be hitting this old thread, I thought it would be useful to reference this post:
https://stackoverflow.com/a/11255174/421195
Linux typically uses two implementations of pthreads:
LinuxThreads and Native
POSIX Thread Library(NPTL),
although the former is largely obsolete. Kernel from 2.6 provides
NPTL, which provides much closer conformance to SUSv3, and perform
better especially when there are many threads.
You can query the
specific implementation of pthreads under shell using command:
getconf GNU_LIBPTHREAD_VERSION
You can also get a more detailed implementation difference in The
Linux Programming Interface.
"Pthreads" is a library, based on the Posix standard. How a pthreads library is implemented will differ from platform to platform and library to library.
I find previous answers not as satisfying or clear as I would have liked so here goes:
When you call
pthread_create(...)
you always create a new user-level thread. And assuming that there is OS, there is always one or more kernel thread...but let's dive deeper:
According to "Operating system concepts" 10th edition,the actual classification we should be looking at (when it comes to thread libraries) is how the user level threads are mapped onto kernel threads (and that's what the question really meant).
The models are one to one (each user-level thread within a single process is mapped to a different kernel thread),many to one (the thread library is "user level" so all of the different threads within a single process are mapped to a single kernel thread,and the threads data structures, context switch etc are dealt with at user level and not by the OS [meaning that if a thread blocks on some I/O call, the entire process might potentially block]), and many to many (something in between,obviously the number of user-level threads is greater or equal to the number of kernel threads it is being mapped onto).
Now,pthreads is a specification and not an implementation, and the implementation does depend on the OS to which it is written. It could be any one of those models (notice that "many to many" is very flexible).
So,as an example,on Linux and Windows (the most popular OSs for years now,where the model is "one to one") the implementation is "one to one".
Pthreads is just a standardized interface for threading libraries. Whether an OS thread or a lightweight thread is created depends on the library you use. Nevertheless, my first guest would be that your threads are “real” OS-level threads.
what are the different types snchronization methods ?(other than semaphore,mutex,binary semaphore)
I'm assuming that you're talking about application development and not Kernel development.
Linux supports Posix synchronization techniques. Some common ones are a mutex (for multi-threaded applications) and semaphores.
You can use these in many ways... An example of semaphore usage would be multiple processes and keeping a semaphore in shared memory.
I recommend you do some reading of the following manpages:
sem_init
pthread_mutex_init
Also, here's a good discussion on the usage of file locking
Hope that helps.
I have two processes which access to the same physical memory(GPIO data addr).
So how can I have synchronized between these apps?
I understand that we have some kind of locking mechanism such as mutex and semaphore, so which method is the fastest?
Thank for your help,
-nm
Mutexes and semaphores are generally considered to be concurrency solutions in the same address space -- meaning that different parts of the same program will lock their access to a resource using one of these contraptions.
When you're dealing with separate processes, the standard way to do this on Linux is to create something in /var/lock, like /var/lock/myapp.lock, and place your PID followed by a newline in it. Then other processes will check for its existence, and if you're crafty check the PID to make sure it's still alive, too.
If you need real-time access to the area, skip the filesystem and the processes will have to communicate via IPC (LET_ME_KNOW_WHEN_DONE, OKAY_IM_DONE, you get the idea), or -- better -- write a process whose sole purpose is to read and write to the GPIO memory, and your other programs communicate with it via IPC (probably the best approach).
mutex means mutual exclusion -- a semaphore is just a variable used to determine if the resource is in use. In windows, there is a Mutex object that can be created to protect a shared resource.
The issue is what language are you using? What OS (I am assuming linux). Most languages provide support for multi-threading and mutual exclusion, and you should use the built-in constructs.
For example, using C on Linux, you might want to
include semaphore.h
and look up the calls for sem_init, sem_wait etc.
I'm writing a Linux application which observes other applications and tracks consumption of resources . I'm planning work with Java, but programming language isn't important for me. The Goal is important, so I can switch to another technology or use modules. My application runs any selected third party application as child process. Mostly child software solves some algorithm like graphs, string search, etc. Observer program tracks child's resources while it ends the job.
If child application is multi-threaded, maybe somehow is possible to track how much resources consumes each thread? Application could be written using any not distributive-memory threads technology: Java threads, Boost threads, POSIX threads, OpenMP, any other.
In modern Linux systems (2.6), each thread has a separate identifier that has nearly the same treatment as the pid. It is shown in the process table (at least, in htop program) and it also has its separate /proc entry, i.e. /proc/<tid>/stat.
Check man 5 proc and pay particular attention to stat, statm, status etc. You should find the information you're interested in there.
An only obstacle is to obtain this thread identifier. It is different with the process id! I.e. getpid() calls in all threads return the same value. To get the actual thread identifier, you should use (within a C program):
pid_t tid = syscall(SYS_gettid);
By the way, java virtual machine (at least, its OpenJDK Linux implementation) does that internally and uses it for debugging purposes in its back-end, but doesn't expose it to the java interface.
Memory is not allocated to threads, and often shared across threads. This makes it generally impossible and at least meaningless to talk about the memory consumption of a thread.
An example could be a program with 11 threads; 1 creating objects and 10 using those objects. Most of the work is done on those 10 threads, but all memory was allocated on the one thread that created the objects. Now how does one account for that?
If you're willing to use Perl take a look at this: Sys-Statistics-Linux
I used it together with some of the GD graphing packages to generate system resource usage graphs for various processes.
One thing to watch out for - you'll really need to read up on /proc and understand jiffies - last time I looked they're not documented correctly in the man pages, you'll need to read kernel source probably:
http://lxr.linux.no/#linux+v2.6.18/include/linux/jiffies.h
Also, remember that in Linux the only difference between a thread and process is that threads share memory - other than that they're identical in how the kernel implements them.