I m thinking to use Posix robust mutexes to protect shared resource among different processes (on Linux). However there are some doubts about safety in difference scenarios. I have the following questions:
Are robust mutexes implemented in the kernel or in user code?
If latter, what would happen if a process happens to crash while in a call to pthread_mutex_lock or pthread_mutex_unlock and while a shared pthread_mutex datastructure is getting updated?
I understand that if a process locked the mutex and dies, a thread in another process will be awaken and return EOWNERDEAD. However, what would happen if the process dies (in unlikely case) exactly when the pthread_mutex datastructure (in shared memory) is being updated? Will the mutex get corrupted in that case? What would happen to another process that is mapped to the same shared memory if it were to call a pthread_mutex function?
Can the mutex still be recovered in this case?
This question applies to any pthread object with PTHREAD_PROCESS_SHARED attribute. Is it safe to call functions like pthread_mutex_lock, pthread_mutex_unlock, pthread_cond_signal, etc. concurrently on the same object from different processes? Are they thread-safe across different processes?
From the man-page for pthreads:
Over time, two threading implementations have been provided by the
GNU C library on Linux:
LinuxThreads
This is the original Pthreads implementation. Since glibc
2.4, this implementation is no longer supported.
NPTL (Native POSIX Threads Library)
This is the modern Pthreads implementation. By comparison
with LinuxThreads, NPTL provides closer conformance to the
requirements of the POSIX.1 specification and better
performance when creating large numbers of threads. NPTL is
available since glibc 2.3.2, and requires features that are
present in the Linux 2.6 kernel.
Both of these are so-called 1:1 implementations, meaning that each
thread maps to a kernel scheduling entity. Both threading
implementations employ the Linux clone(2) system call. In NPTL,
thread synchronization primitives (mutexes, thread joining, and so
on) are implemented using the Linux futex(2) system call.
And from man futex(7):
In its bare form, a futex is an aligned integer which is touched only
by atomic assembler instructions. Processes can share this integer
using mmap(2), via shared memory segments or because they share
memory space, in which case the application is commonly called
multithreaded.
An additional remark found here:
(In case you’re wondering how they work in shared memory: Futexes are keyed upon their physical address)
Summarizing, Linux decided to implement pthreads on top of their "native" futex primitive, which indeed lives in the user process address space. For shared synchronization primitives, this would be shared memory and the other processes will still be able to see it, after one process dies.
What happens in case of process termination? Ingo Molnar wrote an article called Robust Futexes about just that. The relevant quote:
Robust Futexes
There is one race possible though: since adding to and removing from the
list is done after the futex is acquired by glibc, there is a few
instructions window for the thread (or process) to die there, leaving
the futex hung. To protect against this possibility, userspace (glibc)
also maintains a simple per-thread 'list_op_pending' field, to allow the
kernel to clean up if the thread dies after acquiring the lock, but just
before it could have added itself to the list. Glibc sets this
list_op_pending field before it tries to acquire the futex, and clears
it after the list-add (or list-remove) has finished
Summary
Where this leaves you for other platforms, is open-ended. Suffice it to say that the Linux implementation, at least, has taken great care to meet our common-sense expectation of robustness.
Seeing that other operating systems usually resort to Kernel-based synchronization primitives in the first place, it makes sense to me to assume their implementations would be even more naturally robust.
Following the documentation from here: http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutexattr_getrobust.html, it does read that in a fully POSIX compliant OS, shared mutex with the robust flag will behave in the way you'd expect.
The problem obviously is that not all OS are fully POSIX compliant. Not even those claiming to be. Process shared mutexes and in particular robust ones are among those finer points that are often not part of an OS's implementation of POSIX.
Related
I tried to figure out if multiple processes or threads can execute concurrent syscalls, without one of them sleeping.
That's to say: Does OpenBSD use something like a Big Kernel Lock.
One would expect, that parallel Kernel access is possible. I tried to look into the syscall interface (code-reading and kernel debugging) and didn't find anything that would strike me as BKL.
However, when I look into the fork syscall implementation, it appears to me, that some global data is accessed without locking (e.g. nprocesses). I was wondering, if the scheduler (?), somehow, prevents parallel syscalls, or if I am overlooking something.
So: Does OpenBSD support parallel Kernel access and how about other BSDs?
Indeed, OpenBSD has a rather archaic model, which uses priority levels, distinct for each subsystem. See spl(9).
The mechanism originally allowed some preemption, but only from higher priority interrupts. On modern implementations of course the priority levels are implemented by mutexes.
The scheduler uses splsched.
So, there's several locks, and syscalls take place in parallel (across different CPUs) but serialize due to these locks at certain points, depending on the subsystem boundaries they're crossing. In other words, never two threads will be running code from the same subsystem concurrently; of course this could change at any moment if a lock is split or replaced.
Other systems:
This is inherited from NetBSD, so it's about the same.
FreeBSD has transitioned to a more granular approach, with some parts being lockless, much like Linux.
DragonflyBSD improves upon FreeBSD by providing serialization tokens for synchronization, and an inherently lockless approach to key mechanisms, like memory allocation (both userspace and kernel).
Before the futex system calls existed in Linux, what underlying system calls were used by threading libraries like pthreads to block/sleep a thread and to subsequently wake those threads from userland?
For example, if a thread tries to acquire a mutex, the userland implementation will block the thread (perhaps after a short spinning interval), but I can't find the syscalls that are used for this (other than futex which are a relatively recent creation).
Before futex and current implementation of pthreads for Linux, the NPTL (require kernel 2.6 and newer), there were two other threading libraries with POSIX Thread API for Linux: linuxthreads and NGPT (which was based on Gnu Pth. LinuxThreads was the only widely used libpthread for years (and it can still be used in some strange & unmaintained micro-libc to work on 2.4; other micro-libc variants may have own builtin implementation of pthread-like API on top of futex+clone). And Gnu Pth is not thread library, it is single process thread with user-level "thread" switching.
You should know that there are several Threading Models when we check does the kernel knows about some or all of user threads (how many CPU cores can be used with adding threads to the program; what is the cost of having the thread / how many threads may be started). Models are named as M:N where M is userspace thread number and N is thread number schedulable by OS kernel:
"1:1" ''kernel-level threading'' - every userspace thread is schedulable by OS kernel. This is implemented in Linuxthreads, NPTL and many modern OS.
"N:1" ''user-level threading'' - userspace threads are planned by the userspace, they all are invisible to the kernel, it only schedules one process (and it may use only 1 CPU core). Gnu Pth (GNU Portable Threads) is example of it, and there are many other implementations for some computer architectures.
"M:N" ''hybrid threading'' - there are some entities visible and schedulable by OS kernel, but there may be more user-space threads in them. And sometimes user-space threads will migrate between kernel-visible threads.
With 1:1 model there are many classic sleep mechanisms/APIs in Unix like select/poll and signals and other variants of IPC APIs. As I remember, Linuxthreads used separate processes for every thread (with fully shared memory) and there was special manager "thread" (process) to emulate some POSIX thread features. Wikipedia says that SIGUSR1/SIGUSR2 were used in Linuxthreads for some internal communication between threads, same says IBM "The synchronization of primitives is achieved by means of signals. For example, threads block until awoken by signals.". Check also the project FAQ http://pauillac.inria.fr/~xleroy/linuxthreads/faq.html#H.4 "With LinuxThreads, I can no longer use the signals SIGUSR1 and SIGUSR2 in my programs! Why?"
LinuxThreads needs two signals for its internal operation. One is used to suspend and restart threads blocked on mutex, condition or semaphore operations. The other is used for thread cancellation.
On ``old'' kernels (2.0 and early 2.1 kernels), there are only 32 signals available and the kernel reserves all of them but two: SIGUSR1 and SIGUSR2. So, LinuxThreads has no choice but use those two signals.
With "N:1" model thread may call some blocking syscall and block everything (some libraries may convert some blocking syscalls into async, or use some SIGALRM or SIGVTALRM magic); or it may call some (very) special internal threading function which will do user-space thread switching by rewriting machine state register (like switch_to in linux kernel, save IP/SP and other regs, restore IP/SP and regs of other thread). So, kernel does not wake any user thread directly from userland, it just schedules whole process; and user space scheduler implement thread synchronization logic (or just calls sched_yield or select when there is no threads to work).
With M:N model things are very complicated... Don't know much about NGPT... There is one paragraph about NGPT in POSIX Threads and the Linux Kernel, Dave McCracken, OLS2002,330 page 5
There is a new pthread library under development called NGPT. This library is based on the GNU Pth library, which is an M:1 library. NGPT extends Pth by using multiple Linux tasks, thus creating an M:N library. It attempts to preserve Pth’s pthread compatibility while also using multiple Linux tasks for concurrency, but this effort is hampered by the underlying differences in the Linux threading model. The NGPT library at present uses non-blocking wrappers around blocking system calls to avoid
blocking in the kernel.
Some papers and posts: POSIX Threads and the Linux Kernel, Dave McCracken, OLS2002,330, LWN post about NPTL 0.1
The futex system call is used extensively in all synchronization
primitives and other places which need some kind of
synchronization. The futex mechanism is generic enough to support
the standard POSIX synchronization mechanisms with very little
effort. ... Futexes also allow the implementation of inter-process
synchronization primitives, a sorely missed feature in the old
LinuxThreads implementation (Hi jbj!).
NPTL design pdf:
5.5 Synchronization Primitives
The implementation of the synchronization primitives such as mutexes, read-write
locks, conditional variables, semaphores, and barriers requires some form of kernel
support. Busy waiting is not an option since threads can have different priorities (beside wasting CPU cycles). The same argument rules out the exclusive use of sched yield. Signals were the only viable solution for the old implementation. Threads would block in the kernel until woken by a signal. This method has severe drawbacks in terms of speed and reliability caused by spurious wakeups and derogation of the quality of the signal handling in the application.
Fortunately some new functionality was added to the kernel to implement all kinds
of synchronization primitives: futexes [Futex]. The underlying principle is simple but
powerful enough to be adaptable to all kinds of uses. Callers can block in the kernel
and be woken either explicitly, as a result of an interrupt, or after a timeout.
Futex stands for "fast userspace mutex." It's simply an abstraction over mutexes which is considered faster and more convenient than traditional mutex mechanisms because it implements the wait system for you. Before and after futex(), threads were put to sleep and awoken via a change in their process state. The process states are:
Running state
Sleeping state
Un-interruptible sleeping state (i.e. blocking for a syscall like read() or write()
Defunct/zombie state
When a thread is suspended, it is put into (interruptible) 'sleep' state. Later, it can be woken via the wake_up() function, which operates on its task structure within the kernel. As far as I can tell, wake_up is a kernel function, not a syscall. The kernel doesn't need a syscall to wake or sleep a task; it (or a process) simply changes the task structure to reflect the state of the process. When the Linux scheduler next deals with that process, it treats it according to its state (again, the states are listed above).
Short story: futex() implements a wait system for you. Without it, you need a data structure that's accessible from the main thread and from the sleeping thread in order to wake up a sleeping thread. All of this is done with userland code. The only thing you might need from the kernel is a mutex--the specifics of which do include locking mechanisms and mutex datastructures, but don't inherently wake or sleep the thread. The syscalls you're looking for don't exist. Essentially, most of what you're talking about can be achieved from userspace, without a syscall, by manually keeping track of data conditions that determine whether and when to sleep or wake a thread.
I have to implement kernel level thread but while searching on the net I found that there are three ways to create kernel level thread in linux:
NPTL
kthread
linuxThreads
It was written somewhere that linuxThreads are now abandoned. But I am unable to find current support of NPTL & kthread. Also I am unable to find any source that can simply explain me how to use their functionality.
Which is the currently supported and good library to use kernel level thread?
Also pls share any resource for installing these library and also using them?
You are confusing two very different definitions of "kernel thread".
LinuxThreads and NPTL are implementations of POSIX pthreads for user-space processes. They use a 1-to-1 mapping of kernel scheduling entities to user-space threads. They are sometimes described as kernel threads implementations only because they create threads that are scheduled by the kernel.
LinuxThreads is unsupported and entirely obsolete. NPTL is now part of glibc, so you already have it. There's nothing special to install. You use these the same way you use any POSIX threading library, with calls to functions like pthread_create.
Actual kernel threads run kernel code. None of those libraries are relevant since they're all user-space libraries. Have a look at functions like kthread_run. There's no magic, no secret. Write kernel code the way similar kernel code is written. (Knowledge and experience in writing kernel code is needed. It's, unfortunately, not simple.)
I assume that; if you really wanted to create a kernel thread, you would already know about these things.
I think, you want to create multi-threaded applications and trying to find info about user-level multi-threading functions.
And yes, these threads you created will be managed by the kernel itself. This is what you are looking for :: POSIX Threads
The title might not be clear enough because I don't know how to define my questions actually.
I understand Pthread is a thread library meeting POSIX standard (about POSIX, see wikipedia: http://en.wikipedia.org/wiki/Posix). It is available in Unix-like OS.
About thread, I read that there are three different models:
User level thread: the kernel does not know it. User himself creates/implements/destroy threads.
Kernel level thread: kernel directly supports multiple threads of control in a process.
Light weight process(LWP): scheduled by kernel but can be bounded with user threads.
Did you see my confusion? When I call pthread_create() to create a thread, did I create a user level thread? I guess so. So can I say, Pthread offers a user level solution for threads? It can not manipulate kernel/LWP?
#paulsm4 I am doubtful about your comment that kernel knows every thing. In this particular context of user level threads, the kernel is unaware of the fact that such a thing is happening. A user level thread's scheduling is maintained by the user himself (via the interface provided by a library) and the kernel ends up allotting just a single kernel thread to the whole process. Kernel would treat the process as a single threaded and any blocking call by one of the threads would end up blocking all the threads of that process.
Refer to http://www.personal.kent.edu/~rmuhamma/OpSystems/Myos/threads.htm
In Linux, pthread is implemented as a lightweight process. Kernel (v2.6+) is actually implemented with NPTL. Let me quote the wiki content:
NPTL is a so-called 1×1 threads library, in that threads created by the user (via the pthread_create() library function) are in 1-1 correspondence with schedulable entities in the kernel (tasks, in the Linux case). This is the simplest possible threading implementation.
So pthread in linux kernel is actually implemented as kernel thread.
pthreads, per se, isn't really a threading library. pthreads is the interface which a specific threading library implements, using the concurrency resources available on that platform. So there's a pthreads implementation on linux, on bsd, on solaris, etc., and while the interface (the header files and the meaning of the calls) is the same, the implementation of each is different.
So what pthread_create actually does, in terms of kernel thread objects, varies between OSes and pthread library implementations. At a first approximation, you don't need to know (that's stuff that the pthread abstraction allows you to not need to know about). Eventually you might need to see "behind the curtain", but for most pthread users that's not necessary.
If you want to know what a /specific/ pthread implementation does, on a specific OS, you'll need to clarify your question. What Solaris and Linux do, for example, is very different.
Q: I understand Pthread is a thread library meeting POSIX standard
A: Yes. Actually, "Pthreads" stands for "Posix threads":
http://en.wikipedia.org/wiki/Pthreads
Q: It is available in Unix-like OS.
A: Actually, it's available for many different OSs ... including Windows, MacOS ... and, of course, Linux, BSD and Solaris.
Q: About thread, I read that there are three different models
Now you're getting fuzzy. "Threads" is a very generic term. There are many, many different models. And many, many different ways you can characterize and/or implement "threads". Including stuff like the Java threading model, or the Ada threading model.
Q: When I call pthread_create() to create a thread, did I create a
user level thread?
A: Yes: Just about everything you do in user space is "protected" in your own, private "user space".
Q: User level thread: the kernel does not know it.
A: No. The kernel knows everything :)
Q: Kernel level thread: kernel directly supports multiple threads of
control in a process.
A: Yes, there is such a thing as "kernel threads".
And, as it happens, Linux makes EXTENSIVE use of kernel threads. For example, every single process in a Linux system is a "kernel thread". And every user-created pthread is ALSO implemented as a new "kernel thread". As are "worker threads" (which are completely invisible to any user-level process).
But this is an advanced topic you do NOT need to understand in order to effectively use pthreads. Here's a great book that discussed this - and many other topics - in detail:
Linux Kernel Development, Robert Love
Remember: "Pthreads" is an interface. How it's implemented depends on the platform. Linux uses kernel threads; Windows uses Win32 threads, etc.
===========================================================================
ADDENDUM:
Since people still seem to be hitting this old thread, I thought it would be useful to reference this post:
https://stackoverflow.com/a/11255174/421195
Linux typically uses two implementations of pthreads:
LinuxThreads and Native
POSIX Thread Library(NPTL),
although the former is largely obsolete. Kernel from 2.6 provides
NPTL, which provides much closer conformance to SUSv3, and perform
better especially when there are many threads.
You can query the
specific implementation of pthreads under shell using command:
getconf GNU_LIBPTHREAD_VERSION
You can also get a more detailed implementation difference in The
Linux Programming Interface.
"Pthreads" is a library, based on the Posix standard. How a pthreads library is implemented will differ from platform to platform and library to library.
I find previous answers not as satisfying or clear as I would have liked so here goes:
When you call
pthread_create(...)
you always create a new user-level thread. And assuming that there is OS, there is always one or more kernel thread...but let's dive deeper:
According to "Operating system concepts" 10th edition,the actual classification we should be looking at (when it comes to thread libraries) is how the user level threads are mapped onto kernel threads (and that's what the question really meant).
The models are one to one (each user-level thread within a single process is mapped to a different kernel thread),many to one (the thread library is "user level" so all of the different threads within a single process are mapped to a single kernel thread,and the threads data structures, context switch etc are dealt with at user level and not by the OS [meaning that if a thread blocks on some I/O call, the entire process might potentially block]), and many to many (something in between,obviously the number of user-level threads is greater or equal to the number of kernel threads it is being mapped onto).
Now,pthreads is a specification and not an implementation, and the implementation does depend on the OS to which it is written. It could be any one of those models (notice that "many to many" is very flexible).
So,as an example,on Linux and Windows (the most popular OSs for years now,where the model is "one to one") the implementation is "one to one".
Pthreads is just a standardized interface for threading libraries. Whether an OS thread or a lightweight thread is created depends on the library you use. Nevertheless, my first guest would be that your threads are “real” OS-level threads.
I'm monitoring a process with strace/ltrace in the hope to find and intercept a call that checks, and potentially activates some kind of globally shared lock.
While I've dealt with and read about several forms of interprocess locking on Linux before, I'm drawing a blank on what to calls to look for.
Currently my only suspect is futex() which comes up very early on in the process' execution.
Update0
There is some confusion about what I'm after. I'm monitoring an existing process for calls to persistent interprocess memory or equivalent. I'd like to know what system and library calls to look for. I have no intention call these myself, so naturally futex() will come up, I'm sure many libraries will implement their locking calls in terms of this, etc.
Update1
I'd like a list of function names or a link to documentation, that I should monitor at the ltrace and strace levels (and specifying which). Any other good advice about how to track and locate the global lock in mind would be great.
If you can start monitored process in valgrind, then there are two projects:
http://code.google.com/p/data-race-test/wiki/ThreadSanitizer
and Helgrind
http://valgrind.org/docs/manual/hg-manual.html
Helgrind is aware of all the pthread
abstractions and tracks their effects
as accurately as it can. On x86 and
amd64 platforms, it understands and
partially handles implicit locking
arising from the use of the LOCK
instruction prefix.
So, this tools can detect even atomic memory accesses. And they will check pthread usage
flock is another good one
There are many system calls can be used for locking: flock, fcntl, and even create.
When you are using pthreads/sem_* locks they may be executed in user space so you'll never
see them in strace as futex is called only for pending operations. Like when you actually
need to wait.
Some operations can be done in user space only - like spinlocks - you'll never see them
unless they do some waits for timer - backoff so you may see only stuff like nanosleep when one lock waits for other.
So there is no "generic" way to trace them.
on systems with glibc ~ >= 2.5 (glibc + nptl) you can use process shared
semaphores (last parameter to sem_init), more precisely, posix unnamed semaphores
posix mutexes (with PTHREAD_PROCESS_SHARED to pthread_mutexattr_setpshared)
posix named semaphores (got from sem_open/sem_unlink)
system v (sysv) semaphores: semget, semop
On older systems with glibc 2.2, 2.3 with linuxthreads or on embedded systems with uClibc you can use ONLY system v (sysv) semaphores for iterprocess communication.
upd1: any IPC and socker must be checked.