In Linux, I have a scenario where two threads execute a critical section, one acquires the lock (thread A) and the other(thread B) will wait for the lock. Later threadA releases the mutex lock. I am trying to understand how threadB will be moved to the running state and acquire the lock? How threadB(or operating system) knows that the lock is released by threadA?
I have a theory, please correct if I am wrong. threadB enters TASK_INTERRUPTABLE (blocked at the mutex and so waiting) state and it receives signal when threadA unlocks the mutex so it comes back to the running queue(TASK_RUNNING).
The Linux mutex struct keeps track of the current owner of the mutex (if any):
struct mutex {
atomic_long_t owner;
// ...
There's also a struct to keep track of what other tasks are waiting on a mutex:
/*
* This is the control structure for tasks blocked on mutex,
* which resides on the blocked task's kernel stack:
*/
struct mutex_waiter {
struct list_head list;
struct task_struct *task;
struct ww_acquire_ctx *ww_ctx;
#ifdef CONFIG_DEBUG_MUTEXES
void *magic;
#endif
};
Simplifying quite a bit, when you unlock a mutex, the kernel looks at what other tasks are waiting on that mutex. It picks one of them to become the owner, sets the mutex's owner field to refer to the selected task, and removes that task from the list of tasks waiting for the mutex. At that point, there's at least a good chance that task has become un-blocked, in which case it'll be ready to run once it's unblocked. At that point, it's up to the scheduler to decide when to run it.
Optimization
Since mutexes are used a lot, and they get locked and unlocked quite a bit, they use some optimization to help speed. For example, consider the following:
/*
* #owner: contains: 'struct task_struct *' to the current lock owner,
* NULL means not owned. Since task_struct pointers are aligned at
* at least L1_CACHE_BYTES, we have low bits to store extra state.
*
* Bit0 indicates a non-empty waiter list; unlock must issue a wakeup.
* Bit1 indicates unlock needs to hand the lock to the top-waiter
* Bit2 indicates handoff has been done and we're waiting for pickup.
*/
#define MUTEX_FLAG_WAITERS 0x01
#define MUTEX_FLAG_HANDOFF 0x02
#define MUTEX_FLAG_PICKUP 0x04
#define MUTEX_FLAGS 0x07
So, when you ask the kernel to unlock a mutex, it can "glance" at one bit in the owner pointer to figure out whether this is a "simple" case (nobody's waiting on the mutex, so just mark it as unlocked, and off we go), or a more complex one (at least one task is waiting on the mutex, so a task needs to be selected to be unblocked, and be marked as the new owner of the mutex.
References
https://github.com/torvalds/linux/blob/master/include/linux/mutex.h
https://github.com/torvalds/linux/blob/master/kernel/locking/mutex.c
Disclaimer
The code extracts above are (I believe) current as I write this answer. But as noted above, mutexes get used a lot. If you look at the code for a mutex 5 or 10 years from now, chances are you'll find that somebody has done some work on optimizing the code, so it may not precisely match what I've quoted above. Most of the concepts are likely to remain similar, but changes in details (especially the optimizations) are to be expected.
I have some doubts about the C++11/C11 memory model that I was wondering if anyone can clarify. These are questions about the model/abstract machine, not about any real architecture.
Are acquire/release effects guaranteed to "cascade" from one thread to the next?
Here is a pseudo code example of what I mean (assume all variables start as 0)
[Thread 1]
store_relaxed(x, 1);
store_release(a, 1);
[Thread 2]
while (load_acquire(a) == 0);
store_release(b, 1);
[Thread 3]
while (load_acquire(b) == 0);
assert(load_relaxed(x) == 1);
Thread 3's acquire syncs with Thread 2's release, which comes after Thread 2's acquire which syncs with Thread 1's release. Therefore, Thread 3 is guaranteed to see the value that Thread 1 set to x, correct? Or do we need to use seq cst here in order to be guaranteed that the assert will not fire? I have a feeling acquire/release is enough, but I can't quite find any simple explanation that guarantees it. Most explanations of acquire/release mainly focus on the acquiring thread receiving all the stores made by the releasing thread. However in the example above, Thread 2 never touches variable x, and Thread 1/Thread 3 do not touch the same atomic variable. It's obvious that if Thread 2 were to load x, it would see 1, but is that state guaranteed to cascade over into other threads which subsequently do an acquire/release sync with Thread 2? Or does Thread 3 also need to do an acquire on variable a in order to receive Thread 1's write to x?
According to https://en.cppreference.com/w/cpp/atomic/memory_order:
All writes in the current thread are visible in other threads that acquire the same atomic variable
All writes in other threads that release the same atomic variable are visible in the current thread
Since Thread 1 and Thread 3 don't touch the same atomic variable, I'm not sure if acquire/release alone is enough for the above case. There's probably an answer hiding in the formal description, but I can't quite work it out.
*EDIT: Didn't notice until after the fact, but there is an example at the link I posted ("The following example demonstrates transitive release-acquire ordering...") that is almost the same as my example, but it uses the same atomic variable across all three threads, which seems like it might be significant. I am specifically asking about the case where the variables are not the same.
Am I right in believing that according to the standard, there must always be a pair of non-relaxed atomic operations, one in each thread, in order for any kind of memory ordering at all to be guaranteed?
Imagine there is a function "get_data" that allocates a buffer, writes some data to it, and returns a pointer to the buffer. And there is a function "use_data" that takes the pointer to the buffer and does something with the data. Thread 1 gets a buffer from get_data and passes it to Thread 2 using a relaxed atomic store to a global atomic pointer. Thread 2 does relaxed atomic loads in a loop until it gets the pointer, and then passes it off to use_data:
int* get_data() {...}
void use_data(int* buf) {...}
int* global_ptr = nullptr;
[Thread 1]
int* buf = get_data();
super_duper_memory_fence();
store_relaxed(global_ptr, buf);
[Thread 2]
int* buf = nullptr;
while ((buf = load_relaxed(global_ptr)) == nullptr);
use_data(buf);
Is there any kind of operation at all that can be put in "super_duper_memory_fence", that will guarantee that by the time use_data gets the pointer, the data in the buffer is also visible? It is my understanding that there is not a portable way to do this, and that Thread 2 must have a matching fence or other atomic operation in order to guarantee that it receives the writes made into the buffer and not just the pointer value. Is this correct?
Thread 3's acquire syncs with Thread 2's release, which comes after Thread 2's acquire which syncs with Thread 1's release. Therefore, Thread 3 is guaranteed to see the value that Thread 1 set to x, correct?
Yes, this is correct. The acquire/release operations establish synchronize-with relations - i.e., store_release(a) synchronizes-with load_acquire(a) and store_release(b) synchronizes-with load_acquire(b). And load_acquire(a) is sequenced-before store_release(b). synchronize-with and sequenced-before are both part of the happens-before definition, and the happens-before relation is transitive. Therefore, store_relaxed(x, 1) happens-before load_relaxed(x).
Am I right in believing that according to the standard, there must always be a pair of non-relaxed atomic operations, one in each thread, in order for any kind of memory ordering at all to be guaranteed?
This is question is a bit too broad, but overall I would tend to say "yes". In general you have to ensure that there is a proper happens-before relation when operating on some (non-atomic) shared data. If one thread writes to some shared data and some other thread should read that data, you have to ensure that the write happens-before the read. There are different ways to achieve this - atomics with the correct memory orderings are just one way (although one could argue that almost all other methods (like std::mutex) also boil down to atomic operations).
Fences also have to be combined with other fences or atomic operations. Your example would work if super_duper_memory_fence() were a std::atomic_thread_fence(std::memory_order_release) and you put another std::atomic_thread_fence(std::memory_order_acquire) before your call to use_data.
For more details I can recommend this paper which I have co-authored: Memory Models for C/C++ Programmers
OK. The example here is provided using pthread lib in c.
In textbook I came across the following code:
//for thread 2
pthread_mutex_lock(&lock);
should_wake_up = 1;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);
This code works out pretty fine. I just wonder will the following code also works?
//for thread 2
pthread_mutex_lock(&lock);
should_wake_up = 1;
pthread_mutex_unlock(&lock);
pthread_cond_signal(&cond);//signal the conditional variable but the lock is not held
What's the pros and cons for the following code?
PS. Suppose the cooperating thread has the code:
//for thread 1
pthread_mutex_lock(&lock);
while(!should_wake_up)
pthread_cond_wait(&cond, &lock);
pthead_mutex_unlock(&lock);
PS2. I came across some other question, which points out that if we don't want signal being lost, we must use lock to make sure that the associated predicate (in this case, is should_wake_up) can not be changed when the lock is held in thread 1. In this case, this seems not to be the issue. Link to the post: [1]: signal on condition variable without holding lock. I think his issue is that he forget about locking. But my question is different.
For normal usage, you can unlock the mutex before signalling a condition variable.
The mutex protects the shared state (in this case the should_wake_up flag). Provided the mutex is locked when modifying the shared state, and when checking the shared state, pthread_cond_signal can be called without the mutex locked and everything will work as expected.
Under most circumstances, I would recommend calling pthread_cond_signal after calling pthread_mutex_unlock as a slight performance improvement. If you call pthread_cond_signal before pthread_mutex_unlock then the waiting thread can be woken by the signal before the mutex is unlocked, so the waiting thread then has to go back to sleep, as it blocks on the mutex that is still held by the signalling thread.
As per multithreading concepts, I have learned so far:
MUTEX: A mutex_t object can be used for managing access to a resource.
BINARY SEMAPHORE: A sem_t object can also be used to manage access to a resource
Differenrce between two: The concept of ownership i.e. in case of mutex_t the thread which locked the mutex_t, only it can unlock it. But in case of sem_t, there is no concpt of ownership and hence any thread can perform sem_post() on the sem_t object. This is the reson it can be used as Event signals.
Now suppose my crital section appears as:
A) Using mutex_t
typedef struct {
int count;
pthread_mutex_t mutex;
}counter;
counter count;
void increment(counter* c)
{
pthread_mutex_lock(&(c->mutex));
(c->count)++;
pthread_mutex_unlock(&(c->mutex));
}
B) Using sem_t
EDIT: (Binary semaphore i.e. initialized to 1)
typedef struct {
int count;
sem_t sem;
}counter;
counter count;
void increment(counter* c)
{
sem_wait(&(c->sem));
(c->count)++;
sem_post(&(c->sem));
}
In case of B) till the time sem is zero no thread can enter the critical section and hence providing access control. But suppose due to some event sem_post() is executed by some other thread then it will allow access to critical section by other threads.
In this case this is actually a buggy situation and not a proper a access control.And hence programmer has to be careful with use of binary semaphore for resource access control.
I can conclude, it always better to use mutex_t for access control and binary semaphore for event signalling.
Please let me know if my understanding is correct or am I missing something?
Mutexes are very specific in their purpose. Like you said, they can only ever be released by the thread currently holding the mutex, and they only allow single entrancy. A mutex is effectively a semaphore initialized to a count of 1 that also verifies that the thread calling post is the thread that last called wait ('ownership' invariance).
Your example does not show the initalization conditions for your semaphore. If you wanted to use it in the same way as a mutex, it would have to be initialized to a count of 1.
Semaphores do have a wide range of uses, so I wouldn't call them 'unsafe'. For example, lets say that you have some resource that allows, say, 5 total consumers to be running. So you would protect that resource with a semaphore initialized to 5. As consumers invoke the resource, the semaphore will tick down until it hits zero, at which point it'll block new consumers until running consumers increment the semaphore. This is called a counting semaphore.
A great example of how to use a counting semaphore is a blocking queue - a simple Queue that has an Enqueue and a Dequeue method. Each Enqueue increments the semaphore, and each Dequeue decrements the semaphore. In this way, if the queue is empty, Dequeuers will be blocked.
Another example of how a semaphore could be used would be in a simple signaling situation, as you mention:
Thread A enqueues 10 jobs into a thread pool to run them in parallel. Each job has a semaphore associated with it that is initially set to 0. When the job has been completed by the thread pool, the thread pool posts to the jobs semaphore. Meanwhile, Thread A is waiting on each job's semaphore to find out when they complete.
So we see many uses for a semaphore:
Making Mutexs, where the initial count = 1
Protecting limited resources, where the initial count = N, the limit
Counting queued work, where the initial count is = 0 and grows with queued work
Signaling job completion, where the initial count = 0, and the semaphore is used to coordinate two threads.
I would remind you that locking in general is a topic that requires close attention - in systems that have complex interactions, its sometimes very easy to use too little locking, causing unintended concurrency, or too much locking, causing deadlocks.
When should we use mutex and when should we use semaphore ?
Here is how I remember when to use what -
Semaphore:
Use a semaphore when you (thread) want to sleep till some other thread tells you to wake up. Semaphore 'down' happens in one thread (producer) and semaphore 'up' (for same semaphore) happens in another thread (consumer)
e.g.: In producer-consumer problem, producer wants to sleep till at least one buffer slot is empty - only the consumer thread can tell when a buffer slot is empty.
Mutex:
Use a mutex when you (thread) want to execute code that should not be executed by any other thread at the same time. Mutex 'down' happens in one thread and mutex 'up' must happen in the same thread later on.
e.g.: If you are deleting a node from a global linked list, you do not want another thread to muck around with pointers while you are deleting the node. When you acquire a mutex and are busy deleting a node, if another thread tries to acquire the same mutex, it will be put to sleep till you release the mutex.
Spinlock:
Use a spinlock when you really want to use a mutex but your thread is not allowed to sleep.
e.g.: An interrupt handler within OS kernel must never sleep. If it does the system will freeze / crash. If you need to insert a node to globally shared linked list from the interrupt handler, acquire a spinlock - insert node - release spinlock.
A mutex is a mutual exclusion object, similar to a semaphore but that only allows one locker at a time and whose ownership restrictions may be more stringent than a semaphore.
It can be thought of as equivalent to a normal counting semaphore (with a count of one) and the requirement that it can only be released by the same thread that locked it(a).
A semaphore, on the other hand, has an arbitrary count and can be locked by that many lockers concurrently. And it may not have a requirement that it be released by the same thread that claimed it (but, if not, you have to carefully track who currently has responsibility for it, much like allocated memory).
So, if you have a number of instances of a resource (say three tape drives), you could use a semaphore with a count of 3. Note that this doesn't tell you which of those tape drives you have, just that you have a certain number.
Also with semaphores, it's possible for a single locker to lock multiple instances of a resource, such as for a tape-to-tape copy. If you have one resource (say a memory location that you don't want to corrupt), a mutex is more suitable.
Equivalent operations are:
Counting semaphore Mutual exclusion semaphore
-------------------------- --------------------------
Claim/decrease (P) Lock
Release/increase (V) Unlock
Aside: in case you've ever wondered at the bizarre letters (P and V) used for claiming and releasing semaphores, it's because the inventor was Dutch. In that language:
Probeer te verlagen: means to try to lower;
Verhogen: means to increase.
(a) ... or it can be thought of as something totally distinct from a semaphore, which may be safer given their almost-always-different uses.
It is very important to understand that a mutex is not a semaphore with count 1!
This is the reason there are things like binary semaphores (which are really semaphores with count 1).
The difference between a Mutex and a Binary-Semaphore is the principle of ownership:
A mutex is acquired by a task and therefore must also be released by the same task.
This makes it possible to fix several problems with binary semaphores (Accidental release, recursive deadlock, and priority inversion).
Caveat: I wrote "makes it possible", if and how these problems are fixed is up to the OS implementation.
Because the mutex has to be released by the same task it is not very good for the synchronization of tasks. But if combined with condition variables you get very powerful building blocks for building all kinds of IPC primitives.
So my recommendation is: if you got cleanly implemented mutexes and condition variables (like with POSIX pthreads) use these.
Use semaphores only if they fit exactly to the problem you are trying to solve, don't try to build other primitives (e.g. rw-locks out of semaphores, use mutexes and condition variables for these)
There is a lot of misunderstanding between mutexes and semaphores. The best explanation I found so far is in this 3-Part article:
Mutex vs. Semaphores – Part 1: Semaphores
Mutex vs. Semaphores – Part 2: The Mutex
Mutex vs. Semaphores – Part 3 (final part): Mutual Exclusion Problems
While #opaxdiablo answer is totally correct I would like to point out that the usage scenario of both things is quite different. The mutex is used for protecting parts of code from running concurrently, semaphores are used for one thread to signal another thread to run.
/* Task 1 */
pthread_mutex_lock(mutex_thing);
// Safely use shared resource
pthread_mutex_unlock(mutex_thing);
/* Task 2 */
pthread_mutex_lock(mutex_thing);
// Safely use shared resource
pthread_mutex_unlock(mutex_thing); // unlock mutex
The semaphore scenario is different:
/* Task 1 - Producer */
sema_post(&sem); // Send the signal
/* Task 2 - Consumer */
sema_wait(&sem); // Wait for signal
See http://www.netrino.com/node/202 for further explanations
See "The Toilet Example" - http://pheatt.emporia.edu/courses/2010/cs557f10/hand07/Mutex%20vs_%20Semaphore.htm:
Mutex:
Is a key to a toilet. One person can have the key - occupy the toilet - at the time. When finished, the person gives (frees) the key to the next person in the queue.
Officially: "Mutexes are typically used to serialise access to a section of re-entrant code that cannot be executed concurrently by more than one thread. A mutex object only allows one thread into a controlled section, forcing other threads which attempt to gain access to that section to wait until the first thread has exited from that section."
Ref: Symbian Developer Library
(A mutex is really a semaphore with value 1.)
Semaphore:
Is the number of free identical toilet keys. Example, say we have four toilets with identical locks and keys. The semaphore count - the count of keys - is set to 4 at beginning (all four toilets are free), then the count value is decremented as people are coming in. If all toilets are full, ie. there are no free keys left, the semaphore count is 0. Now, when eq. one person leaves the toilet, semaphore is increased to 1 (one free key), and given to the next person in the queue.
Officially: "A semaphore restricts the number of simultaneous users of a shared resource up to a maximum number. Threads can request access to the resource (decrementing the semaphore), and can signal that they have finished using the resource (incrementing the semaphore)."
Ref: Symbian Developer Library
Mutex is to protect the shared resource.
Semaphore is to dispatch the threads.
Mutex:
Imagine that there are some tickets to sell. We can simulate a case where many people buy the tickets at the same time: each person is a thread to buy tickets. Obviously we need to use the mutex to protect the tickets because it is the shared resource.
Semaphore:
Imagine that we need to do a calculation as below:
c = a + b;
Also, we need a function geta() to calculate a, a function getb() to calculate b and a function getc() to do the calculation c = a + b.
Obviously, we can't do the c = a + b unless geta() and getb() have been finished.
If the three functions are three threads, we need to dispatch the three threads.
int a, b, c;
void geta()
{
a = calculatea();
semaphore_increase();
}
void getb()
{
b = calculateb();
semaphore_increase();
}
void getc()
{
semaphore_decrease();
semaphore_decrease();
c = a + b;
}
t1 = thread_create(geta);
t2 = thread_create(getb);
t3 = thread_create(getc);
thread_join(t3);
With the help of the semaphore, the code above can make sure that t3 won't do its job untill t1 and t2 have done their jobs.
In a word, semaphore is to make threads execute as a logicial order whereas mutex is to protect shared resource.
So they are NOT the same thing even if some people always say that mutex is a special semaphore with the initial value 1. You can say like this too but please notice that they are used in different cases. Don't replace one by the other even if you can do that.
Trying not to sound zany, but can't help myself.
Your question should be what is the difference between mutex and semaphores ?
And to be more precise question should be, 'what is the relationship between mutex and semaphores ?'
(I would have added that question but I'm hundred % sure some overzealous moderator would close it as duplicate without understanding difference between difference and relationship.)
In object terminology we can observe that :
observation.1 Semaphore contains mutex
observation.2 Mutex is not semaphore and semaphore is not mutex.
There are some semaphores that will act as if they are mutex, called binary semaphores, but they are freaking NOT mutex.
There is a special ingredient called Signalling (posix uses condition_variable for that name), required to make a Semaphore out of mutex.
Think of it as a notification-source. If two or more threads are subscribed to same notification-source, then it is possible to send them message to either ONE or to ALL, to wakeup.
There could be one or more counters associated with semaphores, which are guarded by mutex. The simple most scenario for semaphore, there is a single counter which can be either 0 or 1.
This is where confusion pours in like monsoon rain.
A semaphore with a counter that can be 0 or 1 is NOT mutex.
Mutex has two states (0,1) and one ownership(task).
Semaphore has a mutex, some counters and a condition variable.
Now, use your imagination, and every combination of usage of counter and when to signal can make one kind-of-Semaphore.
Single counter with value 0 or 1 and signaling when value goes to 1 AND then unlocks one of the guy waiting on the signal == Binary semaphore
Single counter with value 0 to N and signaling when value goes to less than N, and locks/waits when values is N == Counting semaphore
Single counter with value 0 to N and signaling when value goes to N, and locks/waits when values is less than N == Barrier semaphore (well if they dont call it, then they should.)
Now to your question, when to use what. (OR rather correct question version.3 when to use mutex and when to use binary-semaphore, since there is no comparison to non-binary-semaphore.)
Use mutex when
1. you want a customized behavior, that is not provided by binary semaphore, such are spin-lock or fast-lock or recursive-locks.
You can usually customize mutexes with attributes, but customizing semaphore is nothing but writing new semaphore.
2. you want lightweight OR faster primitive
Use semaphores, when what you want is exactly provided by it.
If you dont understand what is being provided by your implementation of binary-semaphore, then IMHO, use mutex.
And lastly read a book rather than relying just on SO.
I think the question should be the difference between mutex and binary semaphore.
Mutex = It is a ownership lock mechanism, only the thread who acquire the lock can release the lock.
binary Semaphore = It is more of a signal mechanism, any other higher priority thread if want can signal and take the lock.
All the above answers are of good quality,but this one's just to memorize.The name Mutex is derived from Mutually Exclusive hence you are motivated to think of a mutex lock as Mutual Exclusion between two as in only one at a time,and if I possessed it you can have it only after I release it.On the other hand such case doesn't exist for Semaphore is just like a traffic signal(which the word Semaphore also means).
As was pointed out, a semaphore with a count of one is the same thing as a 'binary' semaphore which is the same thing as a mutex.
The main things I've seen semaphores with a count greater than one used for is producer/consumer situations in which you have a queue of a certain fixed size.
You have two semaphores then. The first semaphore is initially set to be the number of items in the queue and the second semaphore is set to 0. The producer does a P operation on the first semaphore, adds to the queue. and does a V operation on the second. The consumer does a P operation on the second semaphore, removes from the queue, and then does a V operation on the first.
In this way the producer is blocked whenever it fills the queue, and the consumer is blocked whenever the queue is empty.
A mutex is a special case of a semaphore. A semaphore allows several threads to go into the critical section. When creating a semaphore you define how may threads are allowed in the critical section. Of course your code must be able to handle several accesses to this critical section.
I find the answer of #Peer Stritzinger the correct one.
I wanted to add to his answer the following quote from the book Programming with POSIX Threads by David R Butenhof. On page 52 of chapter 3 the author writes (emphasis mine):
You cannot lock a mutex when the calling thread already has that mutex locked. The result of attempting to do so may be an error return (EDEADLK), or it may be a self-deadlock, where the unfortunate thread waits forever. You cannot unlock a mutex that is unlocked, or that is locked by another thread. Locked mutexes are owned by the thread that locks them. If you need an "unowned" lock, use a semaphore. Section 6.6.6 discusses semaphores)
With this in mind, the following piece of code illustrates the danger of using a semaphore of size 1 as a replacement for a mutex.
sem = Semaphore(1)
counter = 0 // shared variable
----
Thread 1
for (i in 1..100):
sem.lock()
++counter
sem.unlock()
----
Thread 2
for (i in 1..100):
sem.lock()
++counter
sem.unlock()
----
Thread 3
sem.unlock()
thread.sleep(1.sec)
sem.lock()
If only for threads 1 and 2, the final value of counter should be 200. However, if by mistake that semaphore reference was leaked to another thread and called unlock, than you wouldn't get mutual exclusion.
With a mutex, this behaviour would be impossible by definition.
Binary semaphore and Mutex are different. From OS perspective, a binary semaphore and counting semaphore are implemented in the same way and a binary semaphore can have a value 0 or 1.
Mutex -> Can only be used for one and only purpose of mutual exclusion for a critical section of code.
Semaphore -> Can be used to solve variety of problems. A binary semaphore can be used for signalling and also solve mutual exclusion problem. When initialized to 0, it solves signalling problem and when initialized to 1, it solves mutual exclusion problem.
When the number of resources are more and needs to be synchronized, we can use counting semaphore.
In my blog, I have discussed these topics in detail.
https://designpatterns-oo-cplusplus.blogspot.com/2015/07/synchronization-primitives-mutex-and.html