I have a program that requires several processes access a shared resource. This shared resource does not exist when they all start, so one of them needs to create it. Once the shared resource is created, important infrastructure is installed for use later. There is, however, a possibility that if the "creator" process is scheduled out before it can install the infrastructure in the shared resource, that other processes will try to use the uninitialized data (leading to undefined behavior).
In order to control this, I've created a named semaphore (sem_t *sem_init). Any process that is not the creator "downs" or "waits" on this zero-initialized semaphore. When the creator process has finished setup, it "up's" or "posts" the semaphore, releasing the processes. However, there remains one problem. I do not know exactly how many processes are waiting on it.
In order to solve this problem, I have the following options:
I create a counting semaphore. Each process "up's" or "posts" on this semaphore before blocking on the initialization semaphore. This way, I can know how many processes to release.
I just "post" on the initialization semaphore until it is the maximum allowed value.
I don't like these "solutions" though. For one, I am limited by the maximum size of a semaphore when it comes to the number of processes that I can count. It also seems like "posting" so many times would incur a nasty overhead. My question then, is whether there is any way in which I can instruct a semaphore to release all blocked processes, without me having to do any explicit bookkeeping on my end. I'd also not like to be constrained by the maximum value of a semaphore.
Something like: sem_releaseAll (sem_t *sem_p); would be ideal.
Note: I would greatly prefer a Linux-native solution.
Related
According to this link, How do I recover a semaphore when the process that decremented it to zero crashes? it seems that there is no robust semaphore inter-process, and the author finally chose filelock, which is guaranteed to be released properly by the system-level or kernel-level control.
But I also found robust mutex provided by pthread https://man7.org/linux/man-pages/man3/pthread_mutexattr_setrobust.3.html why there is no something like robust semaphore?
And an extra question: what robust alternatives we have regarding IPC synchronization? filelock seems to be the best one. I think providing such mechanism is not that difficult from system or kernel level,since they do implement fielock. then why they don't provide some other approaches?
When you use a mutex, it can be acquired by at most one thread at a time. Therefore, once the mutex has been acquired, the owner can write its process ID or thread ID (depending on the system) into the mutex, and future users can detect whether the owner is still alive or not.
However, a semaphore is ultimately a counter. It is possible that different threads may increment or decrement the counter. There isn't intrinsically one resource that is being shared; there could instead be multiple resources.
For example, if we're trying to limit ourselves to a certain number of outgoing connections (say, 8), then we could create a semaphore with that value and allow threads to acquire it (wait) to make a connection, and then increment it (post) when they're done. If we never want to make more than 8 connections at once, the semaphore will never block; we'll have acquired it successfully each time, even though there's no exclusion.
In such a situation, there isn't going to be space inside the semaphore to store every process's thread ID. Using memory allocation is tricky because that code needs to be synchronized independently, and even if that could be solved, it means that a semaphore value would have at least O(N) performance when acquiring the semaphore. I work on a production system that uses hundreds of threads, so you can imagine the performance problems if we had such a semaphore design.
There are other solutions which you can use when you need robustness, such as file locking or a robust mutex in a shared memory segment, but none of them have the same properties as a semaphore. Therefore, any discussion of what primitives should be used instead depends on the particular needs of the situation (which should probably be placed in a new question).
So I'm trying to use mutex_init(), mutex_lock(), mutex_unlock() for thread synchronization.
I am currently trying to schedule threads in a round robin fashion(but more than 1 thread could be running at a time) and I set the current state of a thread to TASK_INTERRUPTIBLE, followed by waking up another thread whose PID, I have in a list.
I need to iterate over this list for my logic.
As I understand it, I need to lock this list as I access its elements, or another thread might miss a new entry while I'm making changes to it. Also, as one mutex has locked a resource, no other mutex can unlock it, until the original mutex releases it.
But, I'm still not sure if I'm locking it correctly. (I release the lock before I call schedule(), and re-lock after that)
I declare a mutex locally within a thread and lock the list. After my current thread locks
mutex_lock(&lock);
and I iterate over the list, till I find something(or ends if it doesn't find anything), then unlocks.
mutex_unlock(&lock);
I assume locking while I iterate is legal. I have never seen examples of this though.
Also, is it normal for the process to have a state of (TASK_UNINTERRUPTIBLE) while it holds a mutex lock?
EDIT : I am adding some more information based on the answer below.
It is possible my program may be run on a virtual machine with a single core. Therefore, I do not want to risk infinite polling using spin_lock().
I am trying to maintain scheduling between threads that have a certain id. For example if there are 4 threads. 2 in set 'A' and 2 in set 'B'. I allow only 1 thread to run in each set. But I switch between threads in a given set. However, a thread in set 'A' should not switch to any thread in set 'B'
(I know the kernel scheduler wont be perfect, so an approximate switching will do).
My Reasoning for TASK_STATE's:
1) Initial thread that gets created is running.
2) If another thread in the same set is running (and this one hasn't executed for a given time). Set other thread to TASK_INTERRUPTIPLE, while calling schedule(); Note: There can be more than 2 threads in each set, but let's keep it simple by considering only 2 for now.
3) If it has executed for enough time, set this task to TASK_INTERRUPTIPLE, set the other task in the same set to TASK_RUNNING, while calling schedule();
All this logic happens while I am accessing certain data structures which are locked by a (now) Global Mutex. I unlock the mutex just before I call schedule(), and instantly re-lock afterward. After my logic part is done, I completely unlock the mutex.
Is there anything fundamentally wrong with the approach?
As I understand it, I need to lock this list as I access its elements
Yes, that is true. But if you use a mutex, you're going to be really sad because a call to lock/unlock is a call to the scheduler. Therefore, calling it from inside the scheduler should result in deadlock. What you need to do depends on if your processor is multi-core or (the mythical) single-core. (Is this a virtual system?) On a single-core processor you can disable interrupts. On a multi-core processor, disabling interrupts is not sufficient (it only disables interrupts for that one core, and another core may still be interrupted). The simplest thing to do on a multi-core is to use a spinlock. Unlike the mutex, both of these locking mechanisms can be unlocked from different threads.
I set the current state of a thread to TASK_INTERRUPTIBLE
Is the thread being taken off the CPU? If so, it's not running, so I suspect that TASK_INTERRUPTIBLE is the wrong state. It would be helpful if you could list the possible states for me or if you could describe what the state is supposed to indicate. Because to me "TASK_INTERRUPTIBLE" sounds like a running task.
I declare a mutex locally within a thread and lock the list
Local mutexes are a red flag! The resource you are locking should be protected by a mutex with the same scope. If the list is global, it should have a global mutex to protect it. Threads that want to use the list must first acquire its mutex. Of course, as I already talked about, you probably want to use a different kind of locking to protect the list of ready-to-run processes.
I assume locking while I iterate is legal
It is perfectly legal (assuming of course that your mutual exclusion scheme is bug-free). In fact, it's required. If another thread were allowed to, for example, remove a node from the list while you were reading it, you could end up dereferencing a deleted node.
Also, is it normal for the process to have a state of TASK_UNINTERRUPTIBLE while it holds a mutex lock?
No, not while it holds the lock if the process is currently running on a CPU. A mutex is available to user code. If holding a mutex made the process uninterruptible, that would mean that a process could hijack the system by simply locking a mutex and never releasing it. Now, you will find that the lock and unlock functions need to be uninterruptible on a single-core processor. However, it doesn't make sense to set the state for the process because it's actually the scheduler that must not be interrupted.
Example:
A thread finishes writing to a shared variable, and then it unlocks it, but continues to use that variable's value (without changing it).
And immediately, another thread successfully unlocks() that mutex and reads the shared variable.
For my (mis-)understanding, some things could be happening on this situation:
On the WRITER thread:
A compiler optimization could make the write occur only at some later point
The written value could be retained in the current CPU core's cache, and flushed to the memory at some later point
On the READER thread:
The value of the variable may have been read before the mutex lock(), and because of some compiler optimization or just the usual work of the CPU cache, still be considered "already read from memory" and thus, not fetched from the memory again.
Thus, the value we have here is not the updated one from the other thread.
Does the pthread mutex lock/unlock() functions execute any code to "flush" the current cache to the memory and anything else needed to make sure the current thread is synchronized with everything else (I cannot think of anything else than the cache), or is it just not needed (at least in all known architectures)?
Because if all the mutexes do is just what the name does - mutual exclusion to it's reference - then, if I have thousands of threads dealing with the same data and from my algorithm's point of view, I already know that when one thread is using a variable, no other thread will try to use it at the same time, than it means I don't need a mutex? Or will my code be missing some low level and architecture-specific method(s) implemented inside the PTHREAD library to avoid the problems above?
The pthreads mutex lock and unlock functions are among the list of functions in POSIX "...that synchronize thread execution and also synchronize memory with respect to other threads". So yes, they do more than just interlock execution.
Whether or not they need to issue additional instructions to the hardware is of course architecture dependent (noting that almost every modern CPU architecture will at least happily reorder reads with respect to each other unless told otherwise), but in every case those functions must act as "compiler barriers" - that is, they ensure that the compiler won't reorder, coalesce or omit memory accesses in situations where it would otherwise be allowed to.
It is allowed to have multiple threads reading a shared value without mutual exclusion though - all you need to ensure is that both the writing and reading threads executed some synchronising function between the write and the read. For example, an allowable situation is to have many reading threads that defer reading the shared state until they have passed a barrier (pthread_barrier_wait()) and a writing thread that performs all its writes to the shared state before it passes the barrier. Reader-writer locks (pthread_rwlock_*) are also built around this idea.
I read one of the differences between semaphore and mutex is in case of mutex the process/thread (which ever is having the lock) can only release the lock. But in the case of the semaphore any other process can release the semaphore. My doubt arises when a process that does not have the semaphore with it can release the semaphore. What is the use of having a semaphore?
Let's say I have two processes A and B. Assume process A is having a semaphore with it and executing some critical task. Now let us say process B sends a signal to release the semaphore. In this scenario, will process A release the semaphore even if it is executing some critical task?
You are making half-sense. It is not about ownership. Partner-release in semaphores (and mutexes) is usable, for instance, in my favorite interview question of thread ping-pong. As a matter of fact, I have specifically tried to partner-release a mutex on 3 implementations available to me at a time (Linux/Solaris/AIX) and partner-release did work for mutexes as expected - i.e. mutex was successsfully released and threads blocking on it resumed execution. However, this is, of course, prohibited by Posix.
I think you might be confused on the whole set of differences between a semaphore and a mutex. A mutex provides mutual exclusion. A semaphore counts until it reaches a level where it starts excluding. A semaphore that counted to one would give similar semantics to a mutex though.
A good example would be a television set. Only so many people can watch the same television set, so protecting it with a semaphore would make sense. Anyone can stop watching the television. The remote control for the television can only be operated by one person at a time though, so you could protect it with a mutex.
Some reading...
https://en.wikipedia.org/wiki/Mutual_exclusion
https://en.wikipedia.org/wiki/Semaphore_%28programming%29
"Let's say I have two processes A and B. Assume process A is having a semaphore with it and executing some critical task. Now let us say process B sends a signal to release the semaphore. In this scenario, will process A release the semaphore even if it is executing some critical task?"
One key point to note here is the role of OS kernel. Process B can't send a signal to Process A 'to release the semaphore'. What it can do is request the kernel to give it access to the resource. Process A had requested the kernel and the kernel granted it access to the resource.
Now process A, after it finishes its job, will let the kernel know that it is done with the resource and then kernel grants access to B.
"My doubt arises when a process that does not have the semaphore with it can release the semaphore. What is the use of having a semaphore?"
The key difference between a mutex and a semaphore is, a semaphore serializes access to multiple instances of a resource. Mutex does the same when there is one instance of the resource.
A count is maintained by kernel in case of semaphore and mutex is a special case where the count is 1.
Consider the processes as customers waiting in line at a bank.
The use of semaphore is analogous to the case where there are multiple tellers serving the customers. Usage of mutex is analogous to the case where there is just one teller.
Say there are processes A, B and C that need concurrent access to a resource (lock, file or a data structure in memory, etc.). Further suppose there are 2 instances of the resource. So at most two processes can be granted access at a time.
Process A requests access to an instance of the resource following the required semantics. This request to the kernel involves data structures to identify the resource and maximum number of instances as 2. kernel creates the semaphore with a count of 2, grants A access to the resource and decrements the count to 1, because now only one other process can get access.
Now process B requests access to the resource by following the same semantics. Kernel grants it access and decrements the count to 0.
Now process C requests access, but kernel keeps it in waiting state, because count is 0 and no more than 2 processes can get concurrent access.
Process A is done with the resource and lets kernel know. Kernel notices this and grants access to process C that has been waiting.
In case of mutex, kernel grants access to the resource only one process at a time.
A normal binary semaphore is basically used for synchronization. However, the mutex is for exclusive access to a resource. A mutex is a special variant of semaphore that allows only one locker at a time and with more stringency on ownership than a normal semaphore such as the mutex should be released only by the thread that acquired it. Also, please note that in case of pthreads, fast mutex may not check for this error related to ownership, whereas the error checking mutex shall return error.
For the query related to 2 process A and B, the Process A shall intimate via kernel that it is done with its critical work so that the resource can be made available for waiting processes like B.
You could find some related information in this link too :
When should we use mutex and when should we use semaphore
There is no such thing as "having" a semaphore. Semaphores don't have ownership like mutexes do. The code you describe would simply be buggy. Mutexes won't work if your code is buggy either.
Consider the most classic example of a semaphore -- allowing one train at a time on a section of track. You could implement this with a mutex if the train is a thread. The train would lock the track mutex before going on the track and unlock it after leaving the track.
But what if the train itself is multi-threaded? Which thread should own the track?
And what if the signalling devices are the threads, not the train? Here, the signalling device that detects the train entering the track has to lock the track while the signalling device that detects the train leaving the track has to unlock it.
Mutexes are suitable for cases where there is something that is owned by a particular thread for a short period of time. That thread can "own" the mutex. Semaphores are useful for cases where there is no thread to own anything or nothing for the thread to own.
This may sound like a stupid question, but if one locks a resource in a multi-threaded app, then the operation that happens on the resource, is that done atomically?
I.E.: can the processor be interrupted or can a context switch occur while that resource has a lock on it? If it does, then nothing else can access this resource until it's scheduled back in to finish off it's process. Sounds like an expensive operation.
The processor can very definitely still switch to another thread, yes. Indeed, in most modern computers there can be multiple threads running simultaneously anyway. The locking just makes sure that no other thread can acquire the same lock, so you can make sure that an operation on that resource is atomic in terms of that resource. Code using other resources can operate completely independently.
You should usually lock for short operations wherever possible. You can also choose the granularity of locks... for example, if you have two independent variables in a shared object, you could use two separate locks to protect access to those variables. That will potentially provide better concurrency - but at the same time, more locks means more complexity and more potential for deadlock. There's always a balancing act when it comes to concurrency.
You're exactly right. That's one reason why it's so important to lock for short period of time. However, this isn't as bad as it sounds because no other thread that's waiting on the lock will get scheduled until the thread holding the lock releases it.
Yes, a context switch can definitely occur.
This is exactly why when accessing a shared resource it is important to lock it from another thread as well. When thread A has the lock, thread B cannot access the code locked.
For example if two threads run the following code:
1. lock(l);
2. -- change shared resource S here --
3. unlock(l);
A context switch can occur after step 1, but the other thread cannot hold the lock at that time, and therefore, cannot change the shared resource. If access to the shared resource on one of the threads is done without a lock - bad things can happen!
Regarding the wastefulness, yes, it is a wasteful method. This is why there are methods that try to avoid locks altogether. These methods are called lock-free, and some of them are based on strong locking services such as CAS (Compare-And-Swap) or others.
No, it's not really expensive. There are typically only two possibilities:
1) The system has other things it can do: In this case, the system is still doing useful work with all available cores.
2) The system doesn't have anything else to do: In this case, the thread that holds the lock will be scheduled. A sane system won't leave a core unused while there's a ready-to-run thread that's not scheduled.
So, how can it be expensive? If there's nothing else for the system to do that doesn't require acquiring that lock (or not enough other things to occupy all cores) and the thread holding the lock is not ready-to-run. So that's the case you have to avoid, and the context switch or pre-empt issue doesn't matter (since the thread would be ready-to-run).