I was reading the implementation of Linux semaphores. Due to atomicity, signal and wait (up and down in the source code) use spin locks. Then I saw Linux disabled interrupt in spin_lock_irqsave and reenabled interrupt in spin_unlock. This confused me. In my opinion, there is really no point disabling interrupt within a critical section.
For example, proc A (currently active) acquired the lock, proc B (blocked) is waiting for the lock and proc C is doing some unrelated stuff. It makes perfect sense to context switch to C within the critical section between A and B. Even if C also tries to acquire the lock, since the lock is already locked by A, the result would be C being blocked and A resuming execution.
Therefore, I don't know why Linux decided to disable interrupt within critical sections guarded by spin locks. It probably won't cause any problems but seems like a redundant operation to me.
Allow me to start off with a disclaimer that I am not a Linux expert, so my answer may not be the most accurate. Please point out any flaws and problems that you may find.
Imagine if some shared data is used by various parts of the kernel, including operations such as interrupt handlers that need to be fast and cannot block. Let's say system call foo is currently active and has acquired a lock to use/access shared data bar, and interrupts are not disabled when/before acquiring said lock.
Now a (hardware) interrupt handler, e.g. the keyboard, kicks in and also needs access to bar (hardware interrupts have higher priority than system calls). Since bar is currently being locked by syscall foo, the interrupt handler cannot do anything. Interrupt handlers do need to be fast & not be blocked though, so they just keep spinning while trying to acquire the lock, which would cause a deadlock (i.e. system freeze) since syscall foo never gets a chance to finish and release its lock.
If you disable interrupts before trying to acquire the lock in foo, though, then foo will be able to finish up whatever it's doing and ultimately release the lock (and restore interrupts). Any interrupts trying to come in while foo holds the spinlock will be left on the queue, and will be able to start when the lock is released. This way, you won't run into the problem described above. However, care must also be taken to ensure that the lock for bar is held for as short as possible, so that other higher priority operations can take over whenever required.
The answer is very simple: There is no way for the thread that tries to acquire a lock, to know if the ISR that will interrupt it, will try to acquire the same lock. If that will happen, the ISR will spin forever on that same lock and the system will deadlock.
But what if an interrupt wants to signal a waiting thread ? Or want to test the sempahore value ? The irq disabling is not here to prevent context switch between two process, but to protect from irq. It's all in the comment at the beginning of the file :
/*
* Some notes on the implementation:
*
* The spinlock controls access to the other members of the semaphore.
* down_trylock() and up() can be called from interrupt context, so we
* have to disable interrupts when taking the lock. It turns out various
* parts of the kernel expect to be able to use down() on a semaphore in
* interrupt context when they know it will succeed, so we have to use
* irqsave variants for down(), down_interruptible() and down_killable()
* too.
*
* The ->count variable represents how many more tasks can acquire this
* semaphore. If it's zero, there may be tasks waiting on the wait_list.
*/
Related
I was going through the Robert Love Book and was bit confused about this line. What does it mean by code holding semaphore can be preempted?
If an interrupt occurs accessing the same variable which the user-space application has while it is executing the code in critical section then the user-space application can be preempted?
If my above understanding is true then there is no other alternative than spin-locks to disable an interrupt whenever an user-space application is in critical section?
So what is the use of semaphore in the context of OS? Interrupts might occur anytime when the user application is in critical section and in-order to avoid interrupt intervention we need to use spin-locks all the time.
What does it mean by code holding semaphore can be preempted?
It means that a process that is currently running in its critical section holding some lock for the purpose of synchronization can be preempted. Ideally interrupts have the highest
priority, so unless you disable the interrupt on that processor core, the running process
can be preempted and that might happen while the process is in its critical section.
While there are multiple spin_lock_XXX apis to disable interrupts, you might want to use
the spin_lock_irqsave as it saves the interrupt flags on that core and restores them while releasing the lock.
i have passed through this post and i noticed that in Clifford's answer he said that we shouldn't use mutex in an interrupt, i know that in an interrupt we have to avoid too much instructions and delays ext... but am not very clear about the reasons could anyone clarify me for which reason we have to avoid this?
In case that we want establish a synchronous communication between 2 interrupt driven threads what are the other mecahnism to use if using mutex is not allowed?
The original question you cite refers to code on an Atmel ATMegaAVR - a simple 8 mit microcontroller. In that context, one can assume that the mutex machanism is part of a simple RTOS.
In such a system, there is a thread context and an interrupt context. Interrupts are invoked by the hardware, while threads are scheduler by the RTOS scheduler. Now when an interrupt occurs, any thread will be immediately pre-empted; the interrupt must run to completion and can only be preempted by a higher priority interrupt (where nested interrupts are supported). All pending interrupts will run to completion before the scheduler can run.
Blocking on a mutex (or indeed any blocking kernel object) is a secheduling event. If you were to make any blocking call in an interrupt, the scheduler will never run. In prectice an RTOS would either ignore the blocking call, raise an exception, or enter a terminal error handler.
Some OS's such as SMX, Velocity or even WinCE have somewhat more complex interrupt architectures and support variety of deferred interrupt handler. Deferred interrupt handlers are run-to-completion scheduled from an interrupt but running outside of the interrupt context; the rules for blocking in such handlers may differ, but you would need to refer to the specific OS documentation. Without deferred interrupt handlers, the usual solution is to have a thread wait on a some blocking object such as a semaphore, and have the interrupt itself do little more that cause the object to unblock (such as giving a semaphore for example).
Multi-processor/core and parallel processing systems are another issue altogether, such systems are way beyond the scope of the question where the original comment was made, and beyond my experience - my comment may not apply in such a system, but there are no doubt additional complexities and considerations in any case
A mutex is typically used to ensure that a resource is used by only one user at any given time.
When a thread needs to use a resource it attempts to get the mutex first to ensure the resource is available. If the mutex is not available then the thread typically blocks to wait for the mutex to become available.
While a thread owns the mutex, it prevents other threads from obtaining the mutex and interfering with its use of the resource. Higher priority threads are often the concern here because those are the threads that may preempt the mutex owner.
The RTOS kernel assigns ownership of the mutex to a particular thread and typically only the mutex owner can release the mutex.
Now lets imagine this from an interrupt handler's point of view.
If an interrupt handler attempts to get a mutex that is not available, what should it do? The interrupt handler cannot block like the thread (the kernel is not equipped to push the context of an interrupt handler or switch to a thread from an interrupt handler).
If the interrupt handler obtains the mutex, what higher priority code is there that could interrupt the interrupt handler and attempt to use the mutex? Is the interrupt handler going to release the mutex before completing?
How does the kernel assign ownership of the mutex to an interrupt handler? An interrupt handler is not a thread. If the interrupt handler does not release the mutex then how will the kernel validate that the mutex is being released by the owner?
So maybe you have answers for all those questions. Maybe the you can guarantee that the interrupt handler runs only when the mutex is available or that the interrupt handler will not block on the mutex. Or maybe you're trying to protect the resource access from an even higher priority nested interrupt handler that also wants to use the resource. And maybe your kernel doesn't have any hangup with assigning ownership or restricting who releases the mutex. I guess if you've got all these questions answered then maybe you have a case for using a mutex within an interrupt handler.
But perhaps what you really need is a semaphore instead. One common application of a semaphore is to signal an event. Semaphores are very often used this way within interrupt handlers. The interrupt handler posts or sets the semaphore to signal that an event has occurred. The threads pend on the semaphore to wait for the event condition. (A semaphore doesn't have that ownership restriction that a mutex has.) Event signalling semaphores is one common way to establish synchronous communication between 2 interrupt driven threads.
The term "mutex" is often defined both as being the simplest form of synchronization between execution contexts, and also as being a construct that will not only check whether a resource is available, but wait for it to become available if it isn't, acquiring it as soon as it becomes available. These definitions are inconsistent, since the simplest forms of synchronization merely involve testing whether one has been granted ownership of a resource, and don't provide any in-built mechanism to wait for it if it isn't.
It is almost never proper to have code within an interrupt handler that waits for a resource to become available, unless the only things that could hold the resource would be higher-priority interrupts or hardware that will spontaneously release it. If the term "mutex" is only used to describe such constructs, then there would be very few cases where one could properly use a mutex within an interrupt handler. If, however, one uses the term "mutex" more broadly to refer to the simplest structures that will ensure that a piece of code that accesses a resource can only execute at times when no other piece of code anywhere in the universe will be accessing that resource, then the use of such constructs within interrupts is often not only proper, but required.
While there might be unusual cases where there's some problem with using a mutex in an interrupt handler, it's quite common practice and there's nothing wrong with it.
It really only makes sense on systems with more than one core. With just a single core (and no hyper-threading), the mutex would never do anything anyway. If the core is running code that acquires a mutex that interrupt code can acquire, interrupts (or the subset of them that matter) are disabled anyway. So with just one core, the mutex would never see any contention.
However, with multiple cores, it's common to use mutexes to protect structures that communicate between interrupt and non-interrupt code. So long as you know what you're doing, and you have to if you're going to write interrupt handlers, there's nothing wrong with it.
How the mutex blocks and unblocks is heavily implementation dependent. It can put the CPU to sleep and be woken by an inter-process interrupt. It can spin the CPU in some CPU-specific way.
Note that a totally unrelated concept that is often confused with this is using user-space mutexes in user-space signal handlers. That's a completely different question.
Keeping my question short... i am writing simulation for a RTOS. As usual the main problem comes with context switch simulation. In case of interrupts it is really becoming hard not to deviate from 'Good' coding guidelines.
Say Task A is running and user application is calculating its harmless private stuff which will run for a long time. during this task A, an interrupt X is supposed to occur. (hint: task A has nothing to do with triggering this interrupt X)... now how do i perform context switch from Task A to interrupt X handler?
My current implementation is based on a context thread that waits till some context switch is requested; an interrupt controller thread that can generate interrupts if someone request interrupt triggering; and a main thread that is running Task A. Now i use interrupt controller thread to generate a new thread for interrupt X and then request context thread to do the context switch. Context thread Suspends Task A main thread and resumes interrupt X handler thread. At the end of interrupt X handler thread, Task A main thread is resumed..
[Edit] just to clarify, i already know suspending and terminating threads from outside is really bad. That is why i asked this question. Plus please don't recommend using event etc. for controlling Task A. it is user application code and i can't control it. He can even use while(1){} if he wants...
I suspect that you can't do what you want to do in that way.
You mentioned that suspending a thread from outside is really bad. The reason is that you have no idea what the thread is doing when you suspend it. It's impossible to know whether the thread currently owns a mutex; if it does then any other thread that tries to access the same mutex is going to deadlock.
You have the problem that the runtime being used by the threads that might be suspended is the same as the one being used by the supervisor. That means there are many potential such deadlocks between the supervisor and the other threads.
In a real environment (i.e. not a simulator), the operating system kernel can suspend threads because there are checks in place to ensure that these deadlocks can't happen. I don't know the details, but it probably involves masking interrupts at certain critical points, and probably not sharing the same mutexes between user-mode code and critical parts of the kernel scheduler. (In your case that would mean your scheduler could not use any of the same OS API functions, either directly or indirectly, as are allowed to be used by the user threads, in case they involve mutexes. This of course would be virtually impossible to achieve.)
The reason I asked in a comment whether you have any control over the user code compiler is that if you controlled the compiler then you could arrange for the user code to effectively mask interrupts for the duration of each instruction and only yield to another thread at well-defined points between instructions. This is how it is done in a control system that I work on.
The other aspect is platform dependence. In Linux and other unix-like operating systems, you have signals, which are like user-mode interrupts. You could potentially use signals to emulate context switching, although you would still have the same problem with mutexes. There is absolutely no equivalent on Windows (as far as I know) precisely because of the problem already stated. The nearest thing is an asynchronous procedure call, but this will run only when the thread has put itself into an alertable wait state (which means the thread is in a deterministic state and is now safe to interrupt).
I think you are going to have to re-think the whole concept so that your supervisory thread has the sort of privileged control above the user threads that the OS has in a non-emulated environment. That will probably involve replacing the compiler or the run-time libraries, or both, with something of your own making.
I am a beginer in the synchronization,and i have some problems in the critical sections and i konw that lock the critical sections can ensure the synchronization.
// Some Code
Acquire_lock(){disable Interrupts;}
Critical section //Interrupt occurs and arrives to acquire the same lock.
Release_lock(){enable Interrupts;}
My question is :why disable the interrupts when acquire the lock for the critical sections?
Note the following applies to Unix/Linux, not sure if it is valid for Windows as well.
if you acquire a lock, the lock is "owned" by the current process. I.e. other processes are blocked if they try to take the same lock.
Since locks are owned by the process as a whole, they don't suffice to effectively stop simultaneous access inside the process. (this counts for concurrent threads and interrupt handlers).
For instance suppose a process acquires a lock, then an interrupt happens. This leads to the interrupt handler for that signal to be invoked, possibly in the middle of your critical section. Now suppose the interrupt handler tries to acquire the same lock (because it wants to invoke the same critical section too). This will work (because the process already has the lock).
In other words interrupt handlers will not be stopped by locks taken by the main thread. So for the main thread's lock to be effective, the interrupt handlers must be temporarily stopped as well.
Note for multi-threaded applications in Unix/Linux there is a different set of system calls to synchronize so the above is more an issue for interrupt handlers in a single threaded application than for concurrent threads.
Note: the order should be
disable interrupts
take lock
perform critical code
unlock
enable interrupts
I understand new linux kernel allow kernel space threads to be pre-empted.
Can someone briefly explain how pre-empting works when executing in kernel mode?
So, when a system call is made, a software interrupt will switch the thread into kernel mode and it will run whats necessary.
Now, lets say its time slice is up - and another user thread runs and it also wants to execute in kernel space. (Or it could be a h/w interrupt).
How does the kernel maintain the integrity of any structures that it was modifying for T1 when it got interrupted?
The Linux kernel protects its data structures the same way as anything that runs in a multithreaded environment.
It will likely use some sort of lock to protect data structures that must be accessed atomically. Usually, these include spinlocks, mutexes and semaphores.
There are also functions that disable preemption but this isn't normally used explicitly since locking code will take care of this implicitly.
Can someone briefly explain how pre-empting works when executing in kernel mode?
It works just like any other context switch. When an interrupt happens in pre-emptible code the CPU jumps to the corresponding interrupt handler and leaves some info on the stack (typically the RIP/CS/EFLAGS/RSP/SS registers of the interrupted task) to be able to return to the pre-empted task later.
So, when a system call is made, a software interrupt will switch the thread into kernel mode and it will run whats necessary. Now, lets say its time slice is up - and another user thread runs and it also wants to execute in kernel space. (Or it could be a h/w interrupt). How does the kernel maintain the integrity of any structures that it was modifying for T1 when it got interrupted?
Let's call the first (the pre-empted) task T1 and the new task T2. If T1 was accessing some data structures, then T1 had to acquire the locks first. All of the kernel's data structures that could possibly be accessed by multiple threads concurrently are protected by locks (almost). If T2 tried to access the same data structure, then it will fail to acquire the lock since T1 still has it, as a result T2 will block and give the CPU back to another task. After some time T1 will start executing again, release its locks, sleep again, switch back to T2, T2 acquires the lock, does its thing, releases the lock, etc.
If multiple threads try to access the same protected data concurrently, only the first thread will get access typically, all the other threads will have to wait.