I am following the Linux Device Drivers. When it introduces spinlocks, it gives the following example:
Your driver is executing and has just taken out a lock that controls access to its device. While the lock is held, the device issues an interrupt, which causes your interrupt handler to run. The interrupt handler, before accessing the device, must also obtain the lock. Taking out a spinlock in an interrupt handler is a legitimate thing to do; that is one of the reasons that spinlock operations do not sleep. But what happens if the interrupt routine executes in the same processor as the code that took out the lock originally? While the interrupt handler is spinning, the noninterrupt code will not be able to run to release the lock. That processor will spin forever.
I don't understand why if the interrupt handler is spinning, the noninterrupt code cannot be executed.
Is it because the routine in the interrupt handler cannot be preempted? If so, is that to say the interrupt routine must be atomic?
Is it because the routine in the interrupt handler cannot be preempted?
Not by process-context code.
If so, is that to say the interrupt routine must be atomic?
It could be interrupted by a higher priority interrupt or a NMI (or by SMM).
Related
How does a Linux signal lead to the instruction stream of the X86 processor getting interrupted? So what CPU facility is used?
You have synchronous and asynchronous interrupts.
Synchronous interrupts are for issues like page faults, exceptions etc. Problems that are caused by the instructions that are executing on the CPU.
Asynchronous interrupts are from an IPI from the LAPIC, timer interrupt or for an interrupt picked up by the I/O-APIC and routed to the right LAPIC which then interrupts the processor. So these are external events.
But which X86 mechanism does the Signal use to interrupt the instruction stream and start processing the signal handler.
It isn't an asynchronous interrupt AFAIK because interrupts are handled within the kernel and signals in user-space. But its behavior is very similar to that of an asynchronous interrupt.
The kernel has to deliver a signal to user-space. You're right that that doesn't just happen on its own in hardware. That's why signal handling can respect the user-space red-zone, sigaltstack, and default actions if there's no handler registered.
As soon as the kernel has control, it can deliver the signal to user-space (or do the default action of ignoring it or killing the process).
If the signal was sent by a process running on another core, to a process running on this core, then probably it's delivered to user-space from an IPI handler, or else just at the next timer interrupt or system call that gives the kernel a chance to check for a pending signal.
When the IPI interrupt handler is preparing to return to user-space, it notices that there's a pending interrupt for the process that it's about to return to. (Either with a special case for one type of IPI, or by running the scheduler since we're in the kernel anyway). Instead of using iret to return to the interrupt frame pushed by hardware for the async interrupt, the kernel instead can iret to the address of the user-space signal handler.
The whole point of using an IPI (if that's what Linux even does) is to transfer control to the kernel sooner, instead of just waiting for it to notice the pending signal the next time it calls schedule().
If the process the signal is sent to isn't currently running on any core, then it either wakes the process up if there's a free CPU, or the signal just sits there for that task until the scheduler on some core decides to run it on this core. At that point it will notice and deliver the pending signal.
There are three type of interruptions:
External
Internal (software interrupts)
Syscall (based on internal)
I had got the question "can the interruptions be interrupted or preempted by scheduler (which is also interrupt by timer)?"
After some research i am totally confused:
Someone says there are priorities of the interruptions and only interruption with higher priority can interrupt another one. Does it pertain only to external interruptions? / How is it arranged in real OS, say x86-64 Linux? Is it used? / Okay, if some interruption has been interrupted, is it gonna be resumed? Someone says interruptions don't have a "context", but as i know preempting some process / thread occurs with interruptions from a timer, so it's possible to switch context back to the interruption that has occurred to preempt the process. Correct me please if i'm wrong in this.
Someone says there is some flag for interrupt handlers "INTERRUPTIBLE", and let's say if some syscall handler is executing with this flag set it may be interrupted by a signal. Is it related only to software interruptions? Don't external interruptions have that flag? / What if this flag is not set but the timer (for example) interruption occurs to preempt the thread? Is it ignored?
Someone says it only depends on what an interruption handler is doing. For example if there is read() syscall that is waiting for input, it sets particular flags so that scheduler (timer interruption) can interrupt and preempt it. But if the handler is doing something crucial it can forbid to interrupt or preempt it, so it's gonna possess a cpu by itself until it finished. It seems there are a lot of mechanisms in x86, and i don't absolutely get which one is used and how it works in real life.
(Answering for Linux on x86, since those are your tags).
There are only two types of events that could interrupt your code. Interrupts, and exceptions. Syscalls are considered exceptions.
I had got the question "can the interruptions be interrupted or preempted by scheduler (which is also interrupt by timer)?"
The scheduler doesn't interrupt anything. It's just code, it doesn't run on it's own. Only an interrupt or exception can interrupt your code. You can have a timer interrupt occur that then calls the schedule().
If you're in an interrupt, interrupts are likely disabled. So no timer interrupt, thus no schedule(). If you're in an interrupt with interrupts enabled (allowing for nested interrupts), a timer interrupt could try to invoke the scheduler, but it wouldn't run because it would detect that preemption is disabled. And preemption is always disabled when you're in an interrupt via the preempt_count field. The goal here is to prevent preemption in an interrupt context.
You have lots of other questions, but most of them can be answered by reading the available literature on the subject.
I read some related posts:
(1) From Robert Love: http://permalink.gmane.org/gmane.linux.kernel.kernelnewbies/1791
You cannot sleep in an interrupt handler because interrupts do not have a backing
process context, and thus there is nothing to reschedule back into. In other
words, interrupt handlers are not associated with a task, so there is nothing to
"put to sleep" and (more importantly) "nothing to wake up". They must run
atomically.
(2) From Which context are softirq and tasklet in?
If sleep is allowed, then the linux cannot schedule them and finally cause a
kernel panic with a dequeue_task error. The interrupt context does not even
have a data structure describing the register info, so they can never be scheduled
by linux. If it is designed to have that structure and can be scheduled, the
performance for interrupt handling process will be effected.
So in my understanding, interrupt handlers run in interrupt context, and can not sleep, that is to say, can not perform the context switch as normal processes do with backing mechanism.
But a interrupt handler can be interrupted by another interrupt. And when the second interrupt handler finishes its work, control flow would jump back to the first interrupt handler.
How is this "restoring" implemented without normal context switch? Is it like normal function calls with all the registers and other related stuff stored in a certain stack?
The short answer is that an interrupt handler, if it can be interrupted by an interrupt, is interrupted precisely the same way anything else is interrupted by an interrupt.
Say process X is running. If process X is interrupted, then the interrupt handler runs. To the extent there is a context, it's still process X, though it's now running interrupt code in the kernel (think of the state as X->interrupt if you like). If another interrupt occurs, then the interrupt is interrupted, but there is still no special process context. The state is now X->first_interrupt->second_interrupt. When the second interrupt finishes, the first interrupt will resume just as X will resume when the first interrupt finishes. Still, the only process context is process X.
You can describe these as context switches, but they aren't like process context switches. They're more analogous to entering and exiting the kernel -- the process context stays the same but the execution level and unit of code can change.
The interrupt routine will store some CPU state and registers before enter real interrupt handler, and will restore these information before returning to interrupted task. Normally, this kind of storing and restoring is not called context-switch, as the context of interrupted process is not changed.
As of 2020, interrupts (hard IRQ here) in Linux do not nest on a local CPU in general. This is at least mentioned twice by group/maintainer actively contributing to Linux kernel:
From NAPI updates written by Jakub Kicinski in 2020:
…Because normal interrupts don't nest in Linux, the system can't service any new interrupt while it's already processing one.
And from Bootlin in 2022:
…Interrupt handlers are run with all interrupts disabled on the local CPU…
So this question is probably less relevant nowadays, at least for Linux kernel.
I know that linux does nested interrupts where one interrupt can "preempt" another interrupt, but what about with other tasks.
I am just trying to understand how linux handles interrupts. Can they be preempted by some other user task/kernel task.
Reading Why kernel code/thread executing in interrupt context cannot sleep? which links to Robert Loves article, I read this :
some interrupt handlers (known in
Linux as fast interrupt handlers) run
with all interrupts on the local
processor disabled. This is done to
ensure that the interrupt handler runs
without interruption, as quickly as
possible. More so, all interrupt
handlers run with their current
interrupt line disabled on all
processors. This ensures that two
interrupt handlers for the same
interrupt line do not run
concurrently. It also prevents device
driver writers from having to handle
recursive interrupts, which complicate
programming.
So AFIK all IRQ's are disabled while within the interrupt handler, therefore it cannot be interrupted!?
Simple answer: An interrupt can only be interrupted by interrupts of higher priority.
Therefore an interrupt can be interrupted by the kernel or a user task if the interrupt's priority is lower than the kernel scheduler interrupt priority or user task interrupt priority.
Note that by "user task" I mean user-defined interrupt.
I wanted to know when the processor get's interrupted and an ISR (interrupt service routine) is executed, is that executed in the context of the thread that was interrupted to handle this interrupt or is it executed in its own thread and then goes back to where it left of in the original thread?
So a context switch actually occurs when an interrupt occurs?
A thread isn't created to handle the interrupt (part of why system calls can sometimes fail), though you can have a special thread to handle interrupts (read about "second level interrupt handlers" in the Wikipedia article on interrupt handling; I'm not certain if Windows uses SLIHs). There is a potential context switch since the ISR runs in kernel mode. Even if the current thread is in kernel mode, some context will be saved before calling the interrupt handler.
Still looking for documentation.