In Linux, what are the options for handling device interrupts in user space code rather than in kernel space?
Experience tells it is possible to write good and stable user-space drivers for almost any PCI adapter. It just requires some sophistication and a small proxying layer in the kernel. UIO is a step in that direction, but If you want to correctly handle interrupts in user-space then UIO might not be enough, for example if the device doesn't support the PCI-spec's interrupt disable bit which UIO relies on.
Notice that process wakeup latencies are a few microsecs so if your implementation requires very low latency then user-space might be a drag on it.
If I were to implement a user-space driver, I would reduce the kernel ISR to just a "disable & ack & wakeup-userpace" operation, handle the interrupt inside the waked-up process, and then re-enable the interrupt (of course, by writing to mapped PCI memory from the userspace process).
There is Userspace I/O system (UIO), but handling should still be done in kernelspace. OTOH, if you just need to notice the interrupt, you don't need the kernel part.
You may like to take a look at CHAPTER 10: Interrupt Handling from Linux Device Drivers, Third Edition book.
Have to trigger userland code indirectly.
Kernel ISR indicates interrupt by writing file / setting register / signalling. User space application polls this and goes on with the appropriate code.
Edge cases: more or less interrupts than expected (time out / too many interrupts per time interval)
Linux file abstraction is used to connect kernel and user space. This is performed by character devices and ioctl() calls. Some may prefer sysfs entries for this purpose.
This can look odd because event triggered device notifications (interrupts) are hooked with 'time triggered' polling, but it is actually asyncronous blocking (read/select). Anyway some questions are arising according to performance.
So interrupts cannot be directly handled outside the kernel.
E.g. shared memory can be in user space and with some I/O permission settings addresses can be mapped, so U-I/O works, but not for direct interrupt handling.
I have found only one 'minority report' in topic vfio (http://lxr.free-electrons.com/source/Documentation/vfio.txt):
https://stackoverflow.com/a/21197797/5349798
Similar questions:
Running user thread in context of an interrupt in linux
Is it possible in linux to register a interrupt handler from any user-space program?
Linux Kernel: invoke call back function in user space from kernel space
Linux Interrupt vs. Polling
Linux user space PCI driver
How do I inform a user space application that the driver has received an interrupt in linux?
Related
This question already has answers here:
Userspace vs kernel space driver
(2 answers)
Closed 5 years ago.
I have been reading "Linux Device Drivers" by Jonathan Corbet. I have some questions that I want to know:
What are the main differences between a user-space driver and a kernel driver?
What are the limitations of both of them?
Why user-space drivers are commonly used and preferred nowadays over kernel drivers?
What are the main differences between a user-space driver and a kernel driver?
User space drivers run in user space. Kernel drivers run in kernel space.
What are the limitations of both of them?
The kernel driver can do anything the kernel can, so you could say it has no limitations. But kernel drivers are much harder to "prove correct" and debug. It's all-to-easy to introduce race conditions, or use a kernel function in the wrong context or with the wrong locking. Things will appear to work for a while, but cause problems (including crashing the whole system) down the road. Drivers must also be wary when reading all user input (both from the device and from userspace) because invalid data can sometimes cause crashes.
A user-space driver usually needs a small shim in the kernel to do it's bidding. Usually, that 'shim' provides a simpler API. For example, the FUSE layer lets people write file systems in any language. They can be mounted, read/written, then unmounted. The shim must also protect the kernel against all invalid input.
User-space drivers have lots of limitations. For example, the kernel reserves some memory for use during emergencies, but that is not available for users-space. During memory pressure, the kernel will kill random user-space programs, but never kill kernel threads. User-space programs may be swapped out, which could lead to your device being unavailable for several seconds. (Kernel code can not be swapped out.) Running code in user-space requires several context switches. These waste a "lot" of CPU time. If your device is a 300 baud modem, nobody will notice. But if it's a gigabit Ethernet card, and every packet has to go to your userspace driver before it gets to the real user, the system will have major bottlenecks.
User space programs are also "harder" to use because you have to install that user-space software, which often has many library dependencies. Kernel modules "just work".
Why user-space drivers are commonly used and preferred nowadays over kernel drivers?
The question is "Does this complexity really need to be in the kernel?"
I used to work for a company that made USB dongles that talked a particular protocol. We could have written a full kernel driver, but instead just wrote our program on top of libUSB.
The advantages: The program was portable between Linux, Mac, Win. No worrying about our code vs the GPL.
The disadvantages: If the device needed to data to the PC and get a response quickly, there is no guarantee that would happen. For example, if we needed a real-time control loop on the PC, it would be harder to have bounded response times. (Maybe not entirely impossible on Linux.)
If there is a way to do it in userspace, I would try that first. Only if there are significant performance bottlenecks, or significant complexity in keeping it in userspace would you move it. Even then, consider the "shim" approach, and/or the "emulator" approach (where your kernel module makes your device look like a serial port or a block device.)
On the other hand, if there are already several kernel modules similar to what you want, then start there.
I'm currently using a SAMA5D31-EK board running Linux 3.10.0+ to control some hardware devices. I'm using GPIOs, I2C, PWM and UARTS available in that board. Some devices are controlled with just a GPIO line while others need an UART a PWM and 3 GPIOs. So far I'm using an userspace program to control those hardware devices - basically a stepper motor, an ADC and a alphanumeric LCD display.
What would be the advantges of developping a kernel device driver to control those devices? So far (using a userspace program) the only limitation I've found is speed: since I have to bit bang some GPIOs, the result is a bit slow.
I assume that you have the platform-specific drivers available for the I2C/GPIO/PWM/UART interfaces on your board(it should be part of BSP[Board-support-package] ).
It is just that you don't want to use the Kernel device driver framework and want to do things from the user-space. I'd been in this situation hence I know, how tempting it could be,especially , if you are not well-versed in Kernel device drivers.
a. SPEED: You mentioned it. But, you probably didn't grasp the reason completely.
Speed efficiency comes from avoiding the Context-switching between Kernel and User-space process. Here is an example:
/* A loop in kernel code which reads a register 100 time */
for (i = 0 ; i < 100 ; i++ )
{
__kernel_read_reg(...);
}
/* A loop in User-space code which reads a register 100 time */
for ( i= 0 ; i < 100; i++)
{
__user_read_reg(...);
}
Functionality wise both *_read_reg() is same. Assuming that __user_read_reg() will go through a typical-system-call procedure,it has to do a Context-switch for every single __user_read_reg(...) which is too costly.
You may argue, "We can mmap() the hardware registers and avoid system call for such operations".
Of course, you could do that, but the point I was making is:
What is close to hardware (for example: a register read or write or handling an interrupt) should be done as fast as possible. Latencies involved in context-switching will impact the performance.
b. Existing/Tested/Well-built subsystems:
If you see an I2C subsystem in the Linux Kernel, it provides a well-tested, robust framework which could be easily-reused. You don't have to write full I2C subsytem (handling all device types, speed, various configuration etc ) in the user-space.
Re-using" what is already done could be one big advantage while going for kernel device drivers.
c. Move from Polling-based approach to Interrupt-based mechanism
If you are not handling interrupts in Kernel driver,You must be using some sort of polling-mechanism in the user-space process. Depending on the system, it might not be very reliable way of handling the hardware-changes.Definitely not accurate/reliable for fast devices.
Interrupt-based mechanism , in general, where you handle the critical changes as fast as possible( Hardware interrupt context) and move the non-critical work-load either to user-space or some other kernel mechanism is more reliable way of handling devices.
Of-course, there could be several more arguments and counter-arguments besides above three.
Another thread which might be of interest to you is here:
Userspace vs kernel space driver
This more a general question. Consider an external device. From time to time this device writes data via its device driver to a specific memory address. I want to write a small C program which read out this data. Is there a better way than just polling this address to check if the value has been changed? I want to keep the CPU load low.
I have done some further research
Is "memory mapped IO" an option? My naive idea is to let the external device writes a flag to a "memory mapped IO"-address which triggers a kernel device driver. The driver then "informs" the program which proceed the value. Can this work? How can a driver informs the program?
The answer may depend on what processor you intend to use, what the device is and possibly whether you are using an operating system or RTOS.
Memory mapped I/O per se is not a solution, that simply refers to I/O device registers that can be directly addressed via normal memory access instructions. Most devices will generate an interrupt when certain registers are updated or contain new valid data.
In general if using an RTOS you can arrange for the device driver to signal via a suitable IPC mechanism any client thread(s) that need to handle the data. If you are not using an RTOS, you could simply register a callback with the device driver which it would call whenever the data is updated. What the client does in the call back is its business - including reading the new data.
If the device in question generates interrupts, then the handling can be done on interrupt, if the device is capable of DMA, then it can handle blocks of data autonomously before teh DMA controller generates an DMA interrupt to a handler.
I am going to write a PCIe base serial I/O card driver in Linux.
As per my knowledge through the configuration space, it provides the interrupt line, and through the IRQF_SHARED flag we are able to share the interrupt handler with that corresponding IRQ line.
But my confusion is how can I know which line is shared or not shared?
For a device driver, there is no useful way (and especially no portable way) to find out if the interrupt line is actually shared, and this could change at any time by loading/unloading other drivers.
PCI drivers must always assume that their interrupt might be shared.
Note: PCI Express devices are supposed to support MSIs (message-signaled interrupts), which are never shared.
Your driver should enable MSIs if at all possible.
However, it is not guaranteeed that the system supports them.
Kernel-assisted probing
The Linux kernel offers a low-level facility for probing the interrupt number. It works
for only nonshared interrupts, but most hardware that is capable of working in a
shared interrupt mode provides better ways of finding the configured interrupt num-
ber anyway. The facility consists of two functions, declared in <linux/interrupt.h>
(which also describes the probing machinery):
unsigned long probe_irq_on(void);
This function returns a bit mask of unassigned interrupts. The driver must pre-
serve the returned bit mask, and pass it to probe_irq_off later. After this call, the
driver should arrange for its device to generate at least one interrupt.
int probe_irq_off(unsigned long);
After the device has requested an interrupt, the driver calls this function, passing
as its argument the bit mask previously returned by probe_irq_on. probe_irq_off
returns the number of the interrupt that was issued after “probe_on.” If no inter-
rupts occurred, 0 is returned (therefore, IRQ 0 can’t be probed for, but no cus-
tom device can use it on any of the supported architectures anyway). If more than
one interrupt occurred (ambiguous detection), probe_irq_off returns a negative
value.
The programmer should be careful to enable interrupts on the device after the call to
probe_irq_on and to disable them before calling probe_irq_off. Additionally, you
must remember to service the pending interrupt in your device after probe_irq_off.
Run cat /proc/interrupt. In the rightmost column of the output you should see your device on one of the interrupts lines. If it's shared you'll see other devices assigned to that interrupt as well.
I'm using the Luminary LM3S8962 micro-controller and its included Library Guide, but this should be relevant to any ARM Cortex-M3s that have Nested Vector Interrupts.
You can only register one interrupt service routine function with an entire GPIO Port. A GPIO port typically has 8 pins on it, each of which can be configured with an interrupt. For each pin, you can test whether or not an interrupt "happened" on it (is pending), right? and for each pin you can clear a pending interrupt, right?
If a pin on the GPIO port triggers the ISR then the processor is in the ISR. Then what happens if another pin on the same port triggers an interrupt while we're in the ISR? We assume the code detects what pins have pending interrupts.
- Is this ISR interrupted and a new one begins, with the same code, but an updated PinInterruptStatus register ? (I hope not)
- Is this ISR executed until completion, immediately executing the interrupt for the other pin right afterward? (I know ARM Cortex M3 implements tail-chaining of interrupts)
- Or must there be a while loop that loops until all the pins have been cleared, clearing a pin after it has been processed?
maybe this will help:
http://www.ti.com/lit/gpn/lm3s8962
As stated in the comment: generally ISRs should take steps to prevent reentrancy. In something like a PIC, this could be as simple as disabling the interrupt at the "top" of the ISR, and enabling the interrupt at the "bottom". The M3's NVIC is a bit more complicated. This white paper (http://www.arm.com/files/pdf/IntroToCortex-M3.pdf) states the following on p.7:
The NVIC supports nesting (stacking) of interrupts, allowing an
interrupt to be serviced earlier by exerting higher priority. It also
supports dynamic reprioritisation of interrupts. Priority levels can
be changed by software during run time. Interrupts that are being
serviced are blocked from further activation until the interrupt
service routine is completed, so their priority can be changed without
risk of accidental re-entry.
The above discussion directly addresses the possibility of same interrupt reentrancy, and it also introduces the concept of prioritization to handle interrupts of higher priority interrupting your ISR.
This reference is pretty good: http://infocenter.arm.com/help/topic/com.arm.doc.dui0552a/DUI0552A_cortex_m3_dgug.pdf. On p. 4-9, you'll find instructions to enable/disable interrupts. On page 4-6, you'll find a description of the Interrupt Clear-pending Registers. Using these, you can determine what interrupts are pending. If you really want to get fancy with interrupt enable/disable control, check out the BASEPRI and BASEPRO_MAX registers.
Having said that, I'm not sure I agree with your statement that your question is relevant to any Cortex-M3. Keil (my flavor of Cortex-M3) mentions that the EXTI (external interrupt controller) handles GPIO pin interrupts. Interestingly, the ARM documentation briefly discusses "EXTI", but does not refer to it as a "controller" like the Keil STM32 documentation. A quick google on "STM32 EXTI" yeilds lots of hits, a similar search on "Luminary EXTI" does not yield much. Given that, I'm guessing that this particular controller is one of the peripheral devices that ARM leaves to 3rd parties.
This document
bolsters that view: http://www.st.com/internet/com/TECHNICAL_RESOURCES/TECHNICAL_LITERATURE/REFERENCE_MANUAL/CD00171190.pdf. There are several AFIO_EXTI registers mentioned here. These permit the mapping of GPIO lines to interrupts. Unfortunately, I can't find anything similar in the Luminary documentation.
So...what does this mean? It looks like you only have port-level granularity for your interrupt. Thus, your ISR will have to determine which pin transitioned (assuming your are looking for edges). Good luck!
In Cortex-M3, if two interrupts are the same priority (for all GPIO pins), the former will not be interrupted. The interrupt comes later will be in pending state.
When a GPIO interrupt occurs you can check the GPIO Interrupt Status for Rising/Falling IO0IntEnR/IO0IntEnF (depending on ) for the corresponding bit to find the pin that causes the interrupt.