As mentioned above, what is the difference between a dma engine and a dma-controller (on focus on linux)?
When does the linux dma engine come into place? Is this a special device or always part of all periphery devices, which support dma?
When browsing the linux source, I found the driver ste_dma40.c. How does any driver uses this engine?
DMA - Direct memory access. The operation of your driver reading or writing from/to your HW memory without the CPU being involved in it (freeing it to do other stuff).
DMA Controller - reading and writing can't be done by magic. if the CPU doesn't do it, we need another HW to do it. Many years ago (at the time of ISA/EISA) it was common to use a shared HW on the motherboard that did this operation. In recent years , each HW has its own DMA HW mechanism.
But in all cases this specific HW gets the source address and the destination address and passes the data. Usually triggering an interrupt when done.
DMA Engine - Now here I am not sure what you mean. I believe you probably refer to the SW side that handles the DMA.
DMA is a little more complicated than usual I\O since all memory SRC and DST has to be physically present at all times during the DMA operation. If the DST address is swapped to disk, the HW will write to a bad address and the system will crash.
This and other aspects of DMA are handled by the driver with code sections you probably refer to as the "DMA Engine"
*Another interpretation of what 'DMA Engine' is, may be a code part of Firmware (or HW) that handles the DMA HW controller on the HW side.
According to this document, http://www.asprom.com/application/intel_3.pdf:
The 82C37 DMA controllers should not
be confused with the DMA engines found
in some earlier MCH (Memory Controller
Hub) components. These DMA controllers
are tied to the ISA/LPC bus and used
mostly for transfers to/from slow
devices such as floppy disk controllers.
So it seems it is a device found on previous platfroms that used MCHs devices.
Related
What is the difference between programmed (polled) I/O, interrupt-driven I/O, and direct memory access (DMA)? Are these forms of I/O dependent on the operating system?
I've read through the question dma vs interrupt-driven i/o, but it seems to me that the responses are both unclear and contradictory.
Polled (or programmed) I/O: The CPU periodically manually checks if there are any I/O requests available. If there isn't, it keeps executing its normal workflow. If there is, it handles the I/O request instead.
Interrupt-Driven I/O: The CPU doesn't need to manually check for I/O requests. When there is an I/O request available, the CPU is immediately notified using interrupts, and the request is immediately handled using an interrupt service routine.
DMA: The use of DMA allows interrupt-driven I/O to be used. Otherwise, a system must use programmed I/O if DMA is not available.
DMA is a method allowing devices (typically has very slow I/O speeds) to access main memory without needing the CPU to explicitly handle the requests. When the CPU initiates data transfer from I/O devices to main memory, the CPU instructs the DMA controller to handle this task. The CPU will then "forget" about this operation, and proceed with other tasks. When the DMA controller has completed the transfer, it will signal the CPU using an interrupt. The CPU will then "conclude" the task needed associated with the data transfer it initiated.
The availability of DMA and interrupt driven I/O depends on the physical CPU. If the DMA and interrupt hardware exists, then the OS (and your programs) can use interrupt-driven I/O requests. Otherwise, I/O requests must be manually checked periodically by polling.
What is the difference between programmed (polled) I/O, interrupt-driven I/O, and direct memory access (DMA)?
For programmed IO (PIO) you might have a loop, like:
for(i = 0; i < 512; i++) {
dest_buffer[i] = getNextByteFromDevice();
}
The defining characteristic is that CPU time is being spent to move data between the device and RAM. In this case, software/driver knows the transfer completed when the loop finishes, so nothing else is needed.
Note: It is wrong to call this "polled IO". There is no polling involved. The "PIO" acronym has always stood for "Programmed Input/Ouput".
For interrupt driven IO; the device generates in interrupt for each small piece of data that needs to be transferred (e.g. the interrupt handler might do something like "dest_buffer[i++] = getNextByteFromDevice();"). For performance this is horrific (the cost of an interrupt is expensive on modern systems); so interrupt driven IO was mostly only used for ancient and very slow devices (e.g. serial port controllers before FIFOs were added to them in the early 1990s).
For DMA, you'd configure some kind of DMA controller to do the transfer for you (e.g. tell a DMA controller chip the address of your buffer in RAM, the size of the transfer, etc). In this case you don't know when the transfer is finished, so you either poll the DMA controller or poll the device, or arrange some kind of interrupt to occur when the transfer is completed. Note that polling may make it relatively pointless unless the CPU can do other work before it starts polling (otherwise the CPU time spent polling could've been spent doing the transfer without DMA instead).
A fourth option (which is related to DMA but much more common on modern systems) is for devices to do their own bus mastering without any extra/external DMA controller - e.g. instead of telling a disk controller to fetch some data from disk then configuring a DMA controller to transfer the data to RAM; you might tell the disk controller to fetch data from disk and transfer it to RAM itself. This makes it easier to have many devices doing work in parallel (without fighting for a "shared by all devices on the bus" DMA controller), makes device drivers more self contained, and can reduce the need for CPU attention (especially for things like network cards where data can arrive at any time; where the device can use one interrupt to say "data arrived and was transferred to RAM" instead of having 2 different interrupts - one for "data arrived at the device" and a second for "data was transferred from device to RAM"). The disadvantage is that it increases the cost of a device, which is something that became less important as the price of transistors dropped (mostly during the late 1980s).
Are these forms of I/O dependent on the operating system?
They're dependent on the hardware and the OS. If a device only supports one option then the OS/driver must use that option. If a device supports 2 or more options then the OS/driver can choose which to use (and in some cases, may support 2 or more options and may dynamical switch between them based on other criteria - power consumption, IO priority, device load, CPU load, ...).
For hardware; most hardware doesn't provide a useful DMA controller (excluding the old "ISA bus DMA controller chip", which is only used by equally old devices like floppy controller, where the devices themselves are so slow that it doesn't matter that the DMA controller is equally slow). However; modern high-end servers often do have DMA controllers/DMA engines (e.g. "Intel IO Acceleration Technology" - see https://en.wikipedia.org/wiki/I/O_Acceleration_Technology ); and often these DMA controllers are faster and/or more powerful than bus mastering (e.g. able to transfer data from one device to another device bypassing RAM, able to inject data directly into CPUs' caches to avoid cache misses, etc). Sadly; because these modern DMA controllers are restricted to niche hardware, most operating systems and/or most device drivers either don't support them well or don't bother supporting them at all, so (even in their intended niche) the benefits of them existing is diminished by their scarcity.
I understand that the linux kernel uses a driver to communicate with the hard disk device and that there is firmware code on the device to service the driver's requests. My questions are:
what kind of functionality (i.e. api) does the firmware expose? For example, does it only expose an address space that the kernel manages, or is there some code in the linux kernel that deals with some of the physics related to the hard drive (i.e. data layout on track/sector/platter etc...)
Does the kernel schedule the disk's head movement, or is it the firmware?
Is there a standard spec for the apis exposed by hard disk devices?
I understand that the linux kernel uses a driver to communicate with the hard disk device
That's true for all peripherals.
there is firmware code on the device to service the driver's requests
Modern HDDs (since the advent of IDE) have an integrated disk controller.
"Firmware" by itself isn't going to do anything, and is an ambiguous description. I.E. what is executing this "firmware"?
what kind of functionality (i.e. api) does the firmware expose? For example, does it only expose an address space that the kernel manages, or is there some code in the linux kernel that deals with some of the physics related to the hard drive (i.e. data layout on track/sector/platter etc...)
SATA drives use the ATA Packet Interface, ATAPI.
The old SMD and ST506 drive interfaces used cylinder, head, and sector (aka CHS) addressing. Disk controllers for such drives typically kept a similar interface on the host side, so the operating system was obligated to be aware of the drive (physical) geometry. OSes would try to optimize performance by aligning partitions to cylinders, and minimize seek/access time by ordering requests by cylinder address.
Although the disk controller typically required CHS addressing, the higher layers of an OS would use a sequential logical sector address. Conversion between a logical sector address to cylinder, head, & sector address is straightforward so long as the drive geometry is known.
The SCSI and IDE (ATA) interfaces for the host side of the disk controller offered logical block addressing (block = sector) rather than CHS addressing. The OS no longer had to be aware of the physical geometry of the drive, and the disk controller was able to use the abstraction of logical addressing to implement a more consistent areal density per sector using zone-bit recording.
So the OS should only issue a read or write block operation with a logical block address, and not be too concerned with the drive's geometry.
For example, low-level format is no longer possible through the ATA interface, and the drive's geometry is variable (and unknown to the host) due to zone-bit recording. Bad sector management is typically under sole control of the integrated controller.
However you can probably still find some remnants of CHS optimization in various OSes (e.g. drive partitions aligned to a "cylinder").
Does the kernel schedule the disk's head movement, or is it the firmware?
It's possible with a seek operation, but more likely the OS uses R/W operations with auto-seek or LBA R/W operations.
However with LBA and modern HDDs that have sizeable cache and zone-bit recording, such seek operations are not needed and can be counterproductive.
Ultimately the disk controller performs the actual seek.
Is there a standard spec for the apis exposed by hard disk devices?
ATA/ATAPI is a published specification (although it seems to be in a "working draft" state for 20 years).
See http://www.t13.org/Documents/UploadedDocuments/docs2013/d2161r5-ATAATAPI_Command_Set_-_3.pdf
ABSTRACT
This standard specifies the AT Attachment command set used to communicate between host systems and
storage devices. This provides a common command set for systems manufacturers, system integrators, software
suppliers, and suppliers of storage devices. The AT Attachment command set includes the PACKET feature set
implemented by devices commonly known as ATAPI devices. This standard maintains a high degree of
compatibility with the ATA/ATAPI Command Set - 2 (ACS-2).
I would like to ask your advice on the following: I need to write drivers for omap3, for accessing external dsp through fpga (through gpmc interface). The dsp is required to load file to dsp, and to read/write buffer from dsp. There is already FPGA driver in kernel. The kernel is 2.6.32. So I thought of the following options:
writing dsp driver in kernel, which uses the existing fpga driver.
writing a user space driver which interfaces with the fpga kernel driver.
writing user space driver using UIO , which will not use the kernel fpga driver, but shall do the access to fpga, as part of the user space single and complete dsp driver.
What do you think is prefered option ?
What is the advantage of kernel driver over user sace and vise versa ?
Thanks, Ran
* User-space driver:
Easier to debug.
Loads of libraries to support you.
Allows you to hide the details of your IP if you want to ( people will really hate you if you did! )
A crash won't affect the whole system.
Higher latency in handling interrupts since the kernel will have to relay the interrupt to the user space somehow.
You can't control access to your device from user space.
* Kernel-space driver:
Harder to debug.
Only linux kernel frameworks are supported.
You can always provide a binary blob to hide the details of your IP but this is annoying since it has to be generated against a specific kernel.
A crash will bring down the whole system.
Less latency in handling interrupts.
You can control access to your device from kernel space because it's a global context that all processes see.
As a kernel engineer I'm more comfortable/happy hacking code in a kernel context, that's probably why I would write the whole driver in the kernel.
However, I would say that the best thing to do is to divide the functionality of your driver into units and only put the unit in the kernel when there's a reason to do so.
For example:
If your device has a shared resource ( like an MMU, hardware FIFO ) and you want multiple processes to be able to use it safely, then probably you need some buffer manager to be in the kernel and all the processes would be communicating with it through ioctl.
If your driver needs to respond to an interrupt as fast as possible ( very low-latency ), then you will need to put the part of the code that handles interrupts in the kernel interrupt handler rather than putting it in user space and notifying user space when an interrupt occurs.
Linux kernel 2.6
I've got an fpga that is loaded over GPIO connected to a development board running linux.
The fpga will transmit and receive data over the pci-express bus. However, this is enumerated
at boot and as such, no link is discovered (because the fpga is not loaded at boot).
How can I force re-enumeration of the pci-e bus in linux?
Is there a simple command or will I have to make kernel changes?
I need the capability to hotplug pcie devices.
As root, try the following command:
echo "1" > /sys/bus/pci/rescan
See this link for more information: http://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
I wonder what platform you are on: A work around (aka hack) for this that works on x86 systems is to have the BIOS basically statically configure a PCI device at whatever bus, device, function the FPGA normally lands on, then the OS will enumerate the device and reserve the PCI space for it (even though the device isn't really there). Then in your device driver you will have to do some extra things like setup the BARs and int lines manually after the fpga has been programmed. Of course this requires modifying the BIOS, which if you are working with a BIOS vendor you can contract them to make this change for you, if you are not working with a BIOS vendor then it will be much harder... Also keep in mind that I was working on VxWorks on x86, and we had a AMI make a custom BIOS for our boards...
If you don't have a BIOS, then consider programming it in the bootloader, there you already have the ability to read from disk, and adding GPIO capabilities probably isn't too difficult (assuming you are using jtag and GPIOs?), in fact depending on what bootloader you use it might already be able to do GPIO?
The issues with modifying the kernel to do this is that you have to find the sweet spot where you can read the bitfile, before the PCI enumeration... If for example the disk device drivers are initialized after PCI, then obviously you must do some radical changes to the kernel just to read the bitfile prior to PCI enumeration, which might cause other annoying problems...
One other option which you may have already discovered, and which is really only ok for development time: Power up the system, program the fpga board, then do a reset (without power cycle, for example: sudo reboot now), the FPGA should keep its configuration, and linux should enumerate it...
After turning on your computer, the BIOS enumerates the PCI bus and attempts to fulfill all IO space and memory mapped IO (MMIO) requests. It sets up these BAR's initially, and when the operating system loads these BAR's can be changed by the OS as it sees fit while the PCI bus driver enumerates the bus yet again. It is even possible for the superuser of the system to run the command setpci to change these BAR's after the BIOS has already attempted to configure them and the OS has loaded (may cause drivers to fail and several other bad things if done improperly).
I have had to do this in cases where the card in question was not assigned any resources by the BIOS since the region requested required a 64-bit address and the BIOS only operated with 32-bit address assignments. I was able to go in after-the-fact and change these addresses (originally assigned by the BIOS) to whatever addresses I saw fit, insert the kernel module, and my driver would map and use these newly-assigned addresses for the card without knowing the difference.
The problem that exists with hotplugging PCI-Express cards is that the power to the slot, itself, cannot be turned on/off without specific hotplug controllers that need to exist on the motherboard/backplane. Not having these hotplug controllers to turn the slot's power off may lead to shorts between the tiny pins when the card is physically inserted and/or removed if power is still present. Hotplug events, however, can be initiated by either end (the host or the endpoint device). This does not seem to be the case, however if your FPGA already has a link established with the root complex, a possible solution to your problem would be to generate hotplug interrupts to cause a bus rescan in the OS.
There is a major problem, though -- if your card does not actually obtain a link to the root complex, it won't be able to generate any hotplug events; which seems to be the case. After booting, the FPGA should toggle the PRESENT line on the PCIe bus to tell the OS there is a card ready to be enumerated. Once detected, the OS should attempt to establish a link to the card and assign memory regions to the device. After the OS enumerates the card you'll be able to load drivers against it and see it in lspci. You stated you're using kernel 2.6, which does have support for hotplugging and dynamic resource allocation so this method should work as long as your FPGA supports the ability to toggle the PRESENT PCIe line, too.
I have a char device which enables access to an external SPI memory, and I'd like to mmap() the external memory so that I can access it from a program as if it were normal memory.
If I use the usual mmap() page remapping implementation on the char device file, it just enables me to see a device memory region, not its virtual char file...
Is there a trick to allow me to do that?
TIA
If the character device driver provided an mmap implementation, it would work. There's probably a good reason it doesn't:
Memory access instructions create memory transactions on the bus. An SPI memory is not addressable that way (although the SPI controller may use memory-mapped I/O, that's for its own register-level interface, not the memory content). You could build an SPI memory controller with a memory bus interface, I suppose, but you'd lose the device-indepedence of the SPI standard.
Emulating a memory region is possible (grab a page of memory, mark it for no access, and deal with the access violations (SIGBUS and SIGSEGV), but that would be horribly inefficient. Sometimes you find virtual machines doing this, but performance is very bad.
It sounds like you would need some sort of driver that translates the memory region accesses to commands sent through the character-oriented interface. This would probably be a pretty straight-forward block device driver.