Linux kernel 2.6
I've got an fpga that is loaded over GPIO connected to a development board running linux.
The fpga will transmit and receive data over the pci-express bus. However, this is enumerated
at boot and as such, no link is discovered (because the fpga is not loaded at boot).
How can I force re-enumeration of the pci-e bus in linux?
Is there a simple command or will I have to make kernel changes?
I need the capability to hotplug pcie devices.
As root, try the following command:
echo "1" > /sys/bus/pci/rescan
See this link for more information: http://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
I wonder what platform you are on: A work around (aka hack) for this that works on x86 systems is to have the BIOS basically statically configure a PCI device at whatever bus, device, function the FPGA normally lands on, then the OS will enumerate the device and reserve the PCI space for it (even though the device isn't really there). Then in your device driver you will have to do some extra things like setup the BARs and int lines manually after the fpga has been programmed. Of course this requires modifying the BIOS, which if you are working with a BIOS vendor you can contract them to make this change for you, if you are not working with a BIOS vendor then it will be much harder... Also keep in mind that I was working on VxWorks on x86, and we had a AMI make a custom BIOS for our boards...
If you don't have a BIOS, then consider programming it in the bootloader, there you already have the ability to read from disk, and adding GPIO capabilities probably isn't too difficult (assuming you are using jtag and GPIOs?), in fact depending on what bootloader you use it might already be able to do GPIO?
The issues with modifying the kernel to do this is that you have to find the sweet spot where you can read the bitfile, before the PCI enumeration... If for example the disk device drivers are initialized after PCI, then obviously you must do some radical changes to the kernel just to read the bitfile prior to PCI enumeration, which might cause other annoying problems...
One other option which you may have already discovered, and which is really only ok for development time: Power up the system, program the fpga board, then do a reset (without power cycle, for example: sudo reboot now), the FPGA should keep its configuration, and linux should enumerate it...
After turning on your computer, the BIOS enumerates the PCI bus and attempts to fulfill all IO space and memory mapped IO (MMIO) requests. It sets up these BAR's initially, and when the operating system loads these BAR's can be changed by the OS as it sees fit while the PCI bus driver enumerates the bus yet again. It is even possible for the superuser of the system to run the command setpci to change these BAR's after the BIOS has already attempted to configure them and the OS has loaded (may cause drivers to fail and several other bad things if done improperly).
I have had to do this in cases where the card in question was not assigned any resources by the BIOS since the region requested required a 64-bit address and the BIOS only operated with 32-bit address assignments. I was able to go in after-the-fact and change these addresses (originally assigned by the BIOS) to whatever addresses I saw fit, insert the kernel module, and my driver would map and use these newly-assigned addresses for the card without knowing the difference.
The problem that exists with hotplugging PCI-Express cards is that the power to the slot, itself, cannot be turned on/off without specific hotplug controllers that need to exist on the motherboard/backplane. Not having these hotplug controllers to turn the slot's power off may lead to shorts between the tiny pins when the card is physically inserted and/or removed if power is still present. Hotplug events, however, can be initiated by either end (the host or the endpoint device). This does not seem to be the case, however if your FPGA already has a link established with the root complex, a possible solution to your problem would be to generate hotplug interrupts to cause a bus rescan in the OS.
There is a major problem, though -- if your card does not actually obtain a link to the root complex, it won't be able to generate any hotplug events; which seems to be the case. After booting, the FPGA should toggle the PRESENT line on the PCIe bus to tell the OS there is a card ready to be enumerated. Once detected, the OS should attempt to establish a link to the card and assign memory regions to the device. After the OS enumerates the card you'll be able to load drivers against it and see it in lspci. You stated you're using kernel 2.6, which does have support for hotplugging and dynamic resource allocation so this method should work as long as your FPGA supports the ability to toggle the PRESENT PCIe line, too.
Related
Is it possible to flash/write to a BIOS from kernel mode in Linux?
I've been doing some research on this and can't find a definitive answer to this. I'm not so great with kernel level stuff and hardware.
From what I've been able to find, I know certain kernel facilities can interrogate the BIOS (see dmidecode) given the BIOS supports certain interfaces.
I know the difference between real and protected mode. But switching to real mode from Linux seems impossible to do (?). I also know x86 has emulation for 8088 programs but unsure whether the emulation would allow flashing the BIOS.
Wouldn't it be possible to just write to certain addresses in kernel mode to "flash" the BIOS?
Update Working from the answer and comments below it seems the answer is yes depending on the hardware platform. The only, and necessary requirement is that the BIOS flash chip is addressable in the IO address space. You also need software support fir flashing chips, whether kernel or user space. For example I found the user space utility flashrom that can do it on what seems to be a narrow set of hardware platforms.
Yes, you can do that if BIOS flash chip is connected to the IO address bus and you have all necessary drivers.
I would like to ask your advice on the following: I need to write drivers for omap3, for accessing external dsp through fpga (through gpmc interface). The dsp is required to load file to dsp, and to read/write buffer from dsp. There is already FPGA driver in kernel. The kernel is 2.6.32. So I thought of the following options:
writing dsp driver in kernel, which uses the existing fpga driver.
writing a user space driver which interfaces with the fpga kernel driver.
writing user space driver using UIO , which will not use the kernel fpga driver, but shall do the access to fpga, as part of the user space single and complete dsp driver.
What do you think is prefered option ?
What is the advantage of kernel driver over user sace and vise versa ?
Thanks, Ran
* User-space driver:
Easier to debug.
Loads of libraries to support you.
Allows you to hide the details of your IP if you want to ( people will really hate you if you did! )
A crash won't affect the whole system.
Higher latency in handling interrupts since the kernel will have to relay the interrupt to the user space somehow.
You can't control access to your device from user space.
* Kernel-space driver:
Harder to debug.
Only linux kernel frameworks are supported.
You can always provide a binary blob to hide the details of your IP but this is annoying since it has to be generated against a specific kernel.
A crash will bring down the whole system.
Less latency in handling interrupts.
You can control access to your device from kernel space because it's a global context that all processes see.
As a kernel engineer I'm more comfortable/happy hacking code in a kernel context, that's probably why I would write the whole driver in the kernel.
However, I would say that the best thing to do is to divide the functionality of your driver into units and only put the unit in the kernel when there's a reason to do so.
For example:
If your device has a shared resource ( like an MMU, hardware FIFO ) and you want multiple processes to be able to use it safely, then probably you need some buffer manager to be in the kernel and all the processes would be communicating with it through ioctl.
If your driver needs to respond to an interrupt as fast as possible ( very low-latency ), then you will need to put the part of the code that handles interrupts in the kernel interrupt handler rather than putting it in user space and notifying user space when an interrupt occurs.
I have been studying linux driver recently,
as those articles I read said, the device driver modules are likely to be automatically loaded on demand by kernel, I am therefore wondering about the recipe how kernel figures out which module to load for a specific device(sound card, I2C/spi device, etc), I also cannot thoroughly imagine how the kernel detects each hardware device while boot-time .
answers relevant to embedded linux are prefered , PC linux are also welcome !
3Q
I think you are mixing two different things, which is hardware detection, and on demand module loading.
In some cases, the kernel is explicitely doing a module request. However, in most cases, the kernel itself does not do any "on demand loading".
But wait, you must be mistaken, if I plug my shiny new webcam, isn't
the module automagically loaded ?
Yes it is, but not by the kernel. All the kernel does is calling a userspace program with so called "hotplug event" or "uevent" as arguments. On Linux PC, this userspace program is usually udev, but on embedded system, you can use for example mdev. You can find a more detailed explanation here and here
Regarding the second part of your question, the kernel is doing hardware discovery only if the hardware is discoverable. Example of discoverable hardware is USB and PCI. Example of non discoverable harwdare busses is SPI or I2C.
In the latter cases, the presence of a particular device on a given bus is either encoded directly in the kernel, or given to him by the booloader. Google for "device tree" for an example of the latter.
To sum things up : Hardware detection is done by the kernel, and module loading is done by userspace, with information provided by the kernel.
I am looking at some pointers for understanding how the Linux kernel implements the setting up of various hardware clocks. This basically relates to working with setting up the various clocks that hardware features like the LCD, UART etc will use. For example when Linux boots how does it handle setting up the clocks for UART or USB. Maybe something like a Clock manager or something.
I am basically trying to implement something similar for a different OS on a new hardware that i am working on. Any help would be really appreciated.
[Edit]
Thanks for the replies and the links. So here is what i have implemented up until now. This should give you an idea of where I'm headed.
I looked up the Hardware Reference Manual for the particular system I'm targeting and wrote some code to monitor/modify the signals/pins of the peripherals I am interested in i.e. turning them ON/OFF from the command line.Now a collection of these clocks/signals together control a peripheral.The HRM would say that if you want to turn on the UART or something then turn on such and such signals/pins. And #BjoernD yes I am using something like a mmap() function to talk to the peripherals.
The meat of my question is that I want to understand the design and implementation of a Clock/Peripheral Manager which uses the utility that I have already written. This Clock/Peripheral Manager would give me the control of enabling/disabling the peripherals I want.Basically this Manager would enable me to make changes in the init code that is right now running. Also during run time processes can call this Manager to turn ON/OFF the devices so that power consumption is optimized. It might not have made perfect sense but I'm myself trying to wrap my head around this.
Now I'm sure something like this would have been implemented in Linux or for that matter any OS for performance issues (nobody would want to waste power by turning on all peripherals at boot time). I want to understand the Software Architecture of it. Reference from any OS would do as of now to atleast get a headstart. Also I am not writing my own OS, there is an OS in place but Im looking more at a board level software aka BSP to work on. But thanks for the OS link anyways, they are really good. Appreciate it.
Thanks!
What you want to achieve is highly specific to a) the platform you are using and b) the device you want to use. For instance, on x86 there are 3 ways to communicate with a device:
Interrupts allow the device to signal the CPU. The OS usually provides mechanisms to register interrupt handlers - functions that are called upon occurrence of an interrupt. In Linux see request_irq() and friends in linux/include/interrupt.h
Memory-mapped I/O is physical memory of the device that the platform's BIOS makes available in the same way you also access plain physical memory - simply by writing to a memory address. What exactly is behind such memory (e.g., network interface config registers or an LCD frame buffer) depends on the device and is usually specified in the device's data sheet.
I/O ports are accessed through a special address space and special instructions (INB/OUTB & co.). Other than that they work similar to I/O memory.
There's a multitude of ways to find out what resources a device provies and where the BIOS mapped them. Some platforms use ACPI tables (google yourself for the 1,000k page spec), PCI provides info on devices in a standardized way through the PCI config space, USB has similar ways of discovering devices attached to the bus, and some devices, e.g., UARTS, are simply specified to be available at a pre-configured I/O range that is fixed for your platform.
As a start for understanding Linux, I'd recommend "Understanding the Linux kernel". For specifics on how Linux handles devices and what is there to write drivers, have a look at Linux Device Drivers. Furthermore, you will need to have a look at the peculiarities of your platform and the device you want to drive.
If you want to start an own OS, a UART is certainly something that will be veeery helpful to print debug output, so you might want to go for this first.
Now that I wrote down all this, it seems that your actual question is: How to get started with Operating System design. This question should be highly valuable for you: What are some resources for getting started in operating system development?
The two big power users in most computers are the CPU and the disks. Both of these have capabilities for power saving in Linux. The CPU clock can be slowed down when the system is not busy, and the disk motors can be stopped when no I/O is happening. For a UART, even if you save all of the power that it uses by turning off its clock, it is still tiny compared to the others because a UART doesn't have much logic in it.
Best ways to save power are
1) more efficient power supply
2) replace rotating disk with SSD
3) Slow down the CPU and memory bus
I'm looking for either a kernel mode call that I can make from a driver, a userland utility, or a system call that will ask the Kernel to look at the PCI bus and either completely re-run its initialization, or initialize a specific device. Specifically, I need the Kernel to recognize a device that was added to the bus after boot and then configure its address space, interrupt, and other configuration parameters, and finally enable the device so that I can load the driver for it (unless this all happens as part of the driver load).
I'm stuck on the 2.4.x series Kernel for this, and am currently working with 2.4.20, but will be moving to 2.4.37 if it matters. The distro is a stripped down Red Hat 7.3 running in a ram disk, but I can add in whatever tools are needed to get this working (as long as they play nice with 2.4 series).
If some background would help clarify what I'm trying to do: From a cold boot, once in Linux I use GPIO to program an FPGA. Part of the FPGA, once programmed, implements a simple PCI device. Currently, after programming the FPGA, I reboot the system and Linux recognizes the device after coming up and loads the driver for it.
Instead of needing that reboot, I'd like to simply ask the Kernel to do whatever it does during boot up to find PCI devices (I have the Kernel configured to find PCI devices on its own, instead of asking the BIOS for that information, so the BIOS won't need to know about this device (I hope)).
I believe that Linux is capable of seeing the device after it is programmed but before a reboot, because scanpci will show the device after I program it, as will lspci -H 1. I just need a way to get it into /proc/pci, configured and enabled.
This below command will help the user to rescan it complete root hub.
echo "1" > /sys/class/pci_bus/0000\:00/rescan
You could speed up the reboot with kexec, if you don't figure out how to get the PCI scan redone. You could ask this on the LKML, if you haven't already.
unloading/reloading the module doesn't help, does it?
http://www.linuxjournal.com/article/5633 suggests you should be able to do it with 2.4 kernels using pcihpfs.
If that isn't working, maybe the driver doesn't support hotplug?
It would probably crash the system if you reconfigured the addresses of other PCI devices while they are in use.
A better way would be to just configure the new card. If your kernel has support for Cardus devices, it already knows how to configure a newly-inserted PCI device (which is what Cardbus is). You just need to figure out how to get the kernel to do it...
It should be possible for a kernel module to do this. Even if you can't get built-in hotplug code, you should be able to set the pci resources using calls to pci_bus_write_config_dword() and friends. There is probably some IRQ routing setup to do as well.