I am running qemu-system-arm with kvm following this tutorial :
http://www.virtualopensystems.org/media/chromebook/chromebook.pdf
After reading some documentations (mostly blogs, lkml, and mailing list archives) and skim through codes (qemu, kvm, kernel), also, utilizing kernel trace function to trace kvm events, I can ascertain that vcpu thread runs guest code until hyp-trap occurs, and this is, mostly because of guest accessing mmio address for:
1. gpio access (I can't figure out what)
2. virtio-net, accessing ethernet
3. serial access
the execution then goes back to kvm code to request userspace device emulation to qemu, and back again to guest.
My question is, for framebuffer, when does qemu have time to update the display (sdl)? if I am not mistaken, framebuffer memory range is within RAM itself, thus it is not MMIO. So, it does not interrupt cpu, nor it generates hyp-trap.
Related
Background
I am having an issue where on a 32-bit Linux kernel my PCIe driver, with multi-MSI, and custom hardware device would work perfectly, now I've moved to a 64-bit iMX8MM CPU and MSI interrupts no longer trigger. The driver registers the interrupts correctly without any errors and I can view the output of cat /proc/interrupts and see all MSI vectors listed for my driver along with their interrupt count, which is zero. There is a bridge but /sys/bus/pci/devices/<address>/msi_bus has a 1 so it is enabled. Everything from lspci -vv looks in order and CONFIG_PCI_MSI was applied to the kernel. I am able to read and write memory buffers on the PCIe device so communications works correctly, I just can't get MSI interrupts to trigger. This is the exact same driver code running on the 32-bit CPU, which worked, and was recompiled on the new 64-bit CPU.
Question
How do I force an MSI interrupt on the system to see if Linux will pick it up and execute my code? From lspci -vv I can see the MSI address and I know the MSI vector value to write. So where do I write this on the system? I've looked at /dev/mem and other areas where I could hopefully do an echo 0x1 > <address>, or some other command line function, to write a value and trigger it. Where/how do I write to trigger a specific MSI interrupt?
An MSI can’t be triggered by a CPU; the dword write that triggers the interrupt is only recognized as an interrupt when it comes from a device.
A CPU can initiate an interrupt in a similar way by writing to the local APIC ICR register. This can only be done from the kernel, and as far as I know there is no existing kernel service to allow injecting an arbitrary interrupt. Maybe you can add a service to your kernel driver to do it. (Note, as a temporary debugging aid; this would be terribly inappropriate to leave in place in a driver.)
The fields of the ICR aren’t identical to those of an MSI, but they are similar. It should be straightforward to compare the documentation for the two to map one to the other.
In reference to the QEMU x86_64 machine option kernel_irqchip=on|off, the description reads:
Controls in-kernel irqchip support for the chosen accelerator when available
What is an "irqchip"?
An "irqchip" is KVM's name for what is more usually called an "interrupt controller". This is a piece of hardware which takes lots of interrupt signals (from devices like a USB controller, disk controller, PCI cards, serial port, etc) and presents them to the CPU in a way that lets the CPU control which interrupts are enabled, be notified when a new interrupt arrives, dismiss a handled interrupt, and so on.
An emulated system (VM) needs an emulated interrupt controller, in the same way that real hardware has a real hardware interrupt controller. In a KVM VM, it is possible to have this emulated device be in userspace (ie QEMU) like all the other emulated devices. But because the interrupt controller is so closely involved with the handling of simulated interrupts, having to go back and forth between the kernel and userspace frequently as the guest manipulates the interrupt controller is bad for performance. So KVM provides an emulation of an interrupt controller in the kernel (the "in-kernel irqchip") which QEMU can use instead of providing its own version in userspace. (On at least some architectures the in-kernel irqchip is also able to use hardware assists for virtualization of interrupt handling which the userspace version cannot, which further improves VM performance.)
The default QEMU setting is to use the in-kernel irqchip, and this gives the best performance. So you don't need to do anything with this command line option unless you know you have a specific reason why the in-kernel irqchip will not work for you.
I find the boundary between QEMU and KVM very blurred. I find that someone says a virtual machine is a qemu process while others say a kvm process. What is it exactly?
And what role does QEMU and KVM plays in virtual machine I/O? For example, when a vm does PIO/MMIO, is it qemu or kvm that will trap it and turns it to hardware operation. Or both are responsible?
KVM: the code in the Linux kernel which provides a friendly interface to userspace for using the CPU virtualization. This includes functions that userspace can call for "create CPU", "run CPU", etc. For a full virtual machine, you need to have some userspace code which can use this. This is usually either QEMU, or the simpler "kvmtool"; some large cloud providers have their own custom userspace code instead.
QEMU: emulates a virtual piece of hardware, including disks, memory, CPUs, serial port, graphics, and other devices. Also provides mechanisms (a UI, and some programmable interfaces) for doing things like starting, stopping, and migrating. QEMU supports several different 'accelerator' modes for how it handles the CPU emulation:
TCG: pure emulation -- works anywhere, but very slow
KVM: uses the Linux kernel's KVM APIs to allow running guest code using host CPU hardware virtualization support
hax: similar to KVM, but using the Intel HAXM code, which will work on Windows hosts
From an implementation point of view the boundary between KVM and QEMU is very clear -- KVM is a part of the host kernel, whereas QEMU is a separate userspace program. For a user, you typically don't have to care where the boundary is, because that's an implementation detail.
To answer your question about what happens for MMIO:
the guest makes an MMIO access
this is trapped to the host kernel by the hardware
the host kernel (KVM) might then emulate this MMIO access itself, because a few devices are implemented in the kernel for performance reasons. This usually applies only to the interrupt controller and perhaps the iommu.
otherwise, KVM reports the MMIO access back to userspace (ie to QEMU, kvmtool, etc)
userspace then can handle the access, using its device emulation code
userspace then returns the result (eg the data to return for a read) to the kernel
the kernel updates the vcpu's register state as required to complete emulation of the instruction
the kernel then resumes execution of the VM at the following instruction
i'm studying how virtio works with qemu and kvm.
And i have two questions.
so far, i understood virtio frond-end drivers (in guests) write IO requests in vring(virtqueue) and kick qemu.
Then qemu is notified, translate requests in vring and invoke syscalls such as open, write, read, close, and so on.
Q1. how to share vring between virtio front-end drivers and qemu? I got some information that memory map is used from http://www.slideshare.net/zenixls2/052-virtio-introduction-17191942. But, I cannot find it in source codes. Please let me know where it is in source code.
Q2. how to kick qemu?
I cannot understand how front-end drivers kick qemu?
I think qemu's memory listeners recieve and handle the kick.
But i cannot find it in source code.
Front-end drivers kick QEMU by writing to an I/O port (in the PCI virtio device's I/O BARs; you can find the address with lspci).
To share memory between the guest and the virtio device, QEMU does "DMA" with address_space_map and address_space_unmap (or cpu_physical_memory_map and cpu_physical_memory_unmap instead depending on the QEMU version).
Linux kernel 2.6
I've got an fpga that is loaded over GPIO connected to a development board running linux.
The fpga will transmit and receive data over the pci-express bus. However, this is enumerated
at boot and as such, no link is discovered (because the fpga is not loaded at boot).
How can I force re-enumeration of the pci-e bus in linux?
Is there a simple command or will I have to make kernel changes?
I need the capability to hotplug pcie devices.
As root, try the following command:
echo "1" > /sys/bus/pci/rescan
See this link for more information: http://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
I wonder what platform you are on: A work around (aka hack) for this that works on x86 systems is to have the BIOS basically statically configure a PCI device at whatever bus, device, function the FPGA normally lands on, then the OS will enumerate the device and reserve the PCI space for it (even though the device isn't really there). Then in your device driver you will have to do some extra things like setup the BARs and int lines manually after the fpga has been programmed. Of course this requires modifying the BIOS, which if you are working with a BIOS vendor you can contract them to make this change for you, if you are not working with a BIOS vendor then it will be much harder... Also keep in mind that I was working on VxWorks on x86, and we had a AMI make a custom BIOS for our boards...
If you don't have a BIOS, then consider programming it in the bootloader, there you already have the ability to read from disk, and adding GPIO capabilities probably isn't too difficult (assuming you are using jtag and GPIOs?), in fact depending on what bootloader you use it might already be able to do GPIO?
The issues with modifying the kernel to do this is that you have to find the sweet spot where you can read the bitfile, before the PCI enumeration... If for example the disk device drivers are initialized after PCI, then obviously you must do some radical changes to the kernel just to read the bitfile prior to PCI enumeration, which might cause other annoying problems...
One other option which you may have already discovered, and which is really only ok for development time: Power up the system, program the fpga board, then do a reset (without power cycle, for example: sudo reboot now), the FPGA should keep its configuration, and linux should enumerate it...
After turning on your computer, the BIOS enumerates the PCI bus and attempts to fulfill all IO space and memory mapped IO (MMIO) requests. It sets up these BAR's initially, and when the operating system loads these BAR's can be changed by the OS as it sees fit while the PCI bus driver enumerates the bus yet again. It is even possible for the superuser of the system to run the command setpci to change these BAR's after the BIOS has already attempted to configure them and the OS has loaded (may cause drivers to fail and several other bad things if done improperly).
I have had to do this in cases where the card in question was not assigned any resources by the BIOS since the region requested required a 64-bit address and the BIOS only operated with 32-bit address assignments. I was able to go in after-the-fact and change these addresses (originally assigned by the BIOS) to whatever addresses I saw fit, insert the kernel module, and my driver would map and use these newly-assigned addresses for the card without knowing the difference.
The problem that exists with hotplugging PCI-Express cards is that the power to the slot, itself, cannot be turned on/off without specific hotplug controllers that need to exist on the motherboard/backplane. Not having these hotplug controllers to turn the slot's power off may lead to shorts between the tiny pins when the card is physically inserted and/or removed if power is still present. Hotplug events, however, can be initiated by either end (the host or the endpoint device). This does not seem to be the case, however if your FPGA already has a link established with the root complex, a possible solution to your problem would be to generate hotplug interrupts to cause a bus rescan in the OS.
There is a major problem, though -- if your card does not actually obtain a link to the root complex, it won't be able to generate any hotplug events; which seems to be the case. After booting, the FPGA should toggle the PRESENT line on the PCIe bus to tell the OS there is a card ready to be enumerated. Once detected, the OS should attempt to establish a link to the card and assign memory regions to the device. After the OS enumerates the card you'll be able to load drivers against it and see it in lspci. You stated you're using kernel 2.6, which does have support for hotplugging and dynamic resource allocation so this method should work as long as your FPGA supports the ability to toggle the PRESENT PCIe line, too.