I am having trouble getting bus-mastering DMA working on a device driver in Ubuntu with kernel 5.3.0-68-generic.
I have enabled bus mastering with pci_set_master (when using lspci -v, the PCIe device will have the bus_master flag) and I am allocating a DMA buffer with dma_alloc_coherent.
I take the dma_addr_t returned by the dma alloc and pass that to the device and then I use the kernel virtual address with a chrdev mmap driver to map the address into userspace (using remap_pfn_range) where a userspace driver can populate the DMA memory region.
It doesn't appear that the the PCIe device can see the memory updates in the DMA region, is there perhaps some dma, iommu, or pci settings I need to enable to allow the PCIe device to read back into system memory as the bus master?
Related
My understanding is that if a PCI device want to do DMA RW, and IOMMU is enabled, the driver should map CPU PA into a DMA address via pci_map_page(for non-coherent), then PCI device can use this DMA address and IOMMU will translate the DMA address into CPU PA.
My questions are:
Is it possible for driver to disable IOMMU for a given device?
If someone disable IOMMU via bios, does it mean any CPU PA can be directly DMA RW?
The VT-d hardware allows setting pass-through separately for each device, but Linux does not currently provide a driver API to do it.
Yes, DMA from PCI/PCIe devices uses system physical addresses when the IOMMU is disabled, either in the BIOS or by using intel_iommu=off in the Linux command line.
Existing kernel drivers such as xilinx have specific way to be registered (as tty device), if they are mapped directly to cpu memory map as done here with device tree:
https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842249/Uartlite+Driver
But in other cases, there is a PCIe device (like FPGA which has the xilinx uart IPs) which is connected to and the cpu.
How should we make the uart get registered when using PCIe device ?
The device tree I try to register into PCIe is uartlite driver:
https://github.com/Xilinx/linux-xlnx/blob/master/drivers/tty/serial/uartlite.c
I think that what I probably need to do is:
Write a custom pci driver.
Need to prepare platform_device struct and then call the uart probe routine from pci driver:
ulite_probe(struct platform_device *pdev)
I've seen related question with others using FPGA with multiple device connected, but seems that there is no docuemnt, or tutorial which describes how to do this.
Any comment, example or document is appreciated.
So something like a ARM CPU connected to an Artix FPGA over PCIe right?
Yes, you would need a custom PCIe driver. The PCIe configuration and data spaces would have to be mapped. Have a look at pci_resource_{start, len} and pci_remap_bar functions. You can then use pci_get_device to get a pointer to the struct device and retrieve the virtual address of the PCIe configuration space. The UART driver can then use the struct device pointer and it's register map should be at some offset to the virtual address of the PCIe configuration space as per your design. You can invoke the probe call of UARTlite IP driver in your own driver.
"Existing kernel drivers such as xilinx have specific way to be registered (as tty device), if they are mapped directly to cpu memory map as done here with device tree". Note that this is true if we are only talking of tty devices. A GPIO peripheral IP won't be expose as tty but in /sys/class/gpio.
I know that vfio can expose interrupt, DMA and pci I/O to userspace. I read that if someone want to take advantage of vfio for pci devices, he has to unbind the original driver and bind to vfio-pci driver. So my question is, is vfio-pci a userspace driver for all pci devices? Because in my understanding, vfio just offers some basic interfaces. Or, if I need a driver for a new pci device, should I just use vfio-pci driver or use interfaces it offers to write a new driver?
I am working on a PCIe Linux driver. I would like to register an ISR for the device. IRQ number assigned to the device by the Linux system is 16, which is shared by other(USB host controller) device also. (Checked by lspci -v). It is a pin based interrupt.
By searching online I found almost all PCI driver example just provides only IRQF_SHARED as flag in API request_irq(), and does not provide any other flags to mention behaviour like High/Low level interrupt.
My question is, how the Linux kernel determines the behaviour of shared interrupt (for PCIe device), if it is low level or High level ?
PCIe uses MSI, so there is no hi/low level to be concerned with. Traditional PCI cards use level triggered interrupts, but most devices use active low signaling so this isn't something a driver writer has access to modify/tweak.
DMA engine is not responding correctly on PowerPC linux. When my PCIe device sends a read / write request to host, timeout happens. I have 1GB of RAM at lower address range.
I have called the following functions
pci_device_enable()
pci_set_master()
DMA memory is allocated at 0x0F00_0000 and am able to access this memory from user space from host. How can I debug this problem. What could be the reason?
Please check if the PCIe device is using the correct address for the DMA memory. You only mention one address, but the address(es) of the DMA memory in user and kernel space will be different than the address of the DMA memory as seen from the PCIe bus.
You don't say how you are allocating the DMA memory and there are several different ways.
For instance, if you use the following call to get the DMA memory, the dma_addr_t is the address
to use in the PCIe device.
void *pci_alloc_consistent(struct pci_dev *dev, size_t size,
dma_addr_t *dma_handle);
Check if you defined CONFIG_FSL_DMA part of your kernel config to compile the DMA driver. Also make sure your device tree has an entry for the DMA device on chip.