I am interested in profiling a module for the Linux Kernel. I tried it this way:
Built the kernel with CONFIG_PROFILING=y
Booted the kernel with profile=2
Reset the profiler by echo 123 > /proc/profile
The module was successfully loaded by a dedicated daemon
Run a user-space app that is using the module through specific ioctl calls. The test app worked as expected so the module was working OK.
Read the profiling info using readprofile -p /proc/profile -m System.map where System.app was generated along with the kernel.
So far, so good. Almost everything worked as expected. What did not work however, is that I didn't get any info about my module, i.e. there was no mention of any function from the module.
However, on second thoughts, I remebered that I forgot to recompile the module using the CONFIG_PROFILING=y.
So, my question is: Did the module not show up in the profiling log because it had not been compiled with the CONFIG_PROFILING=y or is there some other more obvious reason that I don't know of?
If one wonders, why I am asking rather than trying it straight away, that is because, I can't do it at home and I'd rather like to hear what the knowledgable would say before I get access to the code again.
It doesn't seem possible for modules to be profiled in this way as they are loaded dynamically, which means that they are missing from the System.map file.
A solution would be to build the driver into the kernel. Unfortunately, this does not always work as the system may be designed to run only with the driver built into a module.
I suggest that using oprofile should work.
Related
I'm new to device drivers in Linux. And my first day task is to debug driver using GDB in Linux.
I need to debug some XYZ (PCIe device driver supports ethernet) device driver to know about the flow and what is going on device's registers and all.
I have installed the driver with patch file and insmod command.
The device is working properly. But am not getting any solution to debug the device driver.
All I know is that how to debug C program using GDB in Linux(fedora20). I got one link similar to my Problem but from that also I have not got any knowledge.
Can anybody please share your thoughts that how can I start from scratch.
I am very specific to learn about Debugging device drivers in Linux. Especially that init or probe function inside my driver I need to know the flow.
The gdb debugger is useful to debug user-space application level programs (since it uses ptrace(2)).
For kernel code, things are different. Consider using kgdb (I don't know the details). You might also add debug prints ....
I recommend at least reading more about operating systems, e.g. Operating Systems: Three Easy Pieces (freely downloadable), and reading something about Linux programming (perhaps the old ALP, and also intro(2), syscalls(2) and related stuff). Don't dare coding Linux loadable kernel modules without good familiarity with Linux programming (in user-land). See also kernelnewbies.
BTW, you should prefer writing user-land code than kernel modules. A very simple rule of thumb is to avoid writing kernel code when possible.
To begin with, you may need to understand basics of device driver and kernel in linux. Subsequently focus as per the type of driver in-hand. You also need to understand the functionality (specification / manual / datasheet) of the device you are working on.
The very basic approach for debugging can be using printk. Normally there will be debug logs that can be enabled through compilation flags. If it is present, enable it so that it can give important pointers else you might need to add it on your own.
Start with verification of driver registration and verification of loading of your driver (static or loadable module as per your requirement). Check whether it is getting listed as part of sysfs or proc as applicable. Check whether probe is successful and subsequently the appropriate read/write/open/close/other calls as per your driver / device functionality.
The dmesg shall be very helpfull for viewing the kernel messages. There are also tools like kdb, LTT, strace that can be useful as per the scenario.
Now a days debugging become so advanced that even 'core kernel source code' can be debugged using Virtual environment.
But after reading couple of blog related to Kernel Core development it was not clear whether they are debugging using Virtual environment.
They have mentioned that they rely on 'Printing message' rather than using debugging tool, at-least for core component.
So, I Request from 'Linux Kernel Experts' to let me know what is good practice followed while debugging Kernel?
I've tried multiple approaches when trying to debug the kernel.
Sometimes, the easiest way is to just add a few printk statements based on my own conditional values, monitor the serial log and see what's going on. Its especially useful when the function in question is invoked quite often, but you are interested only in a subset of those.
QEMU GDB debugging. I have a buildroot filesystem setup. This means the kernel is lean and it boots up real fast. I start qemu with the -s -S flags, and attach gdb as target remote :1234. Additionally, there aren't very many userspace processes in this setup so its easier to debug the kernel.
VMWare stub. Assuming you are running an Ubuntu VM, it is possible that you can attach gdb to a VMware stub and debug the kernel. Personally, I never have had to pursue this route, but I look forward to trying it out someday.
If you have a kernel for a device that gets stuck in a bootloop and it does not print out any debug information out onto serial, it still might be helpful to try and boot it up using QEMU. Sure, the booting up will probably fail as the kernel tries to load up drivers, but you should be able to attach gdb, get a stack trace and see what the root cause is(perhaps a recursive call).
I had an issue with trying to get the wpa_supplicant to run on board running a custom Embedded Linux distribution. After some debugging it turned out that it was because the domain type PF_PACKET was not supported by the kernel:
drv->eapol_tx_sock = socket(PF_PACKET, SOCK_DGRAM, 0);
This is supported by the af_packet module and loading it beforehand fixed the problem. I noticed that it also has the alias "pf-net-17" and it appears that the kernel can load this module at runtime if requested from user-space via the alias. I tested this in another Embedded Linux distribution and sure enough this seems to be the case. It was loaded after the supplicant was called.
Does anyone know why this functionality is not working in the first setup? Maybe I missed a kernel setting?
There are basically 2 reasons why this worked on another system and not on your board:
the PF_PACKET was configured to be built in the kernel or not as a module.
The af_packet module was automatically loaded at runtime, either as a dependancy to another module or as a module to be automatically loaded at boot
I suggest that you read this document about module loading which is quite concise, yet complete and precise.
A way to fix your problem would be to make sure that depmod was run for your target system so that the module dependancy for your build is updated of load the af_packet module automatically at boot.
Another place to look at is udev, which is meant to load modules according to the devices detected by the kernel. Maybe this is also missing/misconfigured in your system.
I'm complety new to this but I finally got around to building a Linux kernel so far so good. I am following a guide here:
A10-OLinuXino-LIME
My problem/concern is now I am in a .config menu and I've been searching online to no avail to determine what does it mean by a modularizes features verse includes, like should I switch to if I want those features to be there?
Any help or advice would be greatly appreciated it! I'm primarily doing this to include WiFi usb drivers I will be needing.
It depends on what system you run this kernel. If it's an embedded system, then you will probably be more concerned about memory benefits. In RAM you can win about several Kbytes per module. For the devices are not represented in the system/hardware, it then has a sense to put some drivers on Modules. Some modules can be taking more time when the kernel starts, and it is probably better to load them later, when the system is running.
You will probably be concerned about disk space, if you put some stuff Compiled-in so you can have a benefit because you don't need to have a module loading utility.
Have a look at this thread as well
I'm primarily doing this to include WiFi usb drivers I will be needing.
Its not necessary to build the complete kernel unless to build a USB WiFi driver. All you need is the kernel headers installed. From make menuconfig select the module you want to build, choose M, save the .config. This will build a module which can be loaded, instead of getting compiled as a part of the vmlinux image.
A linux machine freezes few hours after booting and running software (including custom drivers). I'm looking a method to debug such problem. Recently, there has been significant progress in Linux Kernel debugging techniques, hasn't it?
I kindly ask to share some experience on the topic.
If you can reproduce the problem inside a VM, there is indeed a fairly new (AFAIK) technique which might be useful: debugging the virtual machine from the host machine it runs on.
See for example this:
Debugging Linux Kernel in VMWare with Windows host
VMware Workstation 7 also enables a powerful technique that lets you record system execution deterministically and then replay it as desired, even backwards. So as soon as the system crashes you can go backwards and see what was happening then (and even try changing something and see if it still crashes). IIRC I read somewhere you can't do this and debug the kernel using VMware/gdb at the same time.
Obviously, you need a VMM for this. I don't know what VMM's other than VMware's VMM family support this, and I don't know if any free VMware versions support this. Likely not; one can't really expect a commercial company to give away everything for free. The trial version is 30 days.
If your custom drivers are for hardware inside the machine, then I suppose this probably won't work.
SystemTap seems to be to Linux what Dtrace is to Solaris .. however I find it rather hostile to use. Still, you may want to give it a try. NB: compile the kernel with debug info and spend some time with the kernel instrumentation hooks.
This is why so many are still using printk() after empirically narrowing a bug down to a specific module.
I'm not recommending it, just pointing out that it exists. I may not be smart enough to appreciate some underlying beauty .. I just write drivers for odd devices.
There are many and varied techniques depending on the sort of problems you want to debug. In your case the first question is "is the system really frozen?". You can enable the magic sysrq key and examine the system state at freeze and go from there.
Probably the most directly powerful method is to enable the kernel debugger and connect to it via a serial cable.
One option is to use Kprobes. A quick search on google will show you all the information you need. It isn't particularly hard to use. Kprobes was created by IBM I believe as a solution for kernel debugging. It is essentially a elaborate form of printk() however it allows you to handle any "breakpoints" you insert using handlers. It may be what you are looking for. All you need to do is write and 'insmod' a module into the kernel which will handle any "breakpoints" hit that you specify in the module.
Hope that can be a useful option...
How I debug this kind of bug, was to run my OS inside the VirtualBox, and compile the kernel with kgdb builtin. Then I setup a serial console on the VirtualBox so that I can gdb to the kernel inside the VirtualBox's OS via the serial console. Anytime the OS hang, just like magic sysrq key, I can enter ctrl-c on the gdb to stop and understand the kernel at that point in time.
Normally kernel stack tracing is just too difficult to pinpoint the culprit process, so the best way I think is still generic "top" command, just looking at the application logs to see what are the cause of hanging - this will need a reboot to see the log of course.