I need to find a problem of a third party hypervisor which leads the kernel boot failure in the AHCI disk driver boot process.
So, is there any way to add print boot up message while kernel boot up? The kernel boot process is failure, so it is impossible to see printk information after system boot.
How to debug kernel, while the system boot up, thanks
Following are some suggestions you can try out:
Enable CONFIG_EARLY_PRINTK found in Kenrel hacking -> Kernel low-level debugging functions menuconfig option
If you have UART debug access, select appropriate UART port for early printk
Boot your kernel with additional debug arguments likes debug earlyprintk
You can also try adding some prints in do_initcall_level() found in init/main.c. To print the name of the function pointer, you can use printk with %pS format specifier. More details on format specifiers for printk can be found here: https://www.kernel.org/doc/Documentation/printk-formats.txt
Hope this helps!
Related
I think I have once read about this but can't find it now.
I'm running linux-5.4.188 on a qemu arm64 virtual machine. Because I built the kernel from the source, I can debug(analyze) the kernel by attaching to the linux kernel program running on a remote machine(qemu virtual machine). To test an application which uses our device(the device model is in qemu too), I compiled a device driver against kernel 5.4.188 and the linux application and can do insmod the driver and run the application.
Now something is wrong and I have panic while running the application. I can debug linux kernel itself, but I don't know where the kernel module was loaded, so the debugger cannot debug the driver module. How can I debug the device driver? (or even the application? in case I need to someday). I remember by first getting the loaded address of the kernel module, and doing add-symbol-file for the driver image relative to that loaded address, it is possible to do kernel module debug. I think this is what driver developers will be doing always. Please tell me how I can do it. If this is possible, it will save many days for me.
As Ian Abbott said I use printk for kernel debugging. Usually for this I put his this line in /etc/sysctl.conf. (for ubuntu-20.04 case. I'm not sure it's applicable to vanila linux too)
kernel.printk = 5 4 1 7
The first value is the console log level and the second value is default log level. By making default 4 (WARNING) and console log level 5 (NOTICE), every default printk appears in the console (because 5 is higher thatn 4, less value is more important and more important messages than the console log level gets printed in the console. see How can I show printk() message in console?).
This way you don't have to check with dmesg everytime. You can see the kernel __log_buf before the serial port is initialized in setup_arch function in start_kernel. The kernel message are written in memory called __log_buf before serial port is initialized.
I refered to https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html that stark pointed me to and I also found https://wiki.st.com/stm32mpu/wiki/Debugging_the_Linux_kernel_using_the_GDB very useful.
Important things during the kernel config :
set CONFIG_GDB_SCRIPTS=y, CONFIG_DEBUG_INFO_REDUCED=n, CONFIG_FRAME_POINTER=y.
I use nokaslr in the kernel command line, but added CONFIG_RANDOMIZE_BASE=n to make sure.
Now the part I originally wanted to know from my question :
In the shell (in virtual machine linux shell) do,
$ls -la /sys/module/<module_name>/sections
Then you'll see files like .text, .data, .bss, etc. You cd to that directory and do
$cat .text .data .bss
To see the address of each section. In my case,
/sys/module/axpu_ldd_kc/sections # cat .bss .data .text
0xffff800008ca5480
0xffff800008ca5000
0xffff800008ca0000
and in the gdb, (after stopping the program by ctrl-c) I did
add-symbol-file ~/testlin540/axpu_ldd_kc.ko 0xffff800008ca0000 -s .data 0xffff800008ca5000 -s .bss 0xffff800008ca5480
and I tried setting breakpoint at my ioctl function.
(gdb) b axpu_ioctl
and when I pressed c (continue) and when I started my application, I could see the program stop at the axpu_ioctl and I could single step through the code and see the values.
When I did kernel debug for booting using u-boot recently, I frequently wrapped a function with
#pragma GCC push_options
#pragma GCC optimize ("O0")
and
#pragma GCC pop_options
To prevent some parts of the codes from being optimized away. (sometime you should do it for the related #include <xxx.h> statement too to prevent compile error).
I am compiling a linux distribution(openwrt) for beaglebone-black and getting the following o/p on my serial port
uncompressing Linux....done,booting the kernel(kernel hangs) in order to trace the error i have enabled CONFIG_DEBUG_LL=y(enables the low level debuging) and CONFIG_DEBUG_INFO=y(enables kernel debug information) but i found CONFIG_DEBUG_LL_UART_NONE=y so if i am correct this means the platform does not support UART debugging(even though kernel is giving the information i might not be seeing that) so is there any other way to debug the kernel or enable the serial port on beagle bone black? if any 1 can guide me through that it would be a great help.....
thanks,
If you have a debugger, then you can certainly debug why your kernel is not booting up.
Assuming, you have one, do the following:
In Kernel configuration, enable CONFIG_DEBUG_INFO (Kernel Hacking ->
Compile-time checks and compiler options)
Compile the kernel
From the debugger software, load the symbols from the elf. The
command varies from one debugger to another
Find symbol "log_buf", and chose option to "Display memory as dump"
There you can see the kernel logs that are not yet printed but stored in the buffer. Hope it helps!
I'm trying to understand the meaning of the following message:
irq N:nobody cared (try booting with the "irqpoll" option)
Does this mean the IRQ handler not processing the response even it has gotten the interrupt? Or that the scheduler failed to call an irq handler?
In what condition is this happening?
it means that either no handler is registered for that irq
or the one that is returned status indicating that the irq was not for him (from hardware he is supporting) in case of shared interrupts
probably a faulty HW/FW or buggy driver
Ideally, the above message should be followed by a stack trace, which should help you determine which subsystem is causing the issue. This message means the interrupt handler got stuck due to a overhead, and did not return thus causing the system to disable IRQ#X. This is seen in cases of a buggy firmware.
The irqpoll option needs to be added to grub.conf, which means, when an interrupt is not handled, search all known interrupt handlers for the appropriate handlers and also check all handlers on each timer interrupt. This is sometimes useful to get systems with broken firmware running. The kernel command line in grub.conf should look like the following:
kernel /vmlinuz-version ro root=/dev/sda1 quiet irqpoll
Minimal runnable QEMU example
QEMU has an educational device called edu that generates interrupts, and is perfect to explore this.
First, I have created a minimal Linux PCI device driver for it, which handles the interrupt correctly.
Now we can easily generate the error by commenting out request_irq and free_irq from the code.
Then, if we run the userland program that generates IRQs, we get:
irq 11: nobody cared (try booting with the "irqpoll" option)
followed by a stack trace.
So as others mentioned: unhandled IRQs.
In my case after reloading the driver because the network card had billions of errors in a short period of time.
modprobe -r ixgbe && modprobe ixgbe
lspci showed an unknown device where the 'card' used to be
after a reboot the card disappeared never to be seen again.
So the error might also show failing hardware.
see here:
static inline int bad_action_ret(irqreturn_t action_ret)
{
if (likely(action_ret <= (IRQ_HANDLED | IRQ_WAKE_THREAD)))
return 0;
return 1;
}
I did not find a good method when I write and test a linux kernel programs such as multiple kernel-level threads or other general kernel modules, what methods do you take? thanks in advance!
printk, printk and more printk.
use dmesg to view. crash the kernel sometimes deliberately to get the crashinfo, then you can decode the crashinfo
dumptrace(), dumpstack() will print the stacktrace on the dmesg.
As a last option, kgdb. but this requires a connection to another system and is a pain always to get it work.
Is there anything that the kernel need to get from the boot loader.Usually the kernel is capable of bringing up a system from scratch, so why does it require anything from boot-loader?
I have seen boot messages from kernel like this.
"Fetching vars from bootloader... OK"
So what exactly are the variables being passed?
Also how are the variables being passed from the boot-loader? Is it through stack?
The kernel accept so called command-line options, that are text based. This is very useful, because you can do a lot of thing without having to recompile your kernel. As for the argument passing, it is architecture dependent. On ARM it is done through a pointer to a location in memory, or a fixed location in memory.
Here is how it is done on ARM.
Usually a kernel is not capable of booting the machine from scratch. May be from the bios, but then it is not from scratch. It needs some initialisation, this is the job of the bootloader.
There are some parametres that the Linux kernel accepts from the bootloader, of which what I can remember now is the vga parametre. For example:
kernel /vmlinuz-2.6.30 root=/dev/disk/by-uuid/3999cb7d-8e1e-4daf-9cce-3f49a02b00f2 ro vga=0x318
Have a look at 10 boot time parameters you should know about the Linux kernel which explains some of the common parametres.
For the Linux kernel, there are several things the bootloader has to tell the kernel. It includes things like the kernel command line (as several other people already mentioned), where in the memory the initrd has been loaded and its size, if an initrd is being used (the kernel cannot load it by itself; often when using an initrd, the modules needed to acess storage devices are within the initrd, and it can also have to do some quite complex setup before being able to access the storage), and several assorted odds and ends.
See Documentation/x86/boot.txt (link to 2.6.30's version) for more detail for the traditional x86 architecture (both 32-bit and 64-bit), including how these variables are passed to the kernel setup code.
The bootloader doesn't use a stack to pass arguments to the kernel. At least in the case of Linux, there is a rather complex memory structure that the bootloader fills in that the kernel knows how to parse. This is how the bootloader points the kernel to its command line. See Documentaion/x86/boot.txt for more info.
Linux accepts variables from the boot loader to allow certain options to be used. I know that one of the things you can do is make it so that you don't have to log-in (recovery mode) and there are several other options. It mainly just allows fixes to be done if there's an issue with something or for password changing. This is how the Ubuntu Live-CD boots Linux if you select to use another option.
Normally the parameters called command line parameters, which is passed to kernel module from boot loader. Bootloader use many of the BIOS interrupts to detect,
memory
HDD
Processor
Keyboard
Screen
Mouse
ETC...
and all harwares details are going to be detected at boot time, that is in real mode, then pass this parameters to Kernel.