I think I have once read about this but can't find it now.
I'm running linux-5.4.188 on a qemu arm64 virtual machine. Because I built the kernel from the source, I can debug(analyze) the kernel by attaching to the linux kernel program running on a remote machine(qemu virtual machine). To test an application which uses our device(the device model is in qemu too), I compiled a device driver against kernel 5.4.188 and the linux application and can do insmod the driver and run the application.
Now something is wrong and I have panic while running the application. I can debug linux kernel itself, but I don't know where the kernel module was loaded, so the debugger cannot debug the driver module. How can I debug the device driver? (or even the application? in case I need to someday). I remember by first getting the loaded address of the kernel module, and doing add-symbol-file for the driver image relative to that loaded address, it is possible to do kernel module debug. I think this is what driver developers will be doing always. Please tell me how I can do it. If this is possible, it will save many days for me.
As Ian Abbott said I use printk for kernel debugging. Usually for this I put his this line in /etc/sysctl.conf. (for ubuntu-20.04 case. I'm not sure it's applicable to vanila linux too)
kernel.printk = 5 4 1 7
The first value is the console log level and the second value is default log level. By making default 4 (WARNING) and console log level 5 (NOTICE), every default printk appears in the console (because 5 is higher thatn 4, less value is more important and more important messages than the console log level gets printed in the console. see How can I show printk() message in console?).
This way you don't have to check with dmesg everytime. You can see the kernel __log_buf before the serial port is initialized in setup_arch function in start_kernel. The kernel message are written in memory called __log_buf before serial port is initialized.
I refered to https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html that stark pointed me to and I also found https://wiki.st.com/stm32mpu/wiki/Debugging_the_Linux_kernel_using_the_GDB very useful.
Important things during the kernel config :
set CONFIG_GDB_SCRIPTS=y, CONFIG_DEBUG_INFO_REDUCED=n, CONFIG_FRAME_POINTER=y.
I use nokaslr in the kernel command line, but added CONFIG_RANDOMIZE_BASE=n to make sure.
Now the part I originally wanted to know from my question :
In the shell (in virtual machine linux shell) do,
$ls -la /sys/module/<module_name>/sections
Then you'll see files like .text, .data, .bss, etc. You cd to that directory and do
$cat .text .data .bss
To see the address of each section. In my case,
/sys/module/axpu_ldd_kc/sections # cat .bss .data .text
0xffff800008ca5480
0xffff800008ca5000
0xffff800008ca0000
and in the gdb, (after stopping the program by ctrl-c) I did
add-symbol-file ~/testlin540/axpu_ldd_kc.ko 0xffff800008ca0000 -s .data 0xffff800008ca5000 -s .bss 0xffff800008ca5480
and I tried setting breakpoint at my ioctl function.
(gdb) b axpu_ioctl
and when I pressed c (continue) and when I started my application, I could see the program stop at the axpu_ioctl and I could single step through the code and see the values.
When I did kernel debug for booting using u-boot recently, I frequently wrapped a function with
#pragma GCC push_options
#pragma GCC optimize ("O0")
and
#pragma GCC pop_options
To prevent some parts of the codes from being optimized away. (sometime you should do it for the related #include <xxx.h> statement too to prevent compile error).
Related
I have a barebone linux kernel with buildroot setup for debugging using QEMU and GDB. I am using the x86_64 architecture.
I want to check how the memory protection works for each process. So basically, I need to find the base and limit values that govern the access to the physical memory.
If I understood correctly, the GDTR register in the x86 architecture "holds the base address (32 bits in protected mode; 64 bits in IA-32e mode) and the 16-bit table limit for the GDT." If not, please let me know where such information is held.
I tried using the i r in GDB but the output does not show the GDTR/LDTR contents. I read somewhere that we can use another method while inside the kernel in order to display the results of these registers.
I also need to check the PCB (Process Control Block) contents. I can't seem to find a way to do so. I read somewhere that if we do memory dump in the kernel, we can get the PCB contents, but I can't figure out how to do so.
So, how can I check the contents of PCB and GDTR/LDTR from gdb?
The setup is a simple qemu that launches the linux kernel with buildroot, connects gdb by using target remote :1234 and execute a simple C program that has fork and exec inside of it.
I am trying to load a kernel module (out-of-tree) and dmesg shows a panic. The kernel is still up though. I guess the module panic'd.
Where to find the core file? I want to use gdb and see whats the problem.
Where to find the core file?
Core files are strictly a user-space concept.
I want to use gdb and see whats the problem.
You may be looking for KGDB and/or Kdump/Kexec.
Normally, whenever the coredump was generated, it will state "core dumped". This could be one high level easy way to confirm whether coredump got generated however, this statement alone cannot guarantee on coredump file availability. The location where coredump is generated is specified through core_pattern to kernel via sysctl. You need to check the information present in core_pattern of your system. Also, note that in case of Ubuntu, it appears that the coredump file size is kept as zero by default which will avoid generation of coredump. So, you might need to check the corefile size ulimit and change it to 'ulimit -c unlimited', if it is zero. The manpage http://man7.org/linux/man-pages/man5/core.5.html explains about various reasons due to which coredump shall not get generated.
However, from your explanation, it appears that you are facing 'kernel oops' as the kernel is still up(unstable state) even though a particular module got panic'd/killed. In such cases, kernel shall print an oops message. Refer to link https://www.kernel.org/doc/Documentation/oops-tracing.txt that has information regarding the kernel oops messages.
Abstract from the link: Normally the Oops text is read from the kernel buffers by klogd and
handed to syslogd which writes it to a syslog file, typically
/var/log/messages (depends on /etc/syslog.conf). Sometimes klogd
dies, in which case you can run dmesg > file to read the data from the
kernel buffers and save it. Or you can cat /proc/kmsg > file, however
you have to break in to stop the transfer, kmsg is a "never ending
file".
printk is used for generating the oops messages. printk does tagging of severity by means of different loglevels /priorities and allows the classification of messages according to their severity. (Different priorities are defined in file linux/kernel.h or linux/kern_levels.h, in form of macros like KERN_EMERG, KERN_ALERT, KERN_CRIT etc..)So, you may need to check the default logging levels in system by using cat /proc/sys/kernel/printk and change it as per your requirement. Also, check whether the logging daemons are up and incase you want to debug kernel, ensure that the kernel is compiled with CONFIG_DEBUG_INFO.
The method to use GDB to find the location where the kernel panicked or oopsed in ubuntu is in the link https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks which can be one of the method that can be used by you for debugging kernel oops.
there won't be a core file.
You should follow the stack trace in kernel messages. type dmesg to see it.
I am compiling a linux distribution(openwrt) for beaglebone-black and getting the following o/p on my serial port
uncompressing Linux....done,booting the kernel(kernel hangs) in order to trace the error i have enabled CONFIG_DEBUG_LL=y(enables the low level debuging) and CONFIG_DEBUG_INFO=y(enables kernel debug information) but i found CONFIG_DEBUG_LL_UART_NONE=y so if i am correct this means the platform does not support UART debugging(even though kernel is giving the information i might not be seeing that) so is there any other way to debug the kernel or enable the serial port on beagle bone black? if any 1 can guide me through that it would be a great help.....
thanks,
If you have a debugger, then you can certainly debug why your kernel is not booting up.
Assuming, you have one, do the following:
In Kernel configuration, enable CONFIG_DEBUG_INFO (Kernel Hacking ->
Compile-time checks and compiler options)
Compile the kernel
From the debugger software, load the symbols from the elf. The
command varies from one debugger to another
Find symbol "log_buf", and chose option to "Display memory as dump"
There you can see the kernel logs that are not yet printed but stored in the buffer. Hope it helps!
I want to compile a device driver in kernel, and I configure it with *, (not in module ). After the compilation, I can't see the device in /proc/device. Also I check the output of make bzImage, the driver has been compiled. Do I need to delete the two lines:
module_init(mydriver_init);
module_exit(mydriver_exit);
There is not enough information to understand your problem. You should show some code.
Anyway, /proc/devices does not show all kernel module. It shows devices, maybe your driver is not registering a device.
You can put some printk() in your code and read it with the command dmesg from your terminal. If your read your print, your module is loaded.
Evan if you do not compile a module as module, but you built it within the kernel, functions
module_init(mydriver_init);
module_exit(mydriver_exit);
must be there. mydriver_init will be executed when the kernel load your driver, module_exit will be executed when the kernel unload your driver.
The /proc/devices file which is read-only doesn't seem to be editable or viewable using editors like VIM. So try 'cat /proc/devices', your device may show up then.
I want to get address of instruction that causes external program to SIGSEGV. I tried using ptrace for this, but I'm getting EIP from kernel space (probably default signal handler?). How GDB is able to get the correct EIP?
Is there a way to make GDB provide this information using some API?
edit:
I don't have sources of the program, only binary executable. I need automation, so I can't simply use "run", "info registers" in GDB. I want to implement "info registers" in my own mini-debugger :)
You can attach to a process using ptrace. I found an article at Linux Gazette.
It looks like you will want PTRACE_GETREGS for the registers. You will want to look at some example code like strace to see how it manages signal handling and such. It looks to me from reading the documentation that the traced child will stop at every signal and the tracing parent must wait() for the signal from the child then command it to continue using PTRACE_CONT.
Compile your program with -g, run gdb <your_app>, type run and the error will occur. After that use info registers and look in the rip register.
You can use objectdump -D <your_app> to get some more information about the code at that position.
You can enable core dumps with ulimit -c unlimited before running your external program.
Then you can examine the core dump file after a crash using gdb /path/to/program corefile
Because it is binary and not compiled with debugging options you will have to view details at the register and machine code level.
Try making a core dump, then analyse it with gdb. If you meant you wanted to make gdb run all your commands at one touch of a key by 'automate', gdb ca do that too. Type your commands into a file and look into the help user-defined section of manuals, gdb can handle canned commands.