How to debug Linux kernel modules with QEMU? - linux

I am working on academic project that modifies some Kernel Networking code as well as include a new Kernel module.
I am using QEMU to load modified kernel and test.
However, i find that a complete OS is required in some .img to debug.
Is it possible without it ?
Or, which is the distro that can be used with Kernel 2.6 for system. The distro need not have any features, except ability to run programs, including networking support.

The easiest way in my opinion is to use buildroot
http://buildroot.uclibc.org/
clone it, configure it to use your custom kernel (default userspace is fine for a start, you might want to change it later).
it will build your kernel and root filesystem. the entire process takes about half an hour, twenty minutes of which is compiling the monster
my run line looks something:
qemu-system-i386
-hda rootfs.ext2
-kernel bzImage
-m 512M
-append "root=/dev/sda console=ttyS0"
-localtime
-serial stdio
and some more options regarding a tap device

Minimal fully automated QEMU + GDB + Buildroot example
QEMU + GDB on non-module Linux kernel is covered in detail at: How to debug the Linux kernel with GDB and QEMU? and building the kernel modules inside QEMU at: How to add Linux driver as a Buildroot package Get those working first.
Next, I have also fully automated GDB module debugging at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1c29163c3919d4168d5d34852d804fd3eeb3ba67#kernel-module-debugging
These are the main steps you have to take:
Compile the kernel module with debug symbols:
ccflags-y += -g -DDEBUG
as mentioned at: kernel module no debugging symbols found
Stop GDB with Ctrl + C and run:
lx-symbols path/to/parent/of/modules/
This amazing command, which is defined in a GDB Python script inside the Linux kernel source tree, automatically loads symbols for loaded modules present under the given directory recursively whenever GDB stops.
The best way to make that command available is to use:
gdb -ex add-auto-load-safe-path /full/path/to/linux/kernel
as explained at: GDB: lx-symbols undefined command
insmod the kernel module.
This must be done before setting breakpoints, because we don't know where the kernel will insert the module in memory beforehand.
lx-symbols automatically takes care of finding the module location (in host filesystem and guest memory!) for us.
Break GDB again with Ctrl + C, set breakpoints, and enjoy.
If were feeling hardcore, you could also drop lx-symbols entirely, and find the module location after insmod with:
cat /proc/modules
and then add the .ko manually with:
add-symbol-file path/to/mymodule.ko 0xfffffffa00000000

Related

Debugging linux kernel with gdb via QEMU issue

I am newbie in Linux kernel and I'm trying to debug it with gdb via QEMU. My problem is gdb doesn't stop after break start_kernel. More details is below.
My host system is ArchLinux 5.0.10-arch1-1-ARCH x86_64.
Qemu is QEMU emulator version 4.0.0
gdb is GNU gdb (GDB) 8.2.1.
Debugged kernel is linux-4.20.12
I have done following steps:
Compile kernel with
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
CONFIG_GDB_SCRIPTS=y
# CONFIG_DEBUG_INFO_REDUCED is not set
Try to run LFS-8.4 system with it kernel via QEMU
qemu-system-x86_64 lfs-8.4-08052019.raw
System starts good.
Run this system with stop CPU and gdbserver options via QEMU
qemu-system-x86_64 lfs-8.4-08052019.raw -S -s
Start gdb
$ gdb
Load symbols
(gdb) file /mnt/lfs/sources/linux-4.20.12/vmlinux
Reading symbols from /mnt/lfs/sources/linux-4.20.12/vmlinux...done.
Connect to QEMU
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x000000000000fff0 in cpu_hw_events ()
Set break point
(gdb) break start_kernel
Breakpoint 1 at 0xffffffff82761ab5: file init/main.c, line 538.
(I have tried hardware break point hbreak, but result is the same)
Continue
(gdb) c
Continuing.
Further system loads login prompt without any breaks and gdb doesn't show any new messages.
What I do wrong?
UPD: starting QEMU with embed -kernel facility give the same result
qemu-system-x86_64 -kernel /mnt/lfs/sources/linux-4.20.12/arch/x86/boot/bzImage -append 'root=/dev/sda3' -drive file=lfs-8.4-08052019.raw -S -s
UPD2: I have tried to start QEMU without -S key and run target remote localhost:1234 in gdb immediately. QEMU have stopped loading at
Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting kernel.
When I type s in gdb it says
(gdb) s
Cannot find bounds of current function
I suspect that debug symbols in vmlinux isn't related to bzImage. Maybe they was made wrong, but I don't how to make them another way.
UPD3: I have built kernel 2.6 in LFS-8.4 chroot environment. System isn't loaded, but kernel is successfully debugged by method described above! So, I think that it is problem of new kernels. Maybe I should switch off/on something in my 4.20.12 kernel, but I don't know what is exactly. For my purpose (reading Robert Love "Linux Kernel Development") kernel version 2.6 is enough.
You need to pass nokaslr to kernel cmdline.
qemu-system-x86_64 -kernel /mnt/lfs/sources/linux-4.20.12/arch/x86/boot/bzImage -append 'root=/dev/sda3 nokaslr' -drive file=lfs-8.4-08052019.raw -S -s

gdb can't resolve symbols for linux kernel

I have setup Linux Kernel debug environment with VMware Workstation. But When I tried to connect with gdb that connects correctly but I can't set any breakpoint or examine any kernel symbol.
Target Machine (debugee) Ubuntu 18:
I have compiled linux kernel 5.0-0 with the following directives:
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_FS=y
# CONFIG_DEBUG_SECTION_MISMATCH is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
Also my VMX file configuration:
debugStub.listen.guest64 = "TRUE"
debugStub.listen.guest64.remote="TRUE"
After that I transfered vmlinux to debugger machine and use gdb:
bash$ gdb vmlinux
gdb-peda$ target remote 10.251.31.28:8864
Remote debugging using 10.251.31.28:8864
Warning: not running or target is remote
0xffffffff9c623f36 in ?? ()
gdb-peda$ disas sys_open
No symbol "do_sys_open" in current context.
First you need to install kernel-debug-devel, kernel-debuginfo, kernel-debuginfo-common for corresponding kernel version.
Then you can use crash utility to debug kernel, which internally uses gdb
The symbol name you're looking for is sometimes not exactly what you expect it to be. You can use readelf or other similar tools to find the full name of the symbol in the kernel image. These names sometimes differ from the names in the code because of various architecture level differences and their related header and C definitions in kernel code. For example you might be able to disassemble the open() system call by using:
disas __x64_do_sys_open
if you've compiled it for x86-64 architecture.
Also keep in mind that these naming conventions are subject to change in different versions of kernel.

QEMU Showing Black Screen Only

I compiled my custom kernel with different configs and images like u,zImages. But when I try to run qemu with my image, qemu showing only black screen.
Also I looked at this post but it didn't help me.
EDIT
I just tried to compile kernel with these commands.
make ARCH=arm distclean
make ARCH=arm integrator_defconfig
make ARCH=arm menuconfig
NOTE: I used default menuconfig.
make ARCH=arm CROSS_COMPILE=arm-none-eabi- zImage
qemu-system-arm -M integratorcp -kernel arch/arm/boot/zImage
And last i tried this command;
qemu-system-arm -M integratorcp -kernel arch/arm/boot/zImage -append 'console=ttyAMA0 earlyprintk=ttyAMA0' -serial stdio
NOTE: when i try to use -dtb, qemu giving to me "Unable to copy device tree in memory." error.
"QEMU does nothing with a black screen" almost always means "QEMU is running fine, but the guest code crashed or stopped early in the boot process without sending any output". Almost certainly either your kernel is misconfigured, or your QEMU command line is wrong. You don't give enough information to say which. You need to tell us at least:
what the kernel is you're running and what machine you've configured it for
what the QEMU command line you're using is
Given your updated question with a command line, some suggestions:
tell your guest to use the serial port (use the QEMU option -append 'console=ttyAMA0' to set the guest kernel command line)
either check the serial output view in the GUI, or send it to stdout with -nographic or -serial stdio
enable any earlyprintk or earlycon options in the guest config that you can and on the guest command line, so if the guest fails early you have more chance to catch it
pass the device tree for the kernel with -dtb integratorcp.dtb (use the one from your kernel tree; you'll probably have to tell the kernel makefiles to build it for you)
PS: integratorcp is an absolutely ancient development board -- why do you want to use it?

Compiling a kernel - no bzImage/vmlinuz produced

I'm trying to compile a kernel (altered version of 2.6.32.9, found here https://github.com/rabeeh/linux-2.6.32.9). I am doing the compilation on a emulated ARM system (qemu) (yes, I should probably cross-compile, but that's a different topic) running Ubuntu Core (https://wiki.ubuntu.com/Core) and the kernel (vmlinuz) from Ubuntu 11.04 (downloaded from http://ports.ubuntu.com/ubuntu-ports/dists/natty/main/installer-armel/current/images/versatile/netboot/vmlinuz).
After running make bzImage, I look in the arch/arm/boot folder, and find only a file called zImage. I tried using this zImage instead of the vmlinuz I downloaded from ubuntu.com in qemu, but that doesn't work, just shows a black screen. I guess zImage is not the same as bzImage, which is what I think vmlinuz (judging from different articles on the internet) is.
So, a few questions:
Why doesn't make bzImage produce a bzImage/vmlinuz?
Can I convert a vmlinux to a vmlinuz using for example mkimage (there are lots of guides on the opposite...)?
Thanks
The bzImage filename and make target was originally x86-specific (big zImage). Many of the bootloaders on architectures that are not equal to baremetal-x86 (SPARC, PPC, IA64, etc. and also Xen on *) directly take vmlinux (or one of its compressed forms, for example vmlinux.gz, aka zImage). I guess some maintainers just added bzImage as a make target name because they wanted to have the x86 madness on their arch as well.
I get the result you describe by asking qemu to emulate a different cpu than arm926ej-s. But booting versatilepb with the default cpu works. I've cross-compiled my kernel, and I compiled all the drivers into it (so I don't use initrd).
Just download 100MB arm-eabi toolchain from http://www.mentor.com/embedded-software/sourcery-tools/sourcery-codebench/editions/lite-edition/ (it's free but they want your email, like the x86 Intel compiler). It has an installer, just say "next" until it's done, like on Windows. Then add the bin directory to your path:
export PATH=~/CodeSourcery/Sourcery_CodeBench_Lite_for_ARM_EABI/bin/:$PATH
Then go back to your kernel source dir and do
make ARCH=arm CROSS_COMPILE=arm-none-eabi- menuconfig
make ARCH=arm CROSS_COMPILE=arm-none-eabi- zImage modules
You can do
sudo make ARCH=arm CROSS_COMPILE=arm-none-eabi- INSTALL_MOD_PATH=path_to_arm_root modules_install
if you can reach your ARM filesystem from the host. If you're using NFS root it's trivial, but if you're using a disk image you need to either:
use a raw disk image and kpartx (depends on your host kernel having dm-multipath) or
qemu-nbd which supports qcow (and depends on host kernel having network block device support)
To boot in qemu with disk you need the right drivers (SYM53C8XX SCSI). The versatile defconfig doesn't select those.

How to debug my Cross compiled Linux Kernel?

I 've cross compiled a Linux Kernel (for ARM on i686 - using Cross-LFS).
Now I'm trying to boot this Kernel using QEMU.
$ qemu-system-arm -m 128 -kernel /mnt/clfs-dec4/boot/clfskernel-2.6.38.2 --nographic -M versatilepb
Then, it shows this line and waits for infinite time !!
Uncompressing Linux... done, booting the kernel.
So, I want to debug the kernel, so that I can study what exactly is happening.
I'm new to these kernel builds, Can someone please help me to debug my custom built kernel as it is not even showing anything after that statement. Is there any possibility of the kernel being broken? ( I dont think so, b'se it didnot give any error while compiling )
And my aim is to generate a custom build very minimal Linux OS. Any suggestions regarding any tool-chains etc which would be easy & flexible depending on my requirements like drivers etc.,
ThankYou
You can use GDB to debug your kernel with QEMU you can use -s -S options. If you want a simple and reliable toolchain, you can use ELDK from DENX (http://www.denx.de/wiki/DULG/ELDK).
You can install it like this (It's not the last version, but you got the idea):
wget http://ftp.denx.de/pub/eldk/4.2/arm-linux-x86/iso/arm-2008-11-24.iso
sudo mkdir -p /mnt/cdrom (if necessary)
sudo mount -o loop arm-2008-11-24.iso /mnt/cdrom
/mnt/cdrom/install -d $HOME/EMBEDDED_TOOLS/ELDK/
The command above should install the toolchain under $HOLE/EMBEDDED_TOOLS/ELDK (modify it if you need)
echo "export PATH=$PATH:$HOME/EMBEDDED_TOOLS/ELDK/ELDK42/usr/bin" >> $HOME/.bashrc
You can then see the version of your ARM toolchain like this:
arm-linux-gcc -v
You can test a hello_world.c program like this:
arm-linux-gcc hello_world.c -o hello_world
And you type: file hello_wrold to see the target architecture of the binary, it should be something like this:
hello_wrold: ELF 32-bit LSB executable, ARM, version 1 (SYSV)
Now if you want to compile a production kernel, you need to optimize it (i suggest using busybox) and if you want just one for testing now, try this steps:
Create a script to set your chain tool set_toolchain.sh:
#! /usr/bin/sh
PATH=$PATH:$HOME/EMBEDDED_TOOLS/ELDK/ELDK42/usr/bin
ARCH=arm
CROSS_COMPILE=arm-linux-gnueabi-
export PATH ARCH CROSS_COMPILE
And run your script (source ./set_toolchain.sh)
Download a linux kernel and unzip it (Let's assume 2.6.x, it's an old kernel, but there are a lot of chances that it work without compilation errors).
Inside your unzipped kernel:
cd ~/linux-2.6.29/arch/arm/configs
make versatile_defconfig
Here we use versatile chip, you may need to use make menuconfig to modify the option OABI and set it to ARM EABI, this option is under Kernel features menu
After all this steps, you can compile you kernel:
make
if you want verbose compilation make v=1
After this you got your kernel under arch/arm/boot/zImage.
Hope this help.
Regards.
I would suggest to build your kernel by activating the option in the section Kernel hacking of your configuration file.
Then you may use kdb or kgdb which is easier to use but requires another machine running gdb.
`
You can also connect Qemu and GDB. Qemu has the -s and -S options that run a GDB server and allow you to connect to it via TCP to localhost:1234. Then you can load your kernel image (the unzipped one) in GDB and see how far your kernel boots.

Resources