tl;dr I want to use Linux "Dynamic Debug" but the path /sys/kernel/debug/dynamic_debug/ is not found.
I want to use the "Dynamic Debug" feature of Linux to enable debug messages for a particular kernel module (wireguard). To enable a kernel module kernel debug messages requires writing instructions to file /sys/kernel/debug/dynamic_debug/control,
echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control
But there is no directory /sys/kernel/debug/dynamic_debug/ and thus no file /sys/kernel/debug/dynamic_debug/control.
The debugfs is mounted to /sys/kernel/debug/ (as is typical) and it has other files (so it's not like debugfs is mounted unexpectedly). Nor does alternative path /proc/dynamic_debug/ exist.
Running Linux kernel 5.15.
Enabling "Dynamic Debug" requires setting Linux compilation option CONFIG_DYNAMIC_DEBUG. In my case, I'm using Raspbian, and that project decided not to enable CONFIG_DYNAMIC_DEBUG during Linux builds.
I could "roll my own" (compile Raspbian with CONFIG_DYNAMIC_DEBUG) but that's a lot of work and likely will upset the apt package management.
🙁
Related
I made my own linux distribution with buildroot. In "make menuconfig" and "make linux-menuconfig" I checked all options related to "ppp" and "pppd". Unfortunately, after building my distribution I can't use the commands "pon", "poff", "plog" and probably others. The system cannot see them. I looked and they are not in "/ usr / bin" or "/ usr / sbin". What could be causing this? I found out somewhere that these are debian-only commands, but how can I enable the pppd daemon?
ppd, pppd, pon, and etc are userland applications you need to install them using buildroot. So far you have enabled support in the kernel for ppp but you have not installed the actual application that manages the ppp connections which is pppd
If build root doesn't already have a package for ppp/pppd, you can make a recipe to do that. The official website is: https://ppp.samba.org/
pon, poff, and plog can be found in the scripts directory of the sources code.
Debugging the Linux Kernel with kgdb over rs-232 needs several preparation steps. I found awesome documentation, but no single-source that is fully self-contained and summarizes all steps needed, does not explain for ages, and has been tested. And also covers Yocto.
Is there any source that covers all that is needed in one single and short description ?
I.e.:
What files are needed in the directory GDB is started from (e.g. kernel awareness, source, vmlinux) and how to get theese, where to put it ?
When and where to get a cross-gdb from ?
ALL kernel config options needed, also the not-obvious ones (like CONFIG_RANDOMIZE_BASE)
How to configure the serial ports
Explaining a working back and forth of breaking into debugee and debugger to get started.
Explaining one rock-solid option of stopping the kernel that runs everywhere.
Explaining how to get this done not only for PC-PC debugging, but also for Yocto targets.
Debugging the Linux Kernel via a Nullmodem-Cable:
It took me a while to get a kgdb connection with Linux kernel awareness fully running. I share my way of doing this with Ubuntu Eoan (optional: Yocto Warrior) in 2020 here:
Tested with:
Debugging a linux based Intel PC from an Intel MacBook running MacOS Catalina. Using the gdb from the Homebrew package "i386-elf-gdb“. (wituout „-tui“ option in GDB)
Debugging a linux based ARM target (i.mx6, Yocto) from a linux based Intel PC.
Prerequisites:
You need two computers and a serial nullmodem cable. Check the cable by firiing up a serial termianl (e.g. screen or putty) on both hosts, connecting to your serial port (e.g. /dev/ttyS0 or /dev/ttyUSB0) and print characters from each station to the other. Remember the /dev/tty ports you confirmed.
Preparation:
You need on the first debuggee computer, we call it „target":
Special kernel installed that contains symbols, kgdb support etc.
Learn how to compile and install a kernel and use in make menuconfig belows configuration. You can search for Sybmbols with F8 or the / key in menuconfig.
(E.g. wiki.ubuntu.com. There take care in the first paragraph to execute deb-src before apt-get :)
# CONFIG_SERIAL_KGDB_NMI is not set
CONFIG_CONSOLE_POLL=y
# CONFIG_DEBUG_INFO is not set
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
# CONFIG_KGDB_KDB is not set
CONFIG_FRAME_POINTER=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_GDB_SCRIPTS=y
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_RANDOMIZE_BASE is not set
(Note for advanced Yocto use, skip if you're debugging a PC:
In yocto I created in my layer a file: recipes-kernel/linux/linux-mainline_%.bbappend with the content:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://kgdb.cfg“
And in files/kgdb.cfg I added the config fragment shown above (without the on ARM unavailable options CONFIG_RANDOMIZE_BASE and CONFIG_FRAME_POINTER)
)
You need on the second debugger computer, we call it „debugger pc":
Full kernel source code, same code you used to compile the kernel above. (If you compiled the .o and .ko objects in place and not in a build-folder you better not copy the directory from the other pc, where you called make etc. in, but then better grab fresh sources again.)
vmlinux file containing the symbols (lies in the kernel source root, or build folder on the highest level after kernel make).
vmlinux-gdb.py file that was made during the kernel build (also lies at the same position on the highest level.).
All scripts in the folder scripts/gdb (Folder scripts in the same toplevel-position. If you use a dedicated build folder use the script folder from there, not from the source folder.)
(Advanced: If both computers don’t match in CPU, like Intel and Arm, a cross-gdb build. Ignore if you're on Intel/AMD.)
Note for advanced Yocto use, I did something like (ignore if you debug a PC):
bitbake -c patch virtual/kernel #(apply the changed kernel config from above)
bitbake -f -c compile virtual/kernel #(unpack is not sufficient because of vmlinux-gdb.py)
mkdir ~/gdbenv
cp -a tmp/work-shared/phyboard-mira-imx6-14/kernel-source/. ~/gdbenv
cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux ~/gdbenv
cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux-gdb.py ~/gdbenv
mkdir ~/gdbenv/scripts
cp -r tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/scripts/gdb ~/gdbenv/scripts
Then (ignore if you're on a PC)
yocto bitbake -c populate_sdk [my-image]
Then (still ignore on PC) install the sdk .sh-installation file from your deploy directory on the debugger pc and start the environment as guided by the output of the install script (remember that command), then use "$GDB" for starting the cross-gdb instead of „gdb".
Debug execution
Launch on the debugger two console screens:
Console 1, ssh: +++++++++++++++++++++++++++++++++++++++
ssh user#192.168.x.y
sudo -s
echo ttyS0,9600n8 > /sys/module/kgdboc/parameters/kgdboc
echo 1 > /proc/sys/kernel/sysrq
echo g > /proc/sysrq-trigger
Console 2, local: ++++++++++++++++++++++++++++++++++++++++
cd ~/gdbenv
gdb -tui ./vmlinux
add-auto-load-safe-path ~/gdbenv
source ~/gdbenv/vmlinux-gdb.py
set serial baud 9600
target remote /dev/ttyS0 (use the tty port you confirmed in the beginning)
b [name of the c funtion you want to debug]
cont
Back to console 1, ssh: +++++++++++++++++++++++++++++++++++++++
[Now trigger the function, e.g. sudo modprobe yourFancyKernelModule]
Back to console 2, local: ++++++++++++++++++++++++++++++++++++++++
Now use gdb functions, like bt, step, next, finish ...
You can also use linux-aware commands. Call "apropos lx“ in gdb for a list of commands.
I have an out-of-tree Linux kernel module that I need to compile. When I execute "make" in the kernel module directory I am getting:
"fatal error: stdarg.h: No such file or directory"
Before starting the build I installed the header file based on my Linux distribution.
$sudo apt-get install kernel-headers-$(uname -r)
How can I solve this compilation error? (my distribution is Ubuntu 16.04 with linux-headers-4.15.0-42-generic)
I ran a search of stdarg.h with the "locate" command to see if I can sport the file on the system.
I got:
/usr/include/c++/5/tr1/stdarg.h
/usr/lib/gcc/i686-linux-gnu/5/include/cross-stdarg.h
/usr/lib/gcc/i686-linux-gnu/5/include/stdarg.h
...
It tells me there is at least one stdarg.h provided by the compiler.
I tried to include the path "/usr/lib/gcc/i686-linux-gnu/5/include" in the kernel module Makefile so stdarg.h can be picked up. It did not work (while building, another reference to stdarg.h in the official kernel header was not being resolved).
I finally created a symlink directly under:
/usr/src/linux-headers-4.15.0-42-generic/include
$sudo ln -s /usr/lib/gcc/i686-linux-gnu/5/include/stdarg.h stdarg.h
This was just enough to solve the compilation issue.
I am wondering if the kernel headers should come with an implementation of stdarg.h by default (that is the first time I encounter this issue). I have also read that the compiler provide one implementation and most of the time it is better to use the compiler version.
Updated note: if the above solution still does not solve the problem:
Before running make again, do a make clean. Do a ls -la in the folder and look for a ".cache.mk" file. If this is still there, remove it and run "make" again. It should solve the problem.
I had the same issue with CentOS 9, and the other answers didn't work for me. Apparently the problem is that in more recent kernels, it shouldn't be <stdarg.h> but <linux/stdarg.h>. With virtualbox guest additions 6.1.34, it correctly checks for kernel with a version of 5.15.0 or more. But my kernel is the 5.14.xx, meaning the include for stdarg.h is wrong.
Solving the issue
Dependencies
Install all the dependencies for the guest edition
gcc make perl kernel-devel kernel-headers bzip2 dkms
Installation
Run the Guest Addition installation like you would normally. It will fail by saying it is unable to compile the kernel modules. That's expected. It will copy all the file we need to the VM disk.
Editing
We now need to edit the erroneous files.
/opt/VBoxGuestAdditions-6.1.34/src/vboxguest-6.1.34/vboxguest/include/iprt/stdarg.h
/opt/VBoxGuestAdditions-6.1.34/src/vboxguest-6.1.34/vboxsf/include/iprt/stdarg.h
On line 48 (may change for different versions), it check for a version of Linux and select the correct header depending on the version. We need to replace if RTLNX_VER_MIN(5,15,0) with if RTLNX_VER_MIN(5,14,0) in both files.
Compile the kernel modules
We can now compile the kernel modules, and the error should be gone.
sudo rcvboxadd quicksetup all
I personally got an error the first time, but then I recompiled without changing anything and it worked.
Remember that it's just a workaround, it may not work with different versions.
If you using Arch Linux with zen-kernel:
sudo CPATH=/usr/src/linux-zen/include/linux vmware-modconfig --console --install-all
I had the same problem with VirtualBox 6.1.0 running archlinux with kernel 6.1.9.
I downloaded VirtualBoxGuestAdditions_7.2.0.iso file from https://download.virtualbox.org/virtualbox/7.0.2/ link(you may select more appropriate to your VirtualBox version) and assigned as an optical drive to virtualbox machine. After start of the system running blkid command on terminal showed the name of CD rom device which was /dev/sr0. then I created iso folder on
/mnt folder
mkdir /mnt/iso
and mounted cd drive to that folder
mount -o loop /dev/sr0 /mnt/iso
after I cd'ed to /mnt/iso
cd /mnt/iso
and manually run VirtualBoxGuestAdditions.run script
sh ./VirtualBoxGuestAdditions.run
which successfully compiled and istalled required virtualbox guest modules.
Now everytime I update kernel version I redo the same procedure. And it work fine.
It also remove old 6.1.0 guest additons folder.
I compiled my Linux kernel according to the Linux Device Driver Chapter 4: Debugging Techniques. After I loaded my first hello world module and then checked the output by dmesg, however, all I can see is evbug: ........
I know I can turn off evbug's output by execute sudo rmmod evbug. But, obviously, it is inconvenient to execute this command after each reboot.
How could I disable this module's autoloading? I just want to load it manually when I need it.
You need to blacklist the module. For debian systems see https://wiki.debian.org/KernelModuleBlacklisting. For redhat systems see https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/rescuemode_drivers-blacklisting.html
I have started reading the book Essential Linux Device Drivers. I am following the process for compiling and booting into a new kernel. However I am unable to boot into the newly compiled kernel - I select the new kernel in grub and then ..nothing...the screen just stays the same.
Here are the steps I am taking, as described in the book (I am using kernel 2.6.24 as that is what the book is based on) -
cd /usr/src/linux-2.6.24 (the base dir of the kernel I downloaded)
make clean
cp arch/x86/configs/i386_defconfig .config
make bzImage cp arch/x86/boot/bzImage /boot/vmlinuz
The book says that 'you might need to alert your bootloader about the arrival of the new boot image. If you are using the GRUB bootloader, it figures this out automatically'.
Well it didn't figure it out automatically in my case as there was no option to select this kernel in GRUB. So I did it manually by typing update-grub in the terminal. And this didn't work either. However once I changed the filename from vmlinuz to vmlinuz-2.6.24 and type update-grub it picked it up. So on page 11 of the book there seems to already be multiple things left out...
The book says to then reboot the machine.
Anyway, I am now able to select this kernel in GRUB but as I said above it doeesn't boot properly, the screen just goes blank and never changes. So am I missing something? I have followed the instructions in the book exactly.
Most automatic grub setups include kernel options to hide the messages generated as the kernel attempts to boot. For example:
/boot/vmlinuz-x.x.x.x-generic root=UUID=something ro quiet splash
While on the grub menu line of the kernel you wish to boot, press 'e' to temporarily edit it, and delete options such as quiet and splash (but leave the ro).
Then boot the temporarily modified line, and see the progress messages in order to get an idea where the boot is failing.
The default configuration probably requires an initrd (e.g. because the file system and root device needs their driver, from modules, which is what initrd provides).
You could either configure your kernel appropriately (carefully enabling as in-kernel, not in-modules, the essential drivers) for your particular hardware, or use a procedure to build a kernel package with its initrd.
You should find a procedure appropriate for your linux distribution and habits, e.g. something like this or that. I am using on Debian:
time fakeroot make-kpkg -j2 --initrd --revision=3.4.2 --append-to-version=-amd64 binary