i am trying to emulate cavium octeon's mips64 linux kernel on Qemu.I am currently having some issues with use mode init code and want to debug init.i am starting the Qemu using -s -S option in the command line and running the gdb using command
ddd --debugger /OCTEON-SDK/tools/bin/mips64-octeon-linux-gnu-gdb /OCTEON-SDK/linux/kernel_2.6/linux/vmlinux
and then attaching the gdb with command
target remote localhost:1234
The gdb is currently showing only the instructions running in kernel space.
What i want to ask is there any method by which i can debug the usermode init and libraries instructions step by step as in case of kernel space?
For example if printf is issued from init then i want to see that which instructions are being executed in the libraries and how the control is returned to kernel ?
I have managed to step debug BusyBox's /sbin/init using the procedure described at: Is it possible to use gdb and qemu to debug linux user space programs and kernel space simultaneously?
The only additional things you have to keep in mind are:
/sbin/init is just a symlink to busybox, so you must use /bin/busybox as the object file
the "main" function for /sbin/init is actually init_main, following BusyBox' convention of calling the main for each pseudo-executable as <exec>_main
It is a bit flaky, but mostly just works.
Related
Debugging the Linux Kernel with kgdb over rs-232 needs several preparation steps. I found awesome documentation, but no single-source that is fully self-contained and summarizes all steps needed, does not explain for ages, and has been tested. And also covers Yocto.
Is there any source that covers all that is needed in one single and short description ?
I.e.:
What files are needed in the directory GDB is started from (e.g. kernel awareness, source, vmlinux) and how to get theese, where to put it ?
When and where to get a cross-gdb from ?
ALL kernel config options needed, also the not-obvious ones (like CONFIG_RANDOMIZE_BASE)
How to configure the serial ports
Explaining a working back and forth of breaking into debugee and debugger to get started.
Explaining one rock-solid option of stopping the kernel that runs everywhere.
Explaining how to get this done not only for PC-PC debugging, but also for Yocto targets.
Debugging the Linux Kernel via a Nullmodem-Cable:
It took me a while to get a kgdb connection with Linux kernel awareness fully running. I share my way of doing this with Ubuntu Eoan (optional: Yocto Warrior) in 2020 here:
Tested with:
Debugging a linux based Intel PC from an Intel MacBook running MacOS Catalina. Using the gdb from the Homebrew package "i386-elf-gdb“. (wituout „-tui“ option in GDB)
Debugging a linux based ARM target (i.mx6, Yocto) from a linux based Intel PC.
Prerequisites:
You need two computers and a serial nullmodem cable. Check the cable by firiing up a serial termianl (e.g. screen or putty) on both hosts, connecting to your serial port (e.g. /dev/ttyS0 or /dev/ttyUSB0) and print characters from each station to the other. Remember the /dev/tty ports you confirmed.
Preparation:
You need on the first debuggee computer, we call it „target":
Special kernel installed that contains symbols, kgdb support etc.
Learn how to compile and install a kernel and use in make menuconfig belows configuration. You can search for Sybmbols with F8 or the / key in menuconfig.
(E.g. wiki.ubuntu.com. There take care in the first paragraph to execute deb-src before apt-get :)
# CONFIG_SERIAL_KGDB_NMI is not set
CONFIG_CONSOLE_POLL=y
# CONFIG_DEBUG_INFO is not set
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
# CONFIG_KGDB_KDB is not set
CONFIG_FRAME_POINTER=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_GDB_SCRIPTS=y
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_RANDOMIZE_BASE is not set
(Note for advanced Yocto use, skip if you're debugging a PC:
In yocto I created in my layer a file: recipes-kernel/linux/linux-mainline_%.bbappend with the content:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://kgdb.cfg“
And in files/kgdb.cfg I added the config fragment shown above (without the on ARM unavailable options CONFIG_RANDOMIZE_BASE and CONFIG_FRAME_POINTER)
)
You need on the second debugger computer, we call it „debugger pc":
Full kernel source code, same code you used to compile the kernel above. (If you compiled the .o and .ko objects in place and not in a build-folder you better not copy the directory from the other pc, where you called make etc. in, but then better grab fresh sources again.)
vmlinux file containing the symbols (lies in the kernel source root, or build folder on the highest level after kernel make).
vmlinux-gdb.py file that was made during the kernel build (also lies at the same position on the highest level.).
All scripts in the folder scripts/gdb (Folder scripts in the same toplevel-position. If you use a dedicated build folder use the script folder from there, not from the source folder.)
(Advanced: If both computers don’t match in CPU, like Intel and Arm, a cross-gdb build. Ignore if you're on Intel/AMD.)
Note for advanced Yocto use, I did something like (ignore if you debug a PC):
bitbake -c patch virtual/kernel #(apply the changed kernel config from above)
bitbake -f -c compile virtual/kernel #(unpack is not sufficient because of vmlinux-gdb.py)
mkdir ~/gdbenv
cp -a tmp/work-shared/phyboard-mira-imx6-14/kernel-source/. ~/gdbenv
cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux ~/gdbenv
cp tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/vmlinux-gdb.py ~/gdbenv
mkdir ~/gdbenv/scripts
cp -r tmp/work/phyboard_mira_imx6_14-phytec-linux-gnueabi/linux-mainline/4.19.100-phy1-r0.0/build/scripts/gdb ~/gdbenv/scripts
Then (ignore if you're on a PC)
yocto bitbake -c populate_sdk [my-image]
Then (still ignore on PC) install the sdk .sh-installation file from your deploy directory on the debugger pc and start the environment as guided by the output of the install script (remember that command), then use "$GDB" for starting the cross-gdb instead of „gdb".
Debug execution
Launch on the debugger two console screens:
Console 1, ssh: +++++++++++++++++++++++++++++++++++++++
ssh user#192.168.x.y
sudo -s
echo ttyS0,9600n8 > /sys/module/kgdboc/parameters/kgdboc
echo 1 > /proc/sys/kernel/sysrq
echo g > /proc/sysrq-trigger
Console 2, local: ++++++++++++++++++++++++++++++++++++++++
cd ~/gdbenv
gdb -tui ./vmlinux
add-auto-load-safe-path ~/gdbenv
source ~/gdbenv/vmlinux-gdb.py
set serial baud 9600
target remote /dev/ttyS0 (use the tty port you confirmed in the beginning)
b [name of the c funtion you want to debug]
cont
Back to console 1, ssh: +++++++++++++++++++++++++++++++++++++++
[Now trigger the function, e.g. sudo modprobe yourFancyKernelModule]
Back to console 2, local: ++++++++++++++++++++++++++++++++++++++++
Now use gdb functions, like bt, step, next, finish ...
You can also use linux-aware commands. Call "apropos lx“ in gdb for a list of commands.
I'm running a customized linux distribution built with reference to ubuntu 12.04 which has init as user space startup process by default. But now I want to switch over to systemd init system which is available from ubuntu 15.04. I installed the systemd components & libraries and also written unit files to replace init.d service daemon scripts.
I want to know how the system can start the systemd as init process?
The kernel starts the init system located in /sbin/init (see also this wikipedia article)
This process is responsible for starting every other process and should not die or exit (this will result in a panic).
One can also temporarily change the init daemon by using the init boot param on the kernel command line: init=/path/to/init_program.
Depending on your bootloader you should be able to edit the kernel commandline and test your changes.
I compiled my Linux kernel according to the Linux Device Driver Chapter 4: Debugging Techniques. After I loaded my first hello world module and then checked the output by dmesg, however, all I can see is evbug: ........
I know I can turn off evbug's output by execute sudo rmmod evbug. But, obviously, it is inconvenient to execute this command after each reboot.
How could I disable this module's autoloading? I just want to load it manually when I need it.
You need to blacklist the module. For debian systems see https://wiki.debian.org/KernelModuleBlacklisting. For redhat systems see https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/rescuemode_drivers-blacklisting.html
I want to make some modifications in the kvm module in my Linux kernel. For this, I did the following:
Obtained the Kernel sources:
apt-get source linux-image-source-$(uname-r)
Modified a function in the file emulate.c - changed some variables and added a printk statement.
Built the kernel:
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary generic
Installed the packages produced as a result of building:
sudo dpkg -i linux*(version)*.deb
Rebooted the system.
Executed QEMU with kvm enabled.
However the changes I made, didn't seem to reflect when I try to test them in QEMU. Neither did the printk statement execute when I checked with dmesg.
Can anyone please point out which part I am getting wrong?
Installing a new kernel wont necessarily make it boot into it, you may need to change the default in your boot loader. (e.g. grub)
You can check whether the kernel you just compiled and installed is the same as what you booted with using:
cat /proc/version
If this is not as expected, then you need to tweak your Grub config and/or reboot and manually choose the correct kernel.
Having established the correct kernel, you may need to change the level of which messages are reported by the kernel (via dmesg)
This is controlled by a kernel proc file. You can see what the current values are by printing this file:
cat /proc/sys/kernel/printk
Example output:
4 4 1 7
The first argument - messages with severity < 4 (i.e. 0, 1 2 or 3) will be recorded.
The second argument - messages with no specified severity default to 4 and thus not seen by the system in the above example.
So the following will change the log so that all kernel messages are seen:
echo 8 > /proc/sys/kernel/printk
See (for example) http://www.makelinux.net/books/lkd2/ch18lev1sec3 for further information.
I am debugging a program that makes use of libnetfilter_queue. The documentation states that a userspace queue-handling application needs the CAP_NET_ADMIN capability to function. I have done this using the setcap utility as follows:
$ sudo setcap cap_net_raw,cap_net_admin=eip ./a.out
I have verified that the capabilities are applied correctly as a) the program works and b) getcap returns the following output:
$ getcap ./a.out
./a.out = cap_net_admin,cap_net_raw+eip
However, when I attempt to debug this program using gdb (e.g. $ gdb ./a.out) from the command line, it fails on account of not having the correct permissions set. The debugging functionality of gdb works perfectly otherwise and debugs as per normal.
I have even attempted to apply these capabilities to the gdb binary itself to no avail. I did this as it seemed (as documented by the manpages that the "i" flag might allowed the debugee to inherit the capability from the debugger.
Is there something trivial I am missing or can this really not be done?
I run into same problem and at beginning I thought the same as above that maybe gdb is ignoring the executable's capability due to security reason. However, reading source code and even using eclipse debugging gdb itself when it is debugging my ext2fs-prog which opens /dev/sda1, I realize that:
gdb is no special as any other program. (Just like it is in the matrix, even the agents themselves they obey the same physical law, gravity etc, except that they are all door-keepers.)
gdb is not the parent process of debugged executable, instead it is grand father.
The true parent process of debugged executable is "shell", i.e. /bin/bash in my case.
So, the solution is very simple, apart from adding cap_net_admin,cap_net_raw+eip to gdb, you have also apply this to your shell. i.e. setcap cap_net_admin,cap_net_raw+eip /bin/bash
The reason that you have also to do this to gdb is because gdb is parent process of /bin/bash before create debugged process.
The true executable command line inside gdb is like following:
/bin/bash exec /my/executable/program/path
And this is parameter to vfork inside gdb.
For those who have the same problem, you can bypass this one by executing gdb with sudo.
A while ago I did run into the same problem. My guess is that running the debugged program with the additional capabilities is a security issue.
Your program has more privileges than the user that runs it. With a debugger a user can manipulate the execution of the program. So if the program runs under the debugger with the extra privileges then the user could use these privileges for other purposes than for which the program intended to use them. This would be a serious security hole, because the user does not have the privileges in the first place.
For those running GDB through an IDE, sudo-ing GDB (as in #Stéphane J.'s answer) may not be possible. In this case, you can run:
sudo gdbserver localhost:12345 /path/to/application
and then attach your IDE's GDB instance to that (local) GDBServer.
In the case of Eclipse CDT, this means making a new 'C/C++ Remote Application' debug configuration, then under the Debugger > Connection tab, entering TCP / localhost / 12345 (or whatever port you chose above). This lets you debug within Eclipse, whilst your application has privileged access.
I used #NickHuang's solution until, with one of system updates, it broke systemd services (too much capabilities on bash for systemd to start it or some such). Switched to leaving bash alone and instead pass a command to gdb to invoke the executable directly. The command is
set startup-with-shell off
OK, so I struggled a bit with this so I thought I'd combine answers and summarise.
The easy solution is just to sudo gdb as suggested but just be a bit careful. What you're doing here is running the debugged program as root. This may well cause it to operate differently than when you run it from the command line as a normal user. Could be a bit confusing. Not that I would EVER fall into this trap... Oopsies.
This will be fine if you're running the debugged program as root with sudo OR if the debugged program has the setuid bit set. But if the debugged program is running with POSIX capabilities (setcap / getcap) then you need to mirror these more granular permissions in bash and gdb as Nick Huang suggested rather than just brute forcing permissions with 'sudo'.
Doing anything else may lead you to a bad place of extreme learning.