how to debug kernel loading and intrd load in virtualbox - linux

I have cloned the linux kernel repo on my arch hosted machine (host is ubuntu 16.04). Two weeks ago I was able to boot into the new kernel (it was 4.11.rc06 back then), then I did git pull and recompiled everything and it just hangs after "loading initial ramdisk image...".
So I tried git clean -xfd then make localmoduleconfig answering defaults for everything, then make then make modules_install then mkinitcpio -p linux.4.11.custom and of course sudo cp -v arch/x86_64/boot/bzImage /boot/vmlinuz-linux.4.11.custom.
After I verified it does indeed hang I tried more git pulls, more cleans, but nothing changed.
Running the same kernel from the same source on a real machine boots.
I could not find recorded bug in virtualbox or find an update for ubuntu.
Next I tried debugging it myself by adding to the grub's linux command: debug earlyprintk=vga,keep and even removing the initrd line adding noinitrd to the kernel, but I get no error. Just a screen with the grub's "echo" messages that stays like that forever.
How can I debug it?
Has anyone got any idea what can be done?

To check whether the kernel even starts I would use KDB (kernel's built-in debugger), and see if you get a prompt at startup.
For better debugging I would try to get KGDB (GDB for kernel) working.
You can actually activate both to have all options available. See following link for more information about this:
https://www.kernel.org/doc/htmldocs/kgdb/index.html

Related

Cannot get custom kernel to boot - mkinitpcio does not add any modules

1. What I am trying to achieve:
Build a custom kernel so I can install and run Anbox-git from AUR on my Arch laptop. Custom kernel is needed for the package to work.
2. What I did to achieve it:
Download Arch Linux kernel v5.8.5-arch1 from here
I followed the guidelines on tradional compilation Arch wiki to create the custom kernel
Via make nconfig I applied the changes mentioned in the Anbox Arch wiki page.
Via make nconfig I changed EFIVAR_FS option from "M" to "*" to resolve an error from earlier attemps.
Via make nconfig under Location: -> Device Drivers-> Multiple devices driver support (RAID and LVM) (MD [=y])-> Device mapper support (BLK_DEV_DM [=y]) I added a few more options (*) because on earlier builds mkinitpcio gave errors for missing modules for DM_CRYPT, and some more DM_ modules which I cannot easily reproduce (will do if necessary for the answer, but I hope it'll be irrelevant).
After creating the config this way I did:
sudo make modules_install
sudo cp -v arch/x86_64/boot/bzImage /boot/vmlinuz-linux58ac
sudo cp /etc/mkinitcpio.d/linux.preset /etc/mkinitcpio.d/linux58ac.preset
Adapted the preset file per Arch wiki instructions
sudo mkinitcpio -p linux58ac
Important: The mkinitpcio runs fine, but keeps giving me a warning:
WARNING: No modules were added to the image. This is probably not what
you want.
sudo grub-mkconfig -o /boot/grub/grub.cfg
3. Expected result:
I am able to reboot, select the new kernel from grub menu, get the usual LVM password prompt, and launch into it without problems.
4. Result I get:
I can reboot and select new kernel from grub but when I select it I get a
Warning: /lib/modules/5.8.5-arch1/modules.devname not found, ignoring.
Starting version 246.4-1-arch
ERROR device 'dev/mapper/vg0-root' not found. Skipping fsck.
mount /new_root: special device /dev/mapper/vg0-root does not exist.
You are being dropped into an emergy shell.
I checked and the /lib/modules/5.8.5-arch1/modules.devnamedoes indeed exist. But I think the actual problem is that mkinitcpio doesn't load the correct modules into the custom kernel, causing it to become unbootable.
Any help appreciated!

How can I solve stdarg.h No such file or directory while compiling out-of-tree Linux kernel module?

I have an out-of-tree Linux kernel module that I need to compile. When I execute "make" in the kernel module directory I am getting:
"fatal error: stdarg.h: No such file or directory"
Before starting the build I installed the header file based on my Linux distribution.
$sudo apt-get install kernel-headers-$(uname -r)
How can I solve this compilation error? (my distribution is Ubuntu 16.04 with linux-headers-4.15.0-42-generic)
I ran a search of stdarg.h with the "locate" command to see if I can sport the file on the system.
I got:
/usr/include/c++/5/tr1/stdarg.h
/usr/lib/gcc/i686-linux-gnu/5/include/cross-stdarg.h
/usr/lib/gcc/i686-linux-gnu/5/include/stdarg.h
...
It tells me there is at least one stdarg.h provided by the compiler.
I tried to include the path "/usr/lib/gcc/i686-linux-gnu/5/include" in the kernel module Makefile so stdarg.h can be picked up. It did not work (while building, another reference to stdarg.h in the official kernel header was not being resolved).
I finally created a symlink directly under:
/usr/src/linux-headers-4.15.0-42-generic/include
$sudo ln -s /usr/lib/gcc/i686-linux-gnu/5/include/stdarg.h stdarg.h
This was just enough to solve the compilation issue.
I am wondering if the kernel headers should come with an implementation of stdarg.h by default (that is the first time I encounter this issue). I have also read that the compiler provide one implementation and most of the time it is better to use the compiler version.
Updated note: if the above solution still does not solve the problem:
Before running make again, do a make clean. Do a ls -la in the folder and look for a ".cache.mk" file. If this is still there, remove it and run "make" again. It should solve the problem.
I had the same issue with CentOS 9, and the other answers didn't work for me. Apparently the problem is that in more recent kernels, it shouldn't be <stdarg.h> but <linux/stdarg.h>. With virtualbox guest additions 6.1.34, it correctly checks for kernel with a version of 5.15.0 or more. But my kernel is the 5.14.xx, meaning the include for stdarg.h is wrong.
Solving the issue
Dependencies
Install all the dependencies for the guest edition
gcc make perl kernel-devel kernel-headers bzip2 dkms
Installation
Run the Guest Addition installation like you would normally. It will fail by saying it is unable to compile the kernel modules. That's expected. It will copy all the file we need to the VM disk.
Editing
We now need to edit the erroneous files.
/opt/VBoxGuestAdditions-6.1.34/src/vboxguest-6.1.34/vboxguest/include/iprt/stdarg.h
/opt/VBoxGuestAdditions-6.1.34/src/vboxguest-6.1.34/vboxsf/include/iprt/stdarg.h
On line 48 (may change for different versions), it check for a version of Linux and select the correct header depending on the version. We need to replace if RTLNX_VER_MIN(5,15,0) with if RTLNX_VER_MIN(5,14,0) in both files.
Compile the kernel modules
We can now compile the kernel modules, and the error should be gone.
sudo rcvboxadd quicksetup all
I personally got an error the first time, but then I recompiled without changing anything and it worked.
Remember that it's just a workaround, it may not work with different versions.
If you using Arch Linux with zen-kernel:
sudo CPATH=/usr/src/linux-zen/include/linux vmware-modconfig --console --install-all
I had the same problem with VirtualBox 6.1.0 running archlinux with kernel 6.1.9.
I downloaded VirtualBoxGuestAdditions_7.2.0.iso file from https://download.virtualbox.org/virtualbox/7.0.2/ link(you may select more appropriate to your VirtualBox version) and assigned as an optical drive to virtualbox machine. After start of the system running blkid command on terminal showed the name of CD rom device which was /dev/sr0. then I created iso folder on
/mnt folder
mkdir /mnt/iso
and mounted cd drive to that folder
mount -o loop /dev/sr0 /mnt/iso
after I cd'ed to /mnt/iso
cd /mnt/iso
and manually run VirtualBoxGuestAdditions.run script
sh ./VirtualBoxGuestAdditions.run
which successfully compiled and istalled required virtualbox guest modules.
Now everytime I update kernel version I redo the same procedure. And it work fine.
It also remove old 6.1.0 guest additons folder.

Bcache directory not showing up after make-bcache

I'm setting up bcache on an ubuntu virtual machine on azure. I'm following the instructions from http://blog.rralcala.com/2014/08/using-bcache-in-ec2.html.
After running make-bcache -B /dev/sdc1, the /dev/bcache0 device is not yet available.
When running make-bcache -B /dev/sdc1 a second time, /dev/bcache suddenly exists, as does /sys/fs/bcache and /sys/fs/block/bcache0 etc.
A weird side-note is that running make-bcache -B for the second time in a script (even with sleeps in between) does not fix the issue, but running it manually does.
Once bcache has initiated properly it stays stable, also after reboots and VM-relocations. You can read the configuration scripts at https://github.com/okke-formsma/azure-bcache.
Does anyone have a clue how to enable bcache on my ubuntu 16.04 machine without having to resort to a double manual make-bcache -B xxx?
[edit] According to the archlinux wiki not having a /sys/fs/bcache means that "The kernel you booted is not bcache enabled", however that is not the case.
I figured it out!
The module was not loaded by the kernel. Doing a sudo modprobe bcache made all bcache functionality available without weird workarounds.

Yocto menuconfig not working

For some reason the menuconfig menu does not come up when I try launching it from my Yocto installation. I am using the Toradex Yocto 1.6 system as is described here http://developer.toradex.com/software-resources/arm-family/linux/board-support-package/openembedded-%28core%29, with my board set to "apalis-t30". When I run either bitbake virtual/kernal -c menuconfig or bitbake linux-toradex -c menuconfig, it executes fine but finishes (without erros) before actually showing anything. Running devshell also gives the same results.
If I just use the kernel sources on their own as is described here http://developer.toradex.com/software-resources/arm-family/linux/board-support-package/build-u-boot-and-linux-kernel-from-source-code, I can get menuconfig open using make nconfig. From the Yocto scripts it appears as if though the exact same kernel sources are being used. If I try adding adding make nconfig to the do_configure_prepend script in the linux-toradex_git.bb file then the commands get stuck stating that the process (I assume menuconfig) is running and then provides a PID for it, but no window or menu is displayed anywhere and the task does not seem to finish.
PS. I am on Fedora 21 64-bit.
EDIT:
I have now checked the default Yocto image and menuconfig comes up fine there. I am assuming that the Toradex BSP is not entirely compatible enough with Yocto for this to work out of the box. I have spoken to Toradex and they have told me that I should instead fork their kernel, modify it the normal way in my own repo and then tell the script to pull from my modified repo. I guess this could work but its a bit of a hassle and I would like to fix their Yocto system. I am assuming that this cannot be to hard as running make nconfig is usually enough, I just can't figure out how to get that command working with bitbake.
This should work fine with the meta-toradex layer. In the local.conf file, comment out the INHERIT += "rm_work" line:
#INHERIT += "rm_work"
Then do a full build of the kernel:
MACHINE=apalis-t30 bitbake virtual/kernel
Then try menuconfig now that all the sources are in place:
MACHINE=apalis-t30 bitbake -c menuconfig virtual/kernel
If you are using Ubuntu, try to reconfigure system shell to bash instead of dash(that is default for Ubuntu):
$ sudo dpkg-reconfigure dash
press "No" when prompted.
Actually I got the same problem few times. In one case shell reconfigure helped me.

Starting a program in a chroot environment returns immediately

I am working in a virtual environment, trying to start open vm tools in a chroot environment.
I tested with bash and it seems to work fine.
I used ./configure --options --prefix=/home/chroot_env to install the program, then using ldd on vmtoolsd, i copied the corresponding libraries to the /lib directory.
Now when I start chroot /home/chroot_env /bin/vmtoolsd, nothing happens, the chroot returns directly. Launching the same binary in the normal environment does work.
Does someone have an idea why it isn't working, the correct libraries are there, and it works with bash.
EDIT : strace showed that vmtoolsd is trying to access /dev/console, I added mount --bind /dev/ /home/chroot_env/dev/ but it is still failing.
EDIT2 : another strace showed it was looking for another plugin loaded dynamically, i added it and it worked, conclusion strace is great for debugging such issue!
When you run a program and nothing happens, you can always run it with strace in order to see which syscalls are made. This is an easy way to obtain the list of the files (regular or not) that are opened. In your case, check that your program doesn't try to access a file that is not in the chroot.

Resources