Kernel Panic with ramfs on embedded device: No filesystem could mount root - linux

I'm working on an embedded ARM device running Linux (kernel 3.10), with NAND memory for storage. I'm trying to build a minimal linux which will reside on its own partition and carry out updates of the main firmware.
The kernel uses a very minimal root fs which is stored in a ramfs. However, I can't get it to boot. I get the following error:
[ 0.794113] List of all partitions:
[ 0.797600] 1f00 128 mtdblock0 (driver?)
[ 0.802669] 1f01 1280 mtdblock1 (driver?)
[ 0.807697] 1f02 1280 mtdblock2 (driver?)
[ 0.812735] 1f03 8192 mtdblock3 (driver?)
[ 0.817761] 1f04 8192 mtdblock4 (driver?)
[ 0.822794] 1f05 8192 mtdblock5 (driver?)
[ 0.827820] 1f06 82944 mtdblock6 (driver?)
[ 0.832850] 1f07 82944 mtdblock7 (driver?)
[ 0.837876] 1f08 12288 mtdblock8 (driver?)
[ 0.842906] 1f09 49152 mtdblock9 (driver?)
[ 0.847928] No filesystem could mount root, tried: squashfs
[ 0.853569] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
[ 0.861806] CPU: 0 PID: 1 Comm: swapper Not tainted 3.10.73 #11
[ 0.867732] [<800133ec>] (unwind_backtrace+0x0/0x12c) from [<80011a50>] (show_stack+0x10/0x14)
(...etc)
The root fs is built by the build process, using the following (simplified for clarity):
# [Copy some things to $(ROOTFS_OUT_DIR)/mini_rootfs]
cd $(ROOTFS_OUT_DIR)/mini_rootfs && find . | cpio --quiet -o -H newc > $(ROOTFS_OUT_DIR)/backup.cpio
gzip -f -9 $(ROOTFS_OUT_DIR)/backup.cpio
This creates $(ROOTFS_OUT_DIR)/backup.cpio.gz
The kernel is then built like this:
#$(MAKE) -C $(LINUX_SRC_DIR) O=$(LINUX_OUT_DIR) \
CONFIG_INITRAMFS_SOURCE="$(ROOTFS_OUT_DIR)/backup.cpio.gz" \
CONFIG_INITRAMFS_ROOT_UID=0 CONFIG_INITRAMFS_ROOT_GID=0
I think this means it uses the same config as the main firmware (built elsewhere), but supplies the minimal ramfs image using CONFIG_INITRAMFS_SOURCE.
From Kernel.Org, the ramfs is always built anyway, and CONFIG_INITRAMFS_SOURCE is all that is needed to specify a pre-made root fs to use. There are no build errors to indicate that there is a problem creating the ramfs, and the size of the resulting kernel looks about right. backup.cpio.gz is about 3.6 MB; the final zImage is 6.1 MB; the image is written to a partition which is 8 MB in size.
To use this image, I set some flags used by the (custom) boot loader which tell it to boot from the minimal partition, and also set a different command line for the kernel. Here is the command line used to boot:
console=ttyS0 rootfs=ramfs root=/dev/ram rw rdinit=/linuxrc mem=220M
Note that the nimimal root fs contains "/linuxrc", which is actually a link to /bin/busybox:
lrwxrwxrwx 1 root root 11 Nov 5 2015 linuxrc -> bin/busybox
Why doesn't this boot? Why is it trying "squashfs" filesystem, and is this wrong?

SOLVED! It turned out that a file name used by the (custom) build system had changed as part of an update, and so it was not putting the correct kernel image into the firmware package. I was actually trying to boot the wrong kernel with the "rootfs=ramfs" parameter, one which didn't have a ramfs.
So, for future reference, this error occurs if you specify "rootfs=ramfs" but your kernel wasn't built with any rootfs built in (CONFIG_INITRAMFS_SOURCE=... NOT specified)

Related

Booting kernel from SD in qemu (ARM) with u-boot

I'm quite new to embedded systems and I'm playing around with ARM on qemu. So I've run into problem booting linux kernel image from an emulated SD on versatile express with cpu cortex-a9.
I prepared everything in the following order: first, I've built the kernel with vexpress_defconfig using appropriate ARM toolchain. Then I've built u-boot with vexpress_ca9x4_defconfig. Everything went just fine. The linux kernel source I took is at version 4.13 (latest stable from git). U-boot version is 2017.09-00100. Then I prepared an SD image:
dd if=/dev/zero of=sd.img bs=4096 count=4096
mkfs.vfat sd.img
mount sd.img /mnt/tmp -o loop,rw
cp kernel/arch/arm/boot/zImage /mnt/tmp
umount /mnt/tmp
Next I try to run qemu as follows:
qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -m 128M -dtb kernel/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -kernel uboot/u-boot -sd sd.img -nographic
U-boot loads successfully and gives me command prompt. And SD is really there and attached:
=> mmcinfo
Device: MMC
Manufacturer ID: aa
OEM: 5859
Name: QEMU!
Tran Speed: 25000000
Rd Block Len: 512
SD version 1.0
High Capacity: No
Capacity: 16 MiB
Bus Width: 1-bit
Erase Group Size: 512 Bytes
I attempt to load compressed image from SD into memory and boot it:
fatload mmc 0:0 0x4000000 zImage
and everything looks ok
=> fatload mmc 0:0 0x4000000 zImage
reading zImage
3378968 bytes read in 1004 ms (3.2 MiB/s)
but then I want to boot the kernel and get an error:
=> bootz 0x4000000
Bad Linux ARM zImage magic!
I also tried U-boot images, crafted with u-boot's mkimage, like in this example:
uboot/tools/mkimage -A arm -C none -O linux -T kernel -d kernel/arch/arm/boot/Image -a 0x00010000 -e 0x00010000 uImage
also trying out -C gzip on zImage and different load/entry addresses, to no avail. The images were copied to sd.img. When I fatload the image and check it with iminfo, whichever options I try, I constantly get error:
=> iminfo 0x4000000
## Checking Image at 04000000 ...
Unknown image format!
I'm totally confused and this problem drives me nuts, while information on this subject in Internet is rather scarce. Please, hint me what I'm doing wrong and redirect into right direction.
qemu in use is QEMU emulator version 2.9.0.

Mount USB device on router - OpenWrt

I am new to OpenWrt and Linux/Unix shell. Prior to this, was using dd-wrt for 2 years. Recently switched to OpenWrt and got the basic setup done(wifi/internet working). I need to install transmission my router. But, USB device won't mount.
The Details of the router are as follows:
Router Model:
Asus RT-N13U B1
Image flashed:
chaos_calmer 15.05.1 (openwrt-15.05.1-ramips-rt305x-rt-n13u-squashfs-sysupgrade.bin)
Usb Details:
Crusier Blade 16GB usb 2.0
Formatted to ext4 using GParted bootable cd
Drivers Installed:
kmod-usb-core(preinstalled)
kmod-usb-dwc2(preinstalled)
kmod-usb2
kmod-usb-storage
kmod-fs-ext4
kmod-scsi-core(preinstalled)
Here are permissions for dev/sda, /dev/sda1 and /mnt (set 777 manually):
~#ls -l /dev
brwxrwxrwx 1 root root 8, 0 Jan 17 21:56 sda
brwxrwxrwx 1 root root 8, 1 Jan 17 21:56 sda1
~#ls -l ..
drwxrwxrwx 2 root root 0 Jan 16 21:28 mnt
Error while mounting:
~# mount -t ext4 /dev/sda1 /mnt
mount: mounting /dev/sda1 on /mnt failed: No such file or directory
echo $?
255
sda1 is present inside dev and /mnt directory exists. Still it shows an error.
I have tried installing/reinstalling and then mounting, but for the same result. Also when I disconnect the USB sda and sda1 folders disappear and then reappear when USB is reconnected so I guess the device is being detected successfully.
What is it, that I am missing?
Edit:
Added edits as suggested in the comments.
dmesg output (The output is very big so linked it)
Edit 2:
Switched to LEDE 17. Is more stable and have had zero issues in the past week.
the critical part of the dmesg output is
[ 9.410000] mount_root: loading kmods from internal overlay
[ 9.940000] SCSI subsystem initialized
[ 9.960000] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 9.980000] ehci-platform:EHCI generic platform driver [ 9.990000] usb-storage 1-1:1.0: no of_node; not parsing pinctrl DT
[ 9.990000] usb-storage 1-1:1.0: USB Mass Storage device detected
[ 10.010000] scsi host0: usb-storage 1-1:1.0
[ 10.020000] usbcore: registered new interface driver usb-storage
[ 10.100000] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[ 10.120000] block: extroot: not configured
[ 10.130000] mount_root: switching to jffs2 overlay
[ 10.180000] procd: - early -
[ 11.020000] scsi 0:0:0:0: Direct-Access SanDisk Cruzer Blade 1.27 PQ: 0 ANSI: 6
[ 11.030000] sd 0:0:0:0: no of_node; not parsing pinctrl DT
[ 11.050000] sd 0:0:0:0: [sda] 30529536 512-byte logical blocks: (15.6 GB/14.5 GiB)
[ 11.070000] sd 0:0:0:0: [sda] Write Protect is off
[ 11.070000] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 11.090000] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 11.120000] sda: sda1
[ 11.130000] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 11.550000] EXT4-fs (sda1): Cannot load crc32c driver
oh no, this seems to be a bug
Bug#819725: ext4 missing softdep on crc32c module
https://lists.debian.org/debian-kernel/2016/04/msg00013.html
the following workaround is not applicable because initramfs is a native file system at startup time (https://en.wikipedia.org/wiki/Initramfs) and has no influence when the system is fully up (rootfs is mounted):
Until this is fixed in the kernel package, you can work around it by
either:
Setting base-installer/initramfs-tools/driver-policy to "most" instead of "dep"
Setting base-config/late_command to a script that adds crc32c to /etc/initramfs-tools/modules
post the stack trace anyway, maybe there is another workaround
this is overcomplicated stuff...
here is maybe a solution https://forum.openwrt.org/viewtopic.php?id=69175
download kmod-lib-crc32c and kmod-crypto-crc32c
if this is not working maybe the easiest solution is to format the USB stack as VFAT and to wait for a new kernel...
this is not a permissions error. a permission error would return EPERM -> error code 1 Operation not permitted
it would be interesting to know what exit code the mount returns. The 'exit behavior is very different in the several mount version' i.e. mount(2) and mount(8)
for getting the return value, type in a shell the command
mount /dev/sda1 /mnt
then
echo $?
the number is the returned exit code of the mount (the 255 means 'exit status out of range' in this case '-1', http://www.tldp.org/LDP/abs/html/exitcodes.html#EXITCODESREF)
a listing of the mount(8) exit codes is i.e. in http://www.stackoverflow.com/questions/33167585/what-are-the-return-codes-values-of-linux-umount
http://www.becane.com/2014/09/02/understanding-exit-codes-and-how-to-use-them-in-a-bash-script
beside the return value mount(2) also sets an error code in errno (http://man7.org/linux/man-pages/man3/errno.3.html). printing errno in shell is a bit difficult it is easier to get a reference like http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/Errors/unix_system_errors.html and search for the error string, in this case No such file or directory
the string No such file or directory is system error ENOENT
As a system error ENOENT means A pathname was empty or had a nonexistent component. (http://man7.org/linux/man-pages/man2/mount.2.html)
try sudo mount -t ext4 /dev/sda1 /mnt because the stick is formatted in ext4
if this is not working dmesg output would be interesting
detach usb device, reattach usb device, type dmesg and see the output. in the output of dmesg you also see which drivers are loaded for the device
https://wiki.openwrt.org/doc/howto/usb.storage says that you need several more drivers (block-mount, kmod-scsi-core,...) because a usb stick (USB Mass Storage class) is also a SCSI and a block device...
(linux system error codes are in http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/Errors/unix_system_errors.html)
print a stack trace sudo strace -f mount -t ext4 -o default /dev/sda1 /mnt and post it

Busybox SUID on NFS rootfs

I am building a Linux system from the bottom for a Beagle Bone board. I have compiled the vanilla kernel and built a basic root file system with busybox. The system is booted with U-boot, while the rootfs is located on a Linux PC and exported through NFS:
/path/to/rootfs 10.42.0.17(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
The U-boot bootargs are:
bootargs console=ttyO0,115200n8 root=/dev/nfs rw nfsroot=${serverip}:/path/to/rootfs,v3,tcp ip=dhcp
I've encountered a problem when trying to get su working for non-root users. In order to work around the problem people over internet are suggesting to set the suid bit for the busybox binary.
After doing so:
$ sudo chmod u+s busybox
and verifying:
$ ls -la
...
-rwsr-xr-x 1 myuser myuser 1882976 Jan 13 21:47 busybox
...
$ stat -c "%a %n" busybox
4755 busybox
Something went wrong. The kernel is booting and all of the usual messages are displayed, but it is getting stuck at the end, and no login line is displayed. Here are last few lines of the booting sequence:
[ 3.776185] IP-Config: Complete:
[ 3.779656] device=eth0, hwaddr=c8:a0:30:c5:80:e9, ipaddr=10.42.0.17, mask=255.255.255.0, gw=10.42.0.1
[ 3.789877] host=10.42.0.17, domain=, nis-domain=(none)
[ 3.795822] bootserver=10.42.0.1, rootserver=10.42.0.1, rootpath=
[ 3.802492] nameserver0=10.42.0.1
[ 3.871575] VFS: Mounted root (nfs filesystem) on device 0:15.
[ 3.879903] devtmpfs: mounted
[ 3.883713] Freeing unused kernel memory: 380K (c07ef000 - c084e000)
If removing the flag, the things are returning to normal:
....
[ 3.862291] Freeing unused kernel memory: 380K (c07ef000 - c084e000)
10.42.0.17 login:
If setting the flag from within the running shell on the Beagle Bone board itself, the shell is stopping responding right after the chmod is performed.
I suspect it is something to do with the way the NFS is exporting the rootfs, but it's only a guess, so qualified explanation and possible solution would be helpful.
After some research I will answer my question myself. The answer is very simple. In order the above to work, the busybox binary should be owned by root:root. The simplest solution is just to change the ownership.

cgroup blkio files cannot be written

I'm trying to control I/O bandwidth by using cgroup blkio controller.
Cgroup has been setup and mounted successfully, i.e. calling grep cgroup /proc/mounts
returns:
....
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
...
I then make a new folder in the blkio folder and write to the file blkio.throttle.read_bps_device, as follows:
1. mkdir user1; cd user1
2. echo "8:5 10485760" > blkio.throtlle.read_bps_device
----> echo: write error: Invalid argument
My device major:minor number is correct from using df -h and ls -l /dev/sda5 for the storage device.
And I can still write to file that requires no device major:minor number, such as blkio.weight (but the same error is thrown for blkio.weigth_device)
Any idea why I got that error?
Not sure which flavour/version of Linux you are using, on RHEL 6.x kernels, this was did not work for some reason, however it worked when I compiled on a custom kernel on RHEL and on other Fedora versions without any issues.
To check if supported on your kernel, run lssubsys -am | grep blkio. Check the path if you can file the file blkio.throttle.read_bps_device
However, here is an example of how you can do it persistently, set a cgroups to limit the program not to exceed more than 1 Mibi/s:
Get the MARJOR:MINOR device number from /proc/partitions
`cat /proc/partitions | grep vda`
major minor #blocks name
252 0 12582912 vda --> this is the primary disk (with MAJOR:MINOR -> 8:0)
Now if you want to limit your program to 1mib/s (convert the value to bytes/s) as follows. => 1MiB/s => 1024 kiB/1MiB * 1024 B/s = 1048576 Bytes/sec
Edit /etc/cgconfig.conf and add the following entry
group ioload {
blkio.throttle.read_bps_device = "252:0 1048576"
}
}
Edit /etc/cgrules.conf
*: blkio ioload
Restart the required services
`chkconfig {cgred,cgconfig} on;`
`service {cgred,cgconfig} restart`
Refer: blkio-controller.txt
hope this helps!

How to edit FreeBSD .gz bootfile?

I have virtual image of a FreeBSD system and when I mount it I don't see the /etc/ directory and other files, instead is a big loader.gz on the filesystem, that I believe that is extracted during the boot process. I decompressed the loader.gz with gzip and I got it:
$ file loader
loader: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), not stripped
Using grep I'm able to confirm that the files that I need to edit are inside, however I don't know how to edit it. I tried to mount it without success. How can I modify the contents of loader.gz and use it again?
Can you please give me an example?
I have a Linux system and a Mac to install tools and this FreeBSD image.
Please, help me.
The loader program is generally the last stage of the kernel bootstrapping process.
A recent image should have another signature. e.g. for a memory stick image;
> file tmp/FreeBSD-10.0-RELEASE-amd64-memstick.img
tmp/FreeBSD-10.0-RELEASE-amd64-memstick.img: Unix Fast File system
[v1] (little-endian), last mounted on ,
last written at Fri Jan 17 00:24:02 2014,
clean flag 1, number of blocks 681040, number of data blocks 679047,
number of cylinder groups 13, block size 8192, fragment size 1024,
minimum percentage of free blocks 8, rotational delay 0ms,
disk rotational speed 60rps, TIME optimization
Mounting an image on FreeBSD:
# mdconfig -a -t vnode -f tmp/FreeBSD-10.0-RELEASE-amd64-memstick.img -u 1
# mount /dev/md1a /mnt/root/
(Linux has the same capability, I just don't remember what its called.)
This image contains loader in the boot/ directory:
# ls /mnt/root/
.cshrc ERRATA.TXT README.TXT boot/ lib/ proc/ sys#
.profile HARDWARE.HTM RELNOTES.HTM dev/ libexec/ rescue/ tmp/
COPYRIGHT HARDWARE.TXT RELNOTES.TXT docbook.css media/ root/ usr/
ERRATA.HTM README.HTM bin/ etc/ mnt/ sbin/ var/
# ls /mnt/root/boot/
beastie.4th check-password.4th gptzfsboot menu.4th support.4th
boot color.4th kernel/ menu.rc userboot.so
boot0 defaults/ loader* menusets.4th version.4th
boot0sio delay.4th loader.4th modules/ zfs/
boot1 device.hints loader.help pmbr zfsboot
boot2 firmware/ loader.rc pxeboot zfsloader*
brand.4th frames.4th mbr screen.4th
cdboot gptboot menu-commands.4th shortcuts.4th
On my FreeBSD 10 system, loader has another signature;
/boot/loader: FreeBSD/i386 demand paged executable

Resources