(SquashFS is a compressed filesystem - http://en.wikipedia.org/wiki/SquashFS)
I'm looking for a way to read a SquashFS filesystem from a program. So far, I've know about the in-kernel drivers for it, but I'm sure that a userspace library for it must exist somewhere. Any language would be fine, but C is preferred.
Just mounting the filesystem and using it that way is technically possible, but I'd rather avoid that route because the application I'm looking at would involve working with at least a few dozen archives at any given time.
There is a tool called unsquashfs which extracts the squashfs image akin to tar. It should be bundled with mksquashfs
I just downloaded the code tarball from squashfs.sourceforce.net and there is no kernel code in there. Only userland code in C for mksquashfs and unsquashfs.
You could probably extract code from unsquashfs.
mksquashfs and unsquashfs are packaged as "squashfs-tools" on my Red Hat system:
Name : squashfs-tools
Group : System Environment/Base
Size : 160923
URL : http://squashfs.sf.net
Summary : squashfs utilities
Description :
Squashfs is a highly compressed read-only filesystem for Linux. This package
contains the utilities for manipulating squashfs filesystems.
/sbin/mksquashfs
/usr/sbin/unsquashfs
/usr/share/doc/squashfs-tools-3.0
/usr/share/doc/squashfs-tools-3.0/ACKNOWLEDGEMENTS
/usr/share/doc/squashfs-tools-3.0/CHANGES
/usr/share/doc/squashfs-tools-3.0/COPYING
/usr/share/doc/squashfs-tools-3.0/PERFORMANCE.README
/usr/share/doc/squashfs-tools-3.0/README
Be warned that squashing and unsquashing are painfully slow. It takes several minutes for a script I wrote to unsquash, modify, and re-squash an 87M stage2.img file.
Related
I am attempting to get initramfs working on an embedded linux device using buildroot and uboot. I have been following the guides have have initramfs set up in the buildroot and kernel config. However as far as I can tell it is not being built into the kernel, although a CPIO or the root FS is being included in the output.
My questions is twofold. First, from a high level standpoint, is my understanding of the boot process correct?
Stage 1 bootloader (Atmel/ARM specific) starts and hands off
Stage 2 bootloader (uboot) initializes a few things, loads the kernel into memory and runs it
The kernel places the CPIO archive into memory, makes that root and runs init in that enviornment
This means the image I flash onto the chip is an IMG with a boot and system partition, the system partition is ext4 and contains the CPIO archive within. I am not sure how to make item 3 actually happen from here.
Second, in terms of actual configurations, the Buildroot menuconfig has a number of options for filesystems
ext2/3/4 root filesystem (on by default)
cpio the root filesystem (for use as an initial RAM filesystem)
initial RAM filesystem linked into linux kernel
Then in the kernel config
General -> Initial RAM filesystem and RAM disk (initramfs/initrd) support
General -> Initramfs source files (set to the generated cpio file)
What is buildroot generating here? From my end it looks like I am getting duplicate images (CPIO and ext4) and the CPIO is being ignored.
For people who may have the same question, the best way forward is to become more familiar with uboot and it's options. In particular the role uboot-env.txt plays in the process.
My understanding of buildroot was wrong. The process is
Building kernel
Creating all root filesystems (ext and CPIO)
Rebuilding the kernel with the CPIO inside
Packaging the ext file as dictated by my genimage.cfg file
This may be of interest if the process is confusing to you.
Here is what the system was doing
Stage 1 bootloader (Atmel/ARM specific) starts and hands off
Stage 2 bootloader (uboot) initializes a few things, then behaves as dictated by as config, by writing the kernel to memory from the ext4 partition and running it
This kernel did not have the cpio so it did not run.
Better understanding the process and components I was able to restructure my img file and use the bootloader to load the kernel that was built with the cpio.
After running BitBake on several different recipe files, BitBake generates a file of type '.hddimg'. I haven't been able to find a clear explanation on what this file is used for, the closest that I have found is some speculation on the mailing list here. The author Paul states that:
the image isn't an image of a regular bootable system drive, but is a "live
image" of a smaller system that can either boot the real system from a
virtualized file system in RAM whose image is read from a single file in the
first level, or it can install the real system to a different drive.
The 'bootimg.bbclass' is what generates the .hddimg, and in the opening comments it is written that:
A .hddimg file [is] an msdos filesystem containing syslinux, a kernel, an initrd and a rootfs image. These can be written to harddisks directly and also booted on USB flash disks (write them there with dd).
Which appears to corroborate with what Paul wrote, but still leaves a lot of ambiguity on how to go about booting from this file (at least to a greenhorn like me).
Well, the doc says "write them there with dd". So:
dd if=/path/to/your/hddimg of=/path/to/raw/usb/device
so, if you have the file as my.hddimg and the usb flash disk appears as /dev/sdg
dd if=/home/karobar/my.hddimg of=/dev/sdg
As it's name implies, it's an image, so needs to be written as such. The actual file system is inside of the rootfs file, which is similarly an image!
Once you have that on the usb stick, the usb stick itself should be bootable. Depending on what you're trying to do this may not be the easiest kind of output from bitbake to work with.
ubifs has no fsck program, so how do you check the filesystem integrity when using ubifs?
My target system is ARM, Linux 3.2.58.
From what I have found so far on UBIFS' web page:
integrity - UBIFS (as well as UBI) checksums everything it writes to
the flash media to guarantee data integrity, UBIFS does not leave data
or meta-data corruptions unnoticed (JFFS2 is doing the same); By
default, UBIFS checks only meta-data CRC when it reads from the media,
but not the data CRC; however, you may force CRC checking for data
using one of the UBIFS mount options - see here.
If you need to check filesystem for corruption
If your UBIFS filesystem is mounted with chk_data_crc option, then simple cat $FILES > /dev/null should be sufficient. If not, you can only detect and recover metadata corruption. The corruption of file body will pass unnoticed.
I used something like find / -type f -print -exec cat {} + > /dev/null
If you need to recover broken files
Again from the overview section:
to make it more clear, imaging you have wiped out the FAT table on
your FAT file system; for FAT FS this would be fatal; but if you
similarly wipe out UBIFS index, you still may re-construct it,
although a special user-space tool would be required to do this (this
utility is not implemented at the moment, though)
While theoretically possible, you're on your own.
Back up the flash contents, arm yourself with UBIFS data structures (maybe sources) and a hex editor, and good luck.
edit: As I figured, Linux's MTD driver already applies ECC (error correction code) to MTD devices.
I believe, the criterion for a data loss if there are more than /sys/class/mtd/mtd*/ecc_strength errors per /sys/class/mtd/mtd*/ecc_step_size flash block. mtd_read() (it's a MTD API one level below UBIFS) will return EUCLEAN in this case. Don't know if there exists a tool that uses it to check for errors.
The "bit flipped" warnings we get do not mean that there was a data loss yet. You can write to /sys/class/mtd/mtd*/bitflip_threshold to control the amount of warnings you get.
You can just read all files, which essentially causes ubifs to check them. Cfr. the advise given on the mailing list. The ubifs implementation will recover if it is possible. There is however no guarantee that this will capture all corruptions.
In theory, ubifs should never be corrupted, but in practice bugs in ubifs or NAND drivers can still cause corruptions.
On an embedded platform running Linux 2.6.36, I occasionally run into a problem where files do not appear in the root file system that ARE present in our initramfs cpio file.
I am building the initramfs from a cpio listing file (see gen_init_cpio.c), but also ran into the problem before when just using a full directory.
When I say I know the files are present in the cpio file, I mean if I extract usr/initrmafs_data.cpio.gz the files are there.
It seems to be loosely related to the amount of content in the initramfs, but I haven't found the magic number of files and/or total storage size that causes files to start disappearing.
Is there an option in make menuconfig I'm missing that would fix this? A boot argument? Something else?
Any suggestions?
Update: To clarify, this is with a built-in ramdisk using CONFIG_INITRAMFS_SOURCE and it's compressed with gzip via setting CONFIG_INITRAMFS_COMPRESSION_GZIP. Also, this is for a mipsel-linux platform.
Update #2: I've added a printk to init/initramfs.c:clean_path and mysteriously, the previously "disappearing" files are now all there. I think this sorta seems to point to a kernel bug if attempting to log the behavior altered the behavior. I'll compare initramfs.c against a newer kernel tomorrow to see if that sheds any light on the matter.
Probably your image size is bigger than default ramdisk size (4MB afaik). Check if adding ramdisk_size=valuebiggerthanyourimagesize as a kernel parameter (before root=... parameter) solves your problem. You can also try to change kernel config value CONFIG_BLK_DEV_RAM_SIZE.
I'm tasked with writing a reader program for windows that is able read an ext2 partition.
For my testing I'm using a drive I formatted to ext2 and a file I created using mkfs (a file that does mount and work well under linux)
For some reason when I read the superblock from the drive (the real one) I get all the right meta-data (i.e. block size, inode count etc..) but doing the exact same thing to the file returns bad results (which make no sense).
is there a difference between the 2?
I open the drive using \.\X:
and I make the file using mkfs.
There shouldn't be any difference between ext2 on a partition and stored within a file (and indeed there isn't; I just checked); however, IIRC, the offset of the primary superblock is 2048 instead of 1024, if ext2 is installed on a bare disk (e.g. /dev/sda instead of /dev/sda1). This is to accomodate the MBR and other junk. (I can't find it in the docs from skimming just now, but this sticks out in my mind as something I ran into.) However, it's somewhat unusual to install to a bare drive, so I doubt this is your problem.
I wrote some ext2 utilities a few years ago, and after starting writing it by hand, I switched to using Ted Ts'o (the ext2 filesystem creator)'s e2fsprogs, which come with headers/libraries/etc. for doing all this in a more flexible and reliable fashion.
You may also want to check at offset 0x438 into the file/partition for the magic number 0xEF53, and consider it not an ext2/3 filesystem if that's not there, before pulling in the entire superblock, just as a sanity check.
Here's some docs that will probably be helpful: http://www.nongnu.org/ext2-doc/ext2.html