I'm trying to analyze and understand an embedded system that runs linux.
I'm pretty new to linux boot process, embedded linux etc, so I have to learn a lot but prefer learning by doing. So analyzing a system is the way I try to understand all this.
I already got some insight and learnt a lot in just looking at the bootmsg, checking some scripts and provided files (firmware) and trying to understand what's going on by just searching for answers here and elsewhere in the net. This might also be the reason for some wrong expressions I might use, I hope you can understand anyways. Atm I am not able to ask the creator of this system directly, so I hope you might help me out with some answers here.
So the System got a SOC (Marvell Armada-370 88F6707-A1) with some Flash (128 MiB - Hynix NAND (HY27U1G8F2BTR)) and DRAM (512 MB) and seems to load the bootload U-Boot from SPI flash (1MB).
As mentioned above as U-Boot is used as bootloader which then loads some Linux version 3.2.34. I think I already understand quite a bit of the general boot process here, but got some questions depending the provided bootargs.
Following is an excerpt of the printenv command (U-Boot environment)
image_name=uImage
mtdids=nand0=nand0
mtdparts=mtdparts=nand0:8m(kernel),6m(Initrd),-(rootfs)
select0=nand info
load0=nand read.e 0x02000000 0 360000
loadr0=nand read.e 0x04000000 800000 300000
boot=bootm 0x02000000 0x4000000
bootcmd=run beep select0 load0 loadr0 boot || run beep beep beep errled
bootargs=console=ttyS0,115200 root=ubi0:rootfs ubi.mtd=3,2048 initrd=0x12000000 rootfstype=ubifs
beep=beep
I see that they load the kernel and compressed ramdisk content into ram load0=nand read.e 0x02000000 0 360000 and loadr0=nand read.e 0x04000000 800000 300000. The kernel at start looks at 0x04000000 for the ramdisk_image and uncompresses to use the content for initial rootfs. Then the usual process with linuxrc / init begins... at the end the normal file system from nand (here rootfs partition of ubi device 0) is loaded as stated in kernel command line (root=ubi0:rootfs ubi.mtd=3,2048 rootfstype=ubifs). This is what I understood is happening here and might be pretty straight forward.
What I'm wondering now is the following bootargs part initrd=0x12000000. I don't quite get why they provide a different address here when we already let the kernel know about the real of the compressed ramdisk (as a second parameter of bootm).
I started the system with and without this parameter and it seems there are no differences.
As in my understanding of reading about that initrd= argument is that it seems to do the same as the second parameter that is already passed to bootm, it tells the kernel where to find an initrd to load the initial rootfs.
Could anyone explain why we here pass two different locations to an initrd to the kernel? Is this just some obsolete stuff that was forgotten to remove or is there a reason for?
Thanks in advance for your help.
I hope I haven't missed anything and the question wasn't already answered here somewhere, but I couldn't find anything.
edit-2:
Thanks to #sawdust in the comments I understood better what actually is going on here and saw the importance to add some more information to the question so an answer will be clearer to understand.
Here is a short excerpt of the bootmsg
> bootm 0x02000000 0x4000000
## Booting kernel from Legacy Image at 02000000 ...
Image Name: Linux-3.2.34
Created: 2013-08-15 4:18:54 UTC
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 3464064 Bytes = 3.3 MB
Load Address: 00008000
Entry Point: 00008000
Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 04000000 ...
Image Name:
Created: 2013-09-10 10:38:37 UTC
Image Type: ARM Linux RAMDisk Image (gzip compressed)
Data Size: 3030739 Bytes = 2.9 MB
Load Address: 12000000
Entry Point: 12000000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
I also always missed some information in this part of bootm that might have already gave a hint on what's happening here. And might have in correlation to the presence of same address should have lead to the path of answer.
edit-1: Added some more information about the hardware
edit-2: Added bootmsg for more information
Related
Background
I am bringing up a Linux kernel via Yocto for some vendor-provided embedded hardware. I have configured the image to boot via fitImage with an initramfs and no rootfs (there is persistent storage but this is entirely for userspace application use). Think PXE live image and you won't be far off.
Things have been going well until my initramfs image crossed the ~128MB mark. Below this and everything boots as expected and all drivers are bound without issue. Above this mark and the kernel still boots but many drivers, though not all, are not bound. This is quite perplexing as all drivers are statically built into the kernel (no modules are used on this platform). Unfortunately, one of these modules runs the platform watchdog which causes entirely predictable reboots.
Thus far I have verified that all of the symbols are present in the vmlinux image:
$ objdump -x vmlinux | grep mtk_wdt
0000000000000000 l df *ABS* 0000000000000000 mtk_wdt.c
ffffff800880ac40 l F .text 000000000000004c mtk_wdt_stop
ffffff800880ac90 l F .text 0000000000000040 mtk_wdt_shutdown
ffffff80091de778 l F .init.text 0000000000000020 mtk_wdt_driver_init
ffffff800880acd0 l F .text 000000000000004c mtk_wdt_ping
ffffff800880ad20 l F .text 0000000000000070 mtk_wdt_set_timeout
ffffff800880ad90 l F .text 0000000000000074 mtk_wdt_start
ffffff800880ae08 l F .text 0000000000000144 mtk_wdt_resume
ffffff800880af50 l F .text 0000000000000120 mtk_wdt_suspend
ffffff800880b070 l F .text 0000000000000080 mtk_wdt_remove
ffffff80088977a8 l F .text 0000000000000210 mtk_wdt_isr
ffffff80091fe0a0 l F .exit.text 000000000000001c mtk_wdt_driver_exit
ffffff800880b4f0 l F .text 0000000000000310 mtk_wdt_probe
ffffff8008c2acd8 l O .rodata 0000000000000028 mtk_wdt_info
ffffff8008c2ad00 l O .rodata 0000000000000050 mtk_wdt_ops
ffffff8008c2ad98 l O .rodata 00000000000000b8 mtk_wdt_pm_ops
ffffff8008c2ae50 l O .rodata 0000000000000190 mtk_wdt_dt_ids
ffffff80093a3cb8 l O .data 00000000000000b0 mtk_wdt_driver
ffffff800a199368 l O .bss 0000000000000008 mtk_wdt1
ffffff8009285598 l O .init.data 0000000000000008 __initcall_mtk_wdt_driver_init6
Additionally, I have sha256 checksums for the binary kernel (eg. linux.bin), initramfs and device tree in the before fitImage assembly, after fitImage assembly and after unpacking into system memory (via bootloader); all match. Near as I can tell, what gets built is what gets unpacked and booted.
Furthermore, I have enabled initcall_debug and, while I see other __initcall()s, the non-bound drivers are, unsurprisingly, missing.
I know the devices are present in the device tree and correctly configured. After boot, I get about 5 seconds of console access before the watchdog kicks; just enought time to get a command or two off. On a "working" images (initramfs < ~128MB) and on failing images (initramfs > ~128MB) the contents of /sys/bus/platform/devices are identical and I can see (among others), my watchdog:
$ ls -lha /sys/bus/platform/devices
...
lrwxrwxrwx 1 root root 0 Jan 1 00:00 10007000.watchdog -> ../../../devices/platform/10007000.watchdog
Performing the same test but comparing /sys/bus/platform/devices shows the drivers which weren't __initcall()ed as missing.
Some collective other things I have checked:
Device Tree. The same DTB is used in both the working and broken images. I have also verified the device tree in-memory as mentioned above. Device tree overlays are not used.
Real memory load offsets. Everything is where is should be and there is plenty of space in each region. I can move the kernel around in memory an the issue persists regardless of location.
Bad memory. This failure happens identically across multiple units.
Bad compression. The issue manifests regardless of kernel / initramfs compression. Currently I am testing with everything uncompressed to minimize breakage points.
Bad signing. I have disabled signature verification (applied to the fitImage partition image after all else); no dice there either.
Attempting to bundle the initramfs directly into the kernel. No change. Right now I have the initramfs as built into the fitImage but otherwise loaded and verified independently.
Bad kernel command line arguments. I am using root=/dev/ram initrd=0x48000000,384M and have traced this all the way into init/initramfs.c where unpacking is done. I was able to verify the offsets passed are, indeed, in the correct virtual memory space and sum to 384M.
Updating the kernel linker script per this forum post. I am able to see a .initramfs section generated in vmlinux via objdump but yet the issue persists.
Given all of the above, the only thing I don't know how to verify is the jump from vmlinux to linux.bin. This is done in Yocto via objcopy as follows:
[ -n "${vmlinux_path}" ] && ${OBJCOPY} -O binary -R .note -R .comment -S "${vmlinux_path}" linux.bin
Questions
How can I verify a given symbol is included in the final linux.bin?
What mechanisms would affect inclusion or exclusion of a given symbol at build time?
Which pieces of the kernel build and runtime are affected by initramfs size?
Are there any other tools / techniques / tribal wisdom which can help debug this situation?
EDIT 1:
Below is the basic memory map of where everything lands and space utilization. As mentioned above and in the comments, I can relocate the kernel, DTB and initramfs to (almost) arbitrary locations but the issue still persists.
0x40000000 - 0x40001000 = Bootloader arg area (Fixed usage)
0x40080000 - 0x41EDFFFF = Kernel (~12MB / 29.5MB used)
0x41E00000 - 0x42FF5FFF = Trampoline (96 bytes / ~6MB used)
0x42FF6000 - 0x42FFFFFF = ATF BL3-1 (Fixed usage)
0x43000000 - 0x43FFFFFF = Trusted OS (~476K / 16M used)
0x44000000 - 0x44FFFFFF = DTB (~77.3K / 16M used)
0x45000000 - 0x47FFFFFF = Trusted OS memory (dynamic)
0x48000000 - 0x5FFFFFFF = Initramfs (~129MB / 384MB used)
0x60000000 - MEM END = Free
So, like most kernel issues the real problem was not where I thought it was. As it turns out, the problem was caused by one of the other drivers earlier in the init list hanging the core, preventing the watchdog driver from being registered. How this is affected by the initramfs is beyond me and is its own question.
For anyone who comes across this in the future, the answers to my specific questions above are listed below:
How can I verify a given symbol is included in the final linux.bin?
I was not able to figure out how to do this statically. That said, I was able to print the addresses of the init functions at runtime by adding printk()s to do_initcall_level in init/main.c. The addresses printed can then be compared to the output of objdump on vmlinux (see my question for the incantation).
A really useful and in-depth description of the initcall process can be found here.
Note that you can also turn on initcall_debug, which will print each function name. In my case I wanted raw addresses which is why I chose the printk() method.
What mechanisms would affect inclusion or exclusion of a given symbol at build time?
Most of this boils down to your .config. The vast majority of inclusion / exclusion is done via the preprocessor. Other useful items are the linker script common header at include/asm-generic/vmlinux.lds.h and the platform linker script at for your device arch/<arch>/*/*.lds.
Which pieces of the kernel build and runtime are affected by initramfs size?
No idea on this one, still.
Are there any other tools / techniques / tribal wisdom which can help debug this situation?
Don't panic
I am trying to run linux on an Arduino Yun board. The Arduino board contains an Atheros AR9331 chipset
On U-Boot These are the steps I am doing:
1- Download the kernel :
ar7240> tftp 0x80060000 openwrt-ar71xx-generic-uImage-lzma.bin;
Load address: 0x80060000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
######################
done
Bytes transferred = 1441863 (160047 hex)
2- Erase Flash in order to copy the kernel:
ar7240> erase 0x9fEa0000 +0x160047
Error: end address (0xa0000046) not in flash!
Bad address format
This is the problem It seems that 0x9fEa0000 +0x160047 exceeds the total size of the flash.
So my questions are:
1- How can I figure the total amount of memory reserved for the flash in Uboot (From wich address it starts and ends), I am thinking about changing 0x9fEa0000 by a fewer address but i'am afraid i can harm other things
This is the output of the help:
ar7240> help
? - alias for 'help'
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootm - boot application image from memory
cp - memory copy
erase - erase FLASH memory
help - print online help
md - memory display
mm - memory modify (auto-incrementing)
mtest - simple RAM test
mw - memory write (fill)
nm - memory modify (constant address)
ping - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
progmac - Set ethernet MAC addresses
reset - Perform RESET of the CPU
run - run commands in an environment variable
setenv - set environment variables
tftpboot- boot image via network using TFTP protocol
version - print monitor version
2- Is there someone experienced with Atheros AR9331 chipset who can help me find the Flash mapping (From where it starts and ends) from the datasheet
You can determine the flash layout from the kernel boot command line. Either run the printenv command in u-boot or boot into the existing kernel and look through the boot log. You need to find something like the following:
(There are plenty of guides on the internet, I took this one from https://finninday.net/wiki/index.php/Arduino_yun, your board may or may not be the same).
linino> printenv
bootargs=console=ttyATH0,115200 board=linino-yun mem=64M rootfstype=squashfs,jffs2 noinitrd mtdparts=spi0.0:256k(u-boot)ro,64k(u-boot-env)ro,14656k(rootfs),1280k(kernel),64k(nvram),64k(art),15936k#0x50000(firmware)
bootcmd=bootm 0x9fea0000
This means there are the following partitions:
u-boot 0 to 256K (0x0 - 0x40000)
u-boot-env 256k to 320k (0x40000 - 0x50000)
rootfs (squashfs) 320k to 14976k (0x50000 - 0xea0000)
kernel 14976k to 16256k (0xea0000 - 0xfe0000)
nvram 16256k to 16320k (0xfe0000 - 0xff0000)
art 16320k to 16384k (0xff0000 - 0x1000000)
The rootfs partition is 14M, which is much larger than the rootfs image file (less than 8MB) so in theory you can move the kernel image at a lower address. For this you will need to modify the kernel boot line in the u-boot environment block (rootfs aand kernel partition sizes) and the bootcmd parameter so the u-boot know where the new kernel is located.
Flash is mapped to 0x9f000000 so the value in the bootcmd should be 0x9f000000 + the offset of the kernel in bytes.
What I am not sure about is if there is an overlay filesystem for any persistent changes to the flash. Can you boot into the existing system and post the output of df -h and cat /proc/mounts?
I wanted to work through this independent study http://security.cs.rpi.edu/~candej2/syllabus.pdf myself to pick up some basics on exploit writing and actually being able to write an exploit from scratch. So... I was looking at http://security.cs.rpi.edu/~candej2/kernel/trivial_sploit.c
and http://security.cs.rpi.edu/~candej2/kernel/trivial.c, and was trying to understand how it works.
I tried running it by doing
# insmod trivial.ko
$ ./exploit
The exploit got to
[+] mapped 0x31337000
And then then in my VM (I set kgdboc), I saw
BUG: Unable to handle kernel paging request at 0000000031337000
IP: [<0000000031337000>] 0x31337000
PGD 3a89e067 PUD 3aea2067 PMD 3b333067 PTE 31b57067
last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host2/target2:0:0/2:0:0:0/block/sda/dev
KGDB: Waiting for remote debugger
Why might this be the case?
I found out - it's SMEP. I can disable it by adding nosmep as a boot parameter
I am trying to get an original BeagleBoard (revC4) to boot a Angstrom OpenEmbedded image. Using instructions found here:
http://elinux.org/BeagleBoardAndOpenEmbeddedGit
and:
http://www.angstrom-distribution.org/building-angstrom
I have followed everything but the program freeze with message booting the kernel.
Output:
Texas Instruments X-Loader 1.4.2 (Feb 19 2009 - 12:01:24)
Reading boot sector
Loading u-boot.bin from mmc
U-Boot 2009.11 (Feb 23 2010 - 15:33:48)
OMAP3530-GP ES3.1, CPU-OPP2 L3-165MHz
OMAP3 Beagle board + LPDDR/NAND
I2C: ready
DRAM: 256 MB
NAND: 256 MiB
In: serial
Out: serial
Err: serial
Board revision C4
Die ID #40a8000400000000040365fa1301c014
Hit any key to stop autoboot: 0
mmc1 is available
reading boot.scr
** Unable to read "boot.scr" from mmc 0:1 **
reading uImage
4335440 bytes read
Booting from mmc ...
Booting kernel from Legacy Image at 82000000 ...
Image Name: Linux-3.2.28
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4335376 Bytes = 4.1 MB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
I have tried both ttyS2 and ttyO2 in the bootargs without difference and have also upgraded both the xLoader and uBoot.
My end goal is to run a rudimentary ROS (Robot Operating System) and the BeagleBoard.
Any help would truly be appreciated.
I don't know if you ever got past this, but as I have recently been trying to get a Beagleboard up and running again, I'll put an answer here for the sake of anyone else trying to get things working.
TLDR version: Use Angstrom v2013.6 and hold the user button as you boot (or zero out the NAND flash) for the easiest solution that just works. See below for more details.
First of all, the place you're stuck at here is due to the u-boot flashed into NAND being out-of-date for a recent version of Angstrom. Simple fix is to hold down the user button when booting, which will bypass NAND and boot straight from the SD card, which presumably has the version of u-boot you just built. You can then stop the boot and either zero out the NAND or flash the current MLO and u-boot into it.
After that, you'll run into some more issues if you're using an image based on systemd. The meta-ti layer controls the basic parameters for the 'beagleboard' configuration and has its virtual/kernel provider set to linux-mainline, which is also in the meta-ti layer. At some point there was a new recipe version added for it which builds a 3.14 kernel, but the kernel configuration isn't suitable for systemd and the system will hang shortly after booting.
The good news is that Angstrom v2013.06 predates the 3.14 kernel change in meta-ti, and I was able to get that running just fine. I am trying to get a more recent Angstrom working with the 3.2 kernel that works with v2013.6, but I haven't had any luck yet.
i am a newbie to assembly and program in c (use GCC in Linux)
can anyone here tell me how to compile c code into assembly and boot from it using pen drive
i use the command (in linux terminal) :
gcc -S bootcode.c
the code gives me a bootcode.S file
what do i do with that ???
i just wanna compile the following code and run it directly from a USB stick
#include<stdio.h>
void main()
{
printf ("hi");
}
any help here ???
First of all,
You Should be aware that when you are writing bootloader codes , you should know that you are CREATING YOUR OWN ENVIRONMENT of CODE, that means, there is nothing such ready made C Library available to you or anything similar , ONLY and ONLY BIOS SERVICES (or INTERRUPT ROUTINES).
Now, if you got this, you will probably figure out that the above code won't boot since, you don't have the "stdio.h" header, this means that the CPU when executing your compiled code won't find this header and thereby won't understand what is "printf" (since printf is a method of the stdio.h header).
So if you want to print any string you need to write this function by YOUR OWN either in a separate file as a header and link its object file at compilation time when creating the final binary file or in the same file. it is up to you. There could be other ways, I'm not well familiar with them, just do some researches.
Another thing you should know, it is the BIOS who is responsible for loading this boot code (your above code in your case) into memory location 0x07C00 (0x0000h:0x7C00 in segment:offset representation), so you HAVE to mention in your code that you are writing this code on this memory location, either by
1-using the ORG instruction
2-Or by loading the appropriate registers for that (cs,ds,es)
Also, you should get yourself familiar with the segment:offset memory representation scheme, just google it or read intel manuals.
Finally, for the BIOS to load your code into the 0x07C00, the boot code must not exceed 512byte (ONLY ON FIRST SECTOR OF THE BOOTABLE MEDIA, since a sectore is 512byte) and he must find at the last two byte of this first sector (byte 510 & byte 511) of your code the boot signature 0x55AA, otherwise the BIOS won't consider this code AS BOOTABLE.
Usually this is coded as :
ORG 0x7C00
...
your boot code and to load more codes since 512byte won't be sufficient.
...
times 510 - ($ - $$) db 0x00 ; Zerofill up to 510 bytes
dw 0xAA55 ;Boot Sector signature,written in reverse order since it
will be stored as little endian notation
Just to let you know, I'm not covering everything here, because if so, I'll be writing pages about it, you need to look for more resources on the net, and here is a link to start with(coding in assembly):
http://www.brokenthorn.com/Resources/OSDevIndex.html
That's all, hopefully this was helpful to you...^_^
Khilo - ALGERIA
Booting a computer is not that easy. A bootloader needs to be written. The bootloader must obey certain rules and correspond with hardware such as ROM. You also need to disable interrupts, reserve some memory etc. Look up MikeOS, it's a great project that can better help you understand the process.
Cheers