I have a couple of doubts about how the kernel is loaded into memory. Upon inspecting /proc/kallsyms I'm able to find the address of various symbols in the kernel.
$ cat /proc/kallsyms | head -n 10
00000000 t __vectors_start
80008240 T asm_do_IRQ
80008240 T _stext
80008240 T __exception_text_start
80008244 T do_undefinstr
80008408 T do_IPI
8000840c T do_DataAbort
800084a8 T do_PrefetchAbort
80008544 t gic_handle_irq
800085a0 T secondary_startup
Is there any way I can find the base address at which the kernel is loaded?
In userspace, suppose I use a libc with say the puts function at an offset of 0x200. When loaded into memory at say the address 0x8048000, I would be able to find the resolved puts at 0x8048000 + 0x200. Would the same hold for the kernel? i.e. is the kernel image loaded up into memory as 1 contiguous .text section?
for MIPS architecture
file Platform contain the field/variable "load-..." assigned with the location in physical address space.
example:
openwrt/build_dir/target-mips_mips32_musl-1.1.16/linux-brcm63xx_smp/linux-4.4.14/arch/mips/bcm63xx/Platform
#
# Broadcom BCM63XX boards
#
platform-$(CONFIG_BCM63XX) += bcm63xx/
cflags-$(CONFIG_BCM63XX) += \
-I$(srctree)/arch/mips/include/asm/mach-bcm63xx/
load-$(CONFIG_BCM63XX) := 0xffffffff80010000
for ARM architecture
file Makefile.boot contain the field/variable "zreladdr-y" assigned with the location in physical address space.
example:
openwrt/build_dir/target-mips_mips32_musl-1.1.16/linux-brcm63xx_smp/linux-4.4.14/arch/arm/mach-omap1/Makefile.boot
zreladdr-y += 0x10008000
params_phys-y := 0x10000100
initrd_phys-y := 0x10800000
for Microblaze architecture
file Makefile contain the field/variable "UIMAGE_LOADADDR" assigned with the location in physical address space (exported from Xilinx ISE).
example:
openwrt/build_dir/target-mips_mips32_musl-1.1.16/linux-brcm63xx_smp/linux-4.4.14/arch/microblaze/boot/Makefile
UIMAGE_LOADADDR = $(CONFIG_KERNEL_BASE_ADDR)
Kernel is loaded at physical address of 1MiB which is mapped on PAGE_OFFSET + 0x00100000 (virtual address). usually 8MiB of virtual space is reserved for kernel image starting from PAGE_OFFSET + 0x00100000
As other answer states that Kernel base address is fixed for particular architecture. But due to many security issues kernel development community decided to make it random. It is called ASLR (Address Space Layout Randomization).
By reading your question (or because I am reading it in 2017), you may be trying to find offset used in ASLR (or KASLR for kernel).
KASLR offset = address of symbol loaded in memory - address of symbol present in binary.
As your question states you already know address of symbol in memory from /proc/kallsyms.
We can find address of symbol in binary using nm utility and vmlinux file.
nm vmlinux | grep do_IPI
This will print address of symbol do_IPI in vmlinux file. Subtracting these two address will provide you KASLR offset.
If you are using u-boot then at boot time bootloader usually print the kernel load address and entry point.
Erase Group Size: 512 Bytes
reading uImage
4670784 bytes read in 469 ms (9.5 MiB/s)
reading devicetree.dtb
20597 bytes read in 17 ms (1.2 MiB/s)
Booting Linux kernel with ramdisk and devicetree
## Booting kernel from Legacy Image at 02004000 ...
Image Name: Linux-4.9.0-xilinx
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4670720 Bytes = 4.5 MiB
Load Address: 10000000
Entry Point: 10000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 04000000
Booting using the fdt blob at 0x4000000
Loading Kernel Image ... OK
Loading Device Tree to 1cb3d000, end 1cb45074 ... OK
Starting kernel ...
In the case of this ARM kernel the load address was at 0x80008000. Also, the kernel is loaded in a contiguous manner.
Related
Background
I am bringing up a Linux kernel via Yocto for some vendor-provided embedded hardware. I have configured the image to boot via fitImage with an initramfs and no rootfs (there is persistent storage but this is entirely for userspace application use). Think PXE live image and you won't be far off.
Things have been going well until my initramfs image crossed the ~128MB mark. Below this and everything boots as expected and all drivers are bound without issue. Above this mark and the kernel still boots but many drivers, though not all, are not bound. This is quite perplexing as all drivers are statically built into the kernel (no modules are used on this platform). Unfortunately, one of these modules runs the platform watchdog which causes entirely predictable reboots.
Thus far I have verified that all of the symbols are present in the vmlinux image:
$ objdump -x vmlinux | grep mtk_wdt
0000000000000000 l df *ABS* 0000000000000000 mtk_wdt.c
ffffff800880ac40 l F .text 000000000000004c mtk_wdt_stop
ffffff800880ac90 l F .text 0000000000000040 mtk_wdt_shutdown
ffffff80091de778 l F .init.text 0000000000000020 mtk_wdt_driver_init
ffffff800880acd0 l F .text 000000000000004c mtk_wdt_ping
ffffff800880ad20 l F .text 0000000000000070 mtk_wdt_set_timeout
ffffff800880ad90 l F .text 0000000000000074 mtk_wdt_start
ffffff800880ae08 l F .text 0000000000000144 mtk_wdt_resume
ffffff800880af50 l F .text 0000000000000120 mtk_wdt_suspend
ffffff800880b070 l F .text 0000000000000080 mtk_wdt_remove
ffffff80088977a8 l F .text 0000000000000210 mtk_wdt_isr
ffffff80091fe0a0 l F .exit.text 000000000000001c mtk_wdt_driver_exit
ffffff800880b4f0 l F .text 0000000000000310 mtk_wdt_probe
ffffff8008c2acd8 l O .rodata 0000000000000028 mtk_wdt_info
ffffff8008c2ad00 l O .rodata 0000000000000050 mtk_wdt_ops
ffffff8008c2ad98 l O .rodata 00000000000000b8 mtk_wdt_pm_ops
ffffff8008c2ae50 l O .rodata 0000000000000190 mtk_wdt_dt_ids
ffffff80093a3cb8 l O .data 00000000000000b0 mtk_wdt_driver
ffffff800a199368 l O .bss 0000000000000008 mtk_wdt1
ffffff8009285598 l O .init.data 0000000000000008 __initcall_mtk_wdt_driver_init6
Additionally, I have sha256 checksums for the binary kernel (eg. linux.bin), initramfs and device tree in the before fitImage assembly, after fitImage assembly and after unpacking into system memory (via bootloader); all match. Near as I can tell, what gets built is what gets unpacked and booted.
Furthermore, I have enabled initcall_debug and, while I see other __initcall()s, the non-bound drivers are, unsurprisingly, missing.
I know the devices are present in the device tree and correctly configured. After boot, I get about 5 seconds of console access before the watchdog kicks; just enought time to get a command or two off. On a "working" images (initramfs < ~128MB) and on failing images (initramfs > ~128MB) the contents of /sys/bus/platform/devices are identical and I can see (among others), my watchdog:
$ ls -lha /sys/bus/platform/devices
...
lrwxrwxrwx 1 root root 0 Jan 1 00:00 10007000.watchdog -> ../../../devices/platform/10007000.watchdog
Performing the same test but comparing /sys/bus/platform/devices shows the drivers which weren't __initcall()ed as missing.
Some collective other things I have checked:
Device Tree. The same DTB is used in both the working and broken images. I have also verified the device tree in-memory as mentioned above. Device tree overlays are not used.
Real memory load offsets. Everything is where is should be and there is plenty of space in each region. I can move the kernel around in memory an the issue persists regardless of location.
Bad memory. This failure happens identically across multiple units.
Bad compression. The issue manifests regardless of kernel / initramfs compression. Currently I am testing with everything uncompressed to minimize breakage points.
Bad signing. I have disabled signature verification (applied to the fitImage partition image after all else); no dice there either.
Attempting to bundle the initramfs directly into the kernel. No change. Right now I have the initramfs as built into the fitImage but otherwise loaded and verified independently.
Bad kernel command line arguments. I am using root=/dev/ram initrd=0x48000000,384M and have traced this all the way into init/initramfs.c where unpacking is done. I was able to verify the offsets passed are, indeed, in the correct virtual memory space and sum to 384M.
Updating the kernel linker script per this forum post. I am able to see a .initramfs section generated in vmlinux via objdump but yet the issue persists.
Given all of the above, the only thing I don't know how to verify is the jump from vmlinux to linux.bin. This is done in Yocto via objcopy as follows:
[ -n "${vmlinux_path}" ] && ${OBJCOPY} -O binary -R .note -R .comment -S "${vmlinux_path}" linux.bin
Questions
How can I verify a given symbol is included in the final linux.bin?
What mechanisms would affect inclusion or exclusion of a given symbol at build time?
Which pieces of the kernel build and runtime are affected by initramfs size?
Are there any other tools / techniques / tribal wisdom which can help debug this situation?
EDIT 1:
Below is the basic memory map of where everything lands and space utilization. As mentioned above and in the comments, I can relocate the kernel, DTB and initramfs to (almost) arbitrary locations but the issue still persists.
0x40000000 - 0x40001000 = Bootloader arg area (Fixed usage)
0x40080000 - 0x41EDFFFF = Kernel (~12MB / 29.5MB used)
0x41E00000 - 0x42FF5FFF = Trampoline (96 bytes / ~6MB used)
0x42FF6000 - 0x42FFFFFF = ATF BL3-1 (Fixed usage)
0x43000000 - 0x43FFFFFF = Trusted OS (~476K / 16M used)
0x44000000 - 0x44FFFFFF = DTB (~77.3K / 16M used)
0x45000000 - 0x47FFFFFF = Trusted OS memory (dynamic)
0x48000000 - 0x5FFFFFFF = Initramfs (~129MB / 384MB used)
0x60000000 - MEM END = Free
So, like most kernel issues the real problem was not where I thought it was. As it turns out, the problem was caused by one of the other drivers earlier in the init list hanging the core, preventing the watchdog driver from being registered. How this is affected by the initramfs is beyond me and is its own question.
For anyone who comes across this in the future, the answers to my specific questions above are listed below:
How can I verify a given symbol is included in the final linux.bin?
I was not able to figure out how to do this statically. That said, I was able to print the addresses of the init functions at runtime by adding printk()s to do_initcall_level in init/main.c. The addresses printed can then be compared to the output of objdump on vmlinux (see my question for the incantation).
A really useful and in-depth description of the initcall process can be found here.
Note that you can also turn on initcall_debug, which will print each function name. In my case I wanted raw addresses which is why I chose the printk() method.
What mechanisms would affect inclusion or exclusion of a given symbol at build time?
Most of this boils down to your .config. The vast majority of inclusion / exclusion is done via the preprocessor. Other useful items are the linker script common header at include/asm-generic/vmlinux.lds.h and the platform linker script at for your device arch/<arch>/*/*.lds.
Which pieces of the kernel build and runtime are affected by initramfs size?
No idea on this one, still.
Are there any other tools / techniques / tribal wisdom which can help debug this situation?
Don't panic
I am trying to run linux on an Arduino Yun board. The Arduino board contains an Atheros AR9331 chipset
On U-Boot These are the steps I am doing:
1- Download the kernel :
ar7240> tftp 0x80060000 openwrt-ar71xx-generic-uImage-lzma.bin;
Load address: 0x80060000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
######################
done
Bytes transferred = 1441863 (160047 hex)
2- Erase Flash in order to copy the kernel:
ar7240> erase 0x9fEa0000 +0x160047
Error: end address (0xa0000046) not in flash!
Bad address format
This is the problem It seems that 0x9fEa0000 +0x160047 exceeds the total size of the flash.
So my questions are:
1- How can I figure the total amount of memory reserved for the flash in Uboot (From wich address it starts and ends), I am thinking about changing 0x9fEa0000 by a fewer address but i'am afraid i can harm other things
This is the output of the help:
ar7240> help
? - alias for 'help'
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootm - boot application image from memory
cp - memory copy
erase - erase FLASH memory
help - print online help
md - memory display
mm - memory modify (auto-incrementing)
mtest - simple RAM test
mw - memory write (fill)
nm - memory modify (constant address)
ping - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
progmac - Set ethernet MAC addresses
reset - Perform RESET of the CPU
run - run commands in an environment variable
setenv - set environment variables
tftpboot- boot image via network using TFTP protocol
version - print monitor version
2- Is there someone experienced with Atheros AR9331 chipset who can help me find the Flash mapping (From where it starts and ends) from the datasheet
You can determine the flash layout from the kernel boot command line. Either run the printenv command in u-boot or boot into the existing kernel and look through the boot log. You need to find something like the following:
(There are plenty of guides on the internet, I took this one from https://finninday.net/wiki/index.php/Arduino_yun, your board may or may not be the same).
linino> printenv
bootargs=console=ttyATH0,115200 board=linino-yun mem=64M rootfstype=squashfs,jffs2 noinitrd mtdparts=spi0.0:256k(u-boot)ro,64k(u-boot-env)ro,14656k(rootfs),1280k(kernel),64k(nvram),64k(art),15936k#0x50000(firmware)
bootcmd=bootm 0x9fea0000
This means there are the following partitions:
u-boot 0 to 256K (0x0 - 0x40000)
u-boot-env 256k to 320k (0x40000 - 0x50000)
rootfs (squashfs) 320k to 14976k (0x50000 - 0xea0000)
kernel 14976k to 16256k (0xea0000 - 0xfe0000)
nvram 16256k to 16320k (0xfe0000 - 0xff0000)
art 16320k to 16384k (0xff0000 - 0x1000000)
The rootfs partition is 14M, which is much larger than the rootfs image file (less than 8MB) so in theory you can move the kernel image at a lower address. For this you will need to modify the kernel boot line in the u-boot environment block (rootfs aand kernel partition sizes) and the bootcmd parameter so the u-boot know where the new kernel is located.
Flash is mapped to 0x9f000000 so the value in the bootcmd should be 0x9f000000 + the offset of the kernel in bytes.
What I am not sure about is if there is an overlay filesystem for any persistent changes to the flash. Can you boot into the existing system and post the output of df -h and cat /proc/mounts?
I am trying to read kernel image and calculate a checksum value of this image. Firstly, I used a smc instruction to trigger a exception, and in the exception handler, I tried to read the first bytes of the image. I don't really know what the address is, but from some documents, I know the kernel image is decompressed in address like 0x20008000, 0x30008000 or 0xC0008000(they call this ZRELADDR, i don't really know whether this is the right address...). So I tried to read the memory like this:
uint32_t test;
test = * (uint32_t *)0x30008000;
DMSG("test : %x\n",test);
But the system crashed with a data abort exception,
core data-abort at address 0x30008000
fsr 0x00000005 ttbr0 0x7df7006a ttbr1 0x7df7006a cidr 0x0
cpu #0 cpsr 0x200001b3
r0 0x00000090 r4 0x7df4bf51 r8 0x00000000 r12 0x00000000
r1 0x09010000 r5 0x806665e0 r9 0x00000000 sp 0x7df77f50
r2 0x0000000d r6 0x7f002000 r10 0x00000000 lr 0x7df273ff
r3 0x30008000 r7 0x7df77f60 r11 0x00000000 pc 0x7df052f0
ERR TEE-CORE:tee_pager_handle_fault:602: Unexpected page fault! Trap CPU
PANIC: tee_pager_handle_fault core/arch/arm/mm/tee_pager.c:603
I guess I am on the wrong way. Does anyone know how to read the kernel image in runtime environment?
Thanks for your help!
EDIT:Thanks for your reply. I am talking about secure kernel. I am trying to check the integrity of the kernel under TrustZone, and to insure the kernel haven't be compromised. So I guess a checksum like hash value may help me. Also, I am a novice who is trying to be familiar with the memory system of arm, so I tried to start with simple read some certain memory address. I have tried to read 0xc0000000 as Artless Noise said, but the same error occurs again. Again I tried to find "_test" and "stext" address in System.map, which is 0x80008000, and error occurs again.
The beginning of the RAM is usually mapped at 0xC0000000. This depends on CONFIG_PAGE_OFFSET:
- VMSPLIT_3G: 0xC0000000
- VMSPLIT_2G: 0x80000000
- VMSPLIT_1G: 0x40000000
Note that this is a virtual address if you have an MMU (usual case), the physical address will depend on your actual architecture (it may or may not be 0x00000000). The kernel is loaded a few pages after that, at an offset of 0x8000.
So, you can probably find the (uncompressed) kernel at 0xC0008000 but it may as well be located somewhere else.
You can also try to ioremap() offset 0x8000 of your RAM.
Can you give us a bit more information on the particular SoC you are working on?
If you are in secure mode and you believe that in secure mode it access physical address then from below these macro you can deduce the physical address.
The physical address at which kernel loads is PHYS_OFFSET + TEXT_OFFSET (text offset is 0x8000),
PHYS_OFFSET definition will be depend on the CONFIG_ARM_PATCH_PHYS_VIRT patch.
if CONFIG_ARM_PATCH_PHYS_VIRT is defined, then PHYS_OFFSET will be equal to __pv_phys_offset otherwise PHYS_OFFSET will be defined as CONFIG_PHYS_OFFSET in your kernel config file.
In the process of understanding ELF program loading in Linux I was trying to experiment with the load address of a segment.
Using ld with the following linker script:
SECTIONS
{
. = 0x2000;
.text :
{
*(.text)
}
}
With the following link command:
ld -o elftest -z max-page-size=4096 -Telftest.l elftest.o
Please assume that the elftest.o is the compilation result of a trivial non important asm code.
The program links correctly with the entrypoint at 0x2000. What happens is that the program gets killed by the kernel with the output KILLED in the shell.
I want to set one thing straight:
it isn't a segfault or any other exception, the program doesn't even reach entry point
The readelf output for --segments is
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000001000 0x0000000000002000 0x0000000000002000
0x0000000000000009 0x0000000000000009 R E 1000
What I understand is the following:
Using max-page-size=4096 allows me to break the 2MiB aligment requirment for ld https://lkml.org/lkml/2012/7/9/46
The segment must be in aligned to a memory page, i.e. 4KiB so setting to 4096 is correct
The base address is arbitrary, usually for x86_64 is 0x400000
What I don't understand is:
When aligment requirment is met, why does an small address doesn't work (gets killed)?
If there is a minimun required address where is documented?
Why does it work with a higher base address i.e. 0x16000?
I'm doing this test in a 64bit computer with Arch Linux with kernel 3.17.1 and ld 2.24
You're running into kernel limitations on where pages can be mapped by userspace applications; these restrictions are intended to prevent certain kernel exploits from working. The minimum mappable address is set by the vm.mmap_min_addr sysctl value, and is usually at least 4096 (i.e, 0x1000).
For details, see: https://wiki.debian.org/mmap_min_addr (The situation is not unique to Debian; their documentation is just the first I found.)
I'm trying to understand embedded Linux principles and can't figure out addresses at u-boot output.
For example, I have UDOO board based on i.MX6 quad processor and I got following output from U-Boot:
U-Boot 2013.10-rc3 (Jan 20 2014 - 13:33:34)
CPU: Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: UDOO
DRAM: 1 GiB
MMC: FSL_SDHC: 0
No panel detected: default to LDB-WVGA
Display: LDB-WVGA (800x480)
In: serial
Out: serial
Err: serial
Net: using phy at 6
FEC [PRIME]
Warning: FEC MAC addresses don't match:
Address in SROM is 00:c0:08:88:a5:e6
Address in environment is 00:c0:08:88:9c:ce
Hit any key to stop autoboot: 0
Booting from mmc ...
4788388 bytes read in 303 ms (15.1 MiB/s)
## Booting kernel from Legacy Image at 12000000 ...
Image Name: Linux-3.0.35
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4788324 Bytes = 4.6 MiB
Load Address: 10008000
Entry Point: 10008000
Verifying Checksum ... OK
Loading Kernel Image ... OK
Starting kernel ...
I don't understand the value of Load address 0x10008000. According to documentation for this particular processor, at address zone 0x10000000 - 0xffffffff is mapped main memory. But what is 0x8000 offset? I can't figure out reason for this value.
I also don't understand address 0x12000000, where the kernel image is loaded from. Is there mapped memory region for SD card?
Please, can you give me some explanation for these addresses or even better, some references to resources about this topic. My goal is to learn how to port u-boot and Linux kernel to another boards.
Thank you!
If you check the environment variables of the u-boot, you will find that kernel image is copied from boot device to the RAM location(Here, 12000000) through command like fatload.
Now, This is not the LOADADDRESS. You give LOADADDRESS to command line while compiling the kernel, This address is mostly at 32K offset from start of the RAM in Physical address space of the processor.
Your RAM is mapped at 10000000 and kernel LOADADDRESS is 10008000(32K offset). bootm command uncompress the kernel image from 12000000 to 10008000 address and then calls the kernel entry point.
check out include/configs folder. It contains all the board definitions
i.MX uboot include/configs
To port uboot to another port, base on a very similar board and modify from there.