I'm currently developing an application that is meant to run on Linux for MIPSEL platform.
The application uses POSIX semaphores inside one of its DSO, and the attention in my case is for the sem_init() function used inside one DSO on which the app depends.
The application is currently using the GLIBC_2.31, but I observed the same phenomenon also using the GLIBC_2_26.
Now I'm going to explain the anomaly and the current status of the troubleshooting.
All happened when I noticed a process waiting on a semaphore wouldn't wake when the semaphore incremented its value.
Just for the records, I'm aware that for multiprocessing, the semaphore must sit in a memory area that is shared between the processes that use the semaphore.
Now, investigating further using strace, I also noticed that the semaphore I initialized using sem_init(&semaphore, 1, 0) resulted in being private, despite I initialized it to be shared.
futex(0x774ce0bc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {tv_sec=1615204364, tv_nsec=515578961}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
The next step I performed on this issue has been to debug using gdb the semaphore initialization process. There I discovered that in this particular combination, MIPSEL - LINUX - GLIBC, the sem_init() exists in two versions:
$ mipsel-linux-objdump -T libpthread.so.0 | grep sem_init
00012e30 g DF .text 00000040 (GLIBC_2.0) sem_init
00012dd0 g DF .text 00000060 GLIBC_2.2 sem_init
For some reason, the linker decided to link my code against the older (not supporting shared semaphores) instead of the newer.
Looking at the GLIBC code, particularly where the sem_init() is implemented (/nptl/sem_init.c), I realized that sem_init() is an alias and that two symbols identify the functions' implementation univocally.
My next move then has been to call directly the function I needed __new_sem_init(), just to be sure any linker alchemy I'm not aware of would interfere with my intentions.
Unfortunately, the symbol is not exported, and it can't be used.
Looking at my library liba.so which uses sem_wait(), sem_init(), and sem_post() I noticed that these symbols are all exported by libpthread.so.0.
$ mipsel-linux-nm -D libpthread.so.0 | grep sem_
00014a10 T sem_clockwait
00013784 T sem_close
00012e70 T sem_destroy
00012e70 T sem_destroy
00013b14 T sem_getvalue
00013b00 T sem_getvalue
00012e30 T sem_init
00012dd0 T sem_init
000131b8 T sem_open
00014ab0 T sem_post
00014b90 T sem_post
00014508 T sem_timedwait
00013f40 T sem_trywait
00013fac T sem_trywait
00013920 T sem_unlink
00013eb0 T sem_wait
00014020 T sem_wait
But looking at the liba.so dependencies, I've been surprised by the fact that the library depends only on the main libc library (libc.so.6) and has no dependency on libpthread.so.0 where the actual sem_* symbols are implemented.
$ /lib/ld-2.31.so --list /usr/lib/liba.so
linux-vdso.so.1 (0x7ffaa000)
libc.so.6 => /lib/libc.so.6 (0x77ddc000)
/lib/ld-2.31.so (0x77f8a000)
I guess this might be strongly related to my problem, but I have no clue how to link these two facts.
How the liba.so could be compiled and linked using symbols its dependencies do not provide?
$ mipsel-linux-objdump -T libc.so.6 | grep sem
000fab80 g DF .text 00000070 GLIBC_2.0 semget
000fabf0 g DF .text 000000b0 GLIBC_2.2 semctl
000faca0 w DF .text 00000074 GLIBC_2.3.3 semtimedop
000359d0 g DF .text 00000074 GLIBC_2.0 sigisemptyset
00149854 g DF .text 000000b0 (GLIBC_2.0) semctl
000fab60 g DF .text 00000018 GLIBC_2.0 semop
$
How to force the GNU linker to link against sem_init##GLIBC_2.2 instead of sem_init#GLIBC_2.0?
It seems like my problem was indeed an underlinking issue.
After proper research on the literature and a considerable amount of tests, I've finally been able to resolve my original problem. Make my application use the correct version of the sem_init().
My assumption that because my library did not carry the libpthread dependency was on the right path to follow.
Checking about my Makefile, I saw the -libpthread switch was missing in the statement, creating the shared library.
That was the root of the problem, and once I added the switch, everything started working as expected.
One of the things that delayed reaching the solution was building the same library with the same Makefile on a different target; the libpthread dependency was included in the library.
But I do not have any explanation about why the dynamic linker has picked the oldest symbol sem_init#GLIBC_2_0 in place of sem_init#GLIBC_2_2.
I understand the linker has been able to link my sem_init() because the original executable app has libpthread as a dependency, so at link time, the library was there, and it could be able to find the symbol.
What I can not explain is why it picked the sem_init#GLIBC_2_0
According to these sources
http://peeterjoot.com/2019/09/20/an-example-of-linux-glibc-symbol-versioning/ The ## one means that it applies to new code, whereas the #MYSTUFF_1.1 is a load only function, and no new code can use that symbol.
https://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-library-handles-backward-compatibility/ The ## tells the dynamic linker that this version is the default version.
https://web.archive.org/web/20100430151127/http://www.trevorpounds.com/blog/?33 The double ## can only be defined once for a given symbol since it denotes the default version to use.
If a symbol is unversioned, the symbol that carries the double ## should be picked by the dynamic linker.
Is this statement true only if the symbol is unversioned and the object where the symbol is required properly contains a dependency on the DSO containing that symbol?
Related
Background
I am bringing up a Linux kernel via Yocto for some vendor-provided embedded hardware. I have configured the image to boot via fitImage with an initramfs and no rootfs (there is persistent storage but this is entirely for userspace application use). Think PXE live image and you won't be far off.
Things have been going well until my initramfs image crossed the ~128MB mark. Below this and everything boots as expected and all drivers are bound without issue. Above this mark and the kernel still boots but many drivers, though not all, are not bound. This is quite perplexing as all drivers are statically built into the kernel (no modules are used on this platform). Unfortunately, one of these modules runs the platform watchdog which causes entirely predictable reboots.
Thus far I have verified that all of the symbols are present in the vmlinux image:
$ objdump -x vmlinux | grep mtk_wdt
0000000000000000 l df *ABS* 0000000000000000 mtk_wdt.c
ffffff800880ac40 l F .text 000000000000004c mtk_wdt_stop
ffffff800880ac90 l F .text 0000000000000040 mtk_wdt_shutdown
ffffff80091de778 l F .init.text 0000000000000020 mtk_wdt_driver_init
ffffff800880acd0 l F .text 000000000000004c mtk_wdt_ping
ffffff800880ad20 l F .text 0000000000000070 mtk_wdt_set_timeout
ffffff800880ad90 l F .text 0000000000000074 mtk_wdt_start
ffffff800880ae08 l F .text 0000000000000144 mtk_wdt_resume
ffffff800880af50 l F .text 0000000000000120 mtk_wdt_suspend
ffffff800880b070 l F .text 0000000000000080 mtk_wdt_remove
ffffff80088977a8 l F .text 0000000000000210 mtk_wdt_isr
ffffff80091fe0a0 l F .exit.text 000000000000001c mtk_wdt_driver_exit
ffffff800880b4f0 l F .text 0000000000000310 mtk_wdt_probe
ffffff8008c2acd8 l O .rodata 0000000000000028 mtk_wdt_info
ffffff8008c2ad00 l O .rodata 0000000000000050 mtk_wdt_ops
ffffff8008c2ad98 l O .rodata 00000000000000b8 mtk_wdt_pm_ops
ffffff8008c2ae50 l O .rodata 0000000000000190 mtk_wdt_dt_ids
ffffff80093a3cb8 l O .data 00000000000000b0 mtk_wdt_driver
ffffff800a199368 l O .bss 0000000000000008 mtk_wdt1
ffffff8009285598 l O .init.data 0000000000000008 __initcall_mtk_wdt_driver_init6
Additionally, I have sha256 checksums for the binary kernel (eg. linux.bin), initramfs and device tree in the before fitImage assembly, after fitImage assembly and after unpacking into system memory (via bootloader); all match. Near as I can tell, what gets built is what gets unpacked and booted.
Furthermore, I have enabled initcall_debug and, while I see other __initcall()s, the non-bound drivers are, unsurprisingly, missing.
I know the devices are present in the device tree and correctly configured. After boot, I get about 5 seconds of console access before the watchdog kicks; just enought time to get a command or two off. On a "working" images (initramfs < ~128MB) and on failing images (initramfs > ~128MB) the contents of /sys/bus/platform/devices are identical and I can see (among others), my watchdog:
$ ls -lha /sys/bus/platform/devices
...
lrwxrwxrwx 1 root root 0 Jan 1 00:00 10007000.watchdog -> ../../../devices/platform/10007000.watchdog
Performing the same test but comparing /sys/bus/platform/devices shows the drivers which weren't __initcall()ed as missing.
Some collective other things I have checked:
Device Tree. The same DTB is used in both the working and broken images. I have also verified the device tree in-memory as mentioned above. Device tree overlays are not used.
Real memory load offsets. Everything is where is should be and there is plenty of space in each region. I can move the kernel around in memory an the issue persists regardless of location.
Bad memory. This failure happens identically across multiple units.
Bad compression. The issue manifests regardless of kernel / initramfs compression. Currently I am testing with everything uncompressed to minimize breakage points.
Bad signing. I have disabled signature verification (applied to the fitImage partition image after all else); no dice there either.
Attempting to bundle the initramfs directly into the kernel. No change. Right now I have the initramfs as built into the fitImage but otherwise loaded and verified independently.
Bad kernel command line arguments. I am using root=/dev/ram initrd=0x48000000,384M and have traced this all the way into init/initramfs.c where unpacking is done. I was able to verify the offsets passed are, indeed, in the correct virtual memory space and sum to 384M.
Updating the kernel linker script per this forum post. I am able to see a .initramfs section generated in vmlinux via objdump but yet the issue persists.
Given all of the above, the only thing I don't know how to verify is the jump from vmlinux to linux.bin. This is done in Yocto via objcopy as follows:
[ -n "${vmlinux_path}" ] && ${OBJCOPY} -O binary -R .note -R .comment -S "${vmlinux_path}" linux.bin
Questions
How can I verify a given symbol is included in the final linux.bin?
What mechanisms would affect inclusion or exclusion of a given symbol at build time?
Which pieces of the kernel build and runtime are affected by initramfs size?
Are there any other tools / techniques / tribal wisdom which can help debug this situation?
EDIT 1:
Below is the basic memory map of where everything lands and space utilization. As mentioned above and in the comments, I can relocate the kernel, DTB and initramfs to (almost) arbitrary locations but the issue still persists.
0x40000000 - 0x40001000 = Bootloader arg area (Fixed usage)
0x40080000 - 0x41EDFFFF = Kernel (~12MB / 29.5MB used)
0x41E00000 - 0x42FF5FFF = Trampoline (96 bytes / ~6MB used)
0x42FF6000 - 0x42FFFFFF = ATF BL3-1 (Fixed usage)
0x43000000 - 0x43FFFFFF = Trusted OS (~476K / 16M used)
0x44000000 - 0x44FFFFFF = DTB (~77.3K / 16M used)
0x45000000 - 0x47FFFFFF = Trusted OS memory (dynamic)
0x48000000 - 0x5FFFFFFF = Initramfs (~129MB / 384MB used)
0x60000000 - MEM END = Free
So, like most kernel issues the real problem was not where I thought it was. As it turns out, the problem was caused by one of the other drivers earlier in the init list hanging the core, preventing the watchdog driver from being registered. How this is affected by the initramfs is beyond me and is its own question.
For anyone who comes across this in the future, the answers to my specific questions above are listed below:
How can I verify a given symbol is included in the final linux.bin?
I was not able to figure out how to do this statically. That said, I was able to print the addresses of the init functions at runtime by adding printk()s to do_initcall_level in init/main.c. The addresses printed can then be compared to the output of objdump on vmlinux (see my question for the incantation).
A really useful and in-depth description of the initcall process can be found here.
Note that you can also turn on initcall_debug, which will print each function name. In my case I wanted raw addresses which is why I chose the printk() method.
What mechanisms would affect inclusion or exclusion of a given symbol at build time?
Most of this boils down to your .config. The vast majority of inclusion / exclusion is done via the preprocessor. Other useful items are the linker script common header at include/asm-generic/vmlinux.lds.h and the platform linker script at for your device arch/<arch>/*/*.lds.
Which pieces of the kernel build and runtime are affected by initramfs size?
No idea on this one, still.
Are there any other tools / techniques / tribal wisdom which can help debug this situation?
Don't panic
I am learning assembly, but I'm having trouble understanding how a program is executed by a CPU when using gnu/linux on arm. I will elaborate.
Problem:
I want my program to return 5 as it's exit status.
Assembly for this is:
.text
.align 2
.global main
.type main, %function
main:
mov w0, 5 //move 5 to register w0
ret //return
I then assemble it with:
as prog.s -o prog.o
Everything ok up to here. I understand I then have to link my object file to the C library in order to add additional code that will make my program run. I then link with(paths are omitted for clarity):
ld crti.o crtn.o crt1.o libc.so prog.o ld-linux-aarch64.so.1 -o prog
After this, things work as expected:
./prog; echo $?
5
My problem is that I can't figure out what the C standard library is actually doing here. I more or less understand crti/n/1 are adding entry code to my program (eg the .init and .start sections), but no clue what's libc purpose.
I am interested at what would be a minimal assembly implementation of "returning 5 as exit status"
Most resources on the web focus on the instructions and program flow once you are in main. I am really interested at what are all the steps that go on once I execute with ./. I am now going through computer architecture textbooks, but I hope I can get a little help here.
The C language starts at main() but for C to work you in general need at least a minimal bootstrap. For example before main can be called from the C bootstrap
1) stack/stackpointer
2) .data initialized
3) .bss initalized
4) argc/argv prepared
And then there is C library which there are many/countless C libraries and each have their own designs and requirements to be satisfied before main is called. The C library makes system calls into the system so this starts to become system (operating system, Linux, Windows, etc) dependent, depending on the design of the C library that may be a thin shim or heavily integrated or somewhere in between.
Likewise for example assuming that the operating system is taking the "binary" (binary formats supported by the operating system and rules for that format are defined by the operating system and the toolchain must conform likewise the C library (even though you see the same brand name sometimes assume toolchain and C library are separate entities, one designed to work with the other)) from a non volatile media like a hard drive or ssd and copying the relevant parts into memory (some percentage of the popular, supported, binary file formats, are there for debug or file format and not actually code or data that is used for execution).
So this leaves a system level design option of does the binary file format indicate .data, .bss, .text, etc (note that .data, .bss, .text are not standards just convention most people know what that means even if a particular toolchain did not choose to use those names for sections or even the term sections).
If so the operating systems loader that takes the program and loads it into memory can choose to put .data in the right place and zero .bss for you so that the bootstrap does not have to. In a bare-metal situation the bootstrap would normally handle the read/write items because it is not loaded from media by some other software it is often simply in the address space of the processor on a rom of some flavor.
Likewise argv/argc could be handled by the operating systems tool that loads the binary as it had to parse out the location of the binary from the command line assuming the operating system has/uses a command line interface. But it could as easily simply pass the command line to the bootstrap and the bootstrap has to do it, these are system level design choices that have little to do with C but everything to do with what happens before main is called.
The memory space rules are defined by the operating system and between the operating system and the C library which often contains the bootstrap due to its intimate nature but I guess the C library and bootstrap could be separate. So linking plays a role as well, does this operating system support protection is it just read/write memory and you just need to spam it in there or are there separate regions for read/only (.text, .rodata, etc) and read/write (.data, .bss, etc). Someone needs to handle that, linker script and bootstrap often have a very intimate relationship and the linker script solution is specific to a toolchain not assumed to be portable, why would it, so while there are other solutions the common solution is that there is a C library with a bootstrap and linker solution that are heavily tied to the operating system and target processor.
And then you can talk about what happens after main(). I am happy to see you are using ARM not x86 to learn first, although aarch64 is a nightmare for a first one, not the instruction set just the execution levels and all the protections, you can go a long long way with this approach but there are some things and some instructions you cannot touch without going bare metal. (assuming you are using a pi there is a very good bare-metal forum with a lot of good resources).
The gnu tools are such that binutils and gcc are separate but intimately related projects. gcc knows where things are relative to itself so assuming you combined gcc with binutils and glibc (or you just use the toolchain you found), gcc knows relative to where it executed to find these other items and what items to pass when it calls the linker (gcc is to some extent just a shell that calls a preprocessor a compiler the assembler then linker if not instructed not to do these things). But the gnu binutils linker does not. While as distasteful as it feels to use, it is easier just to
gcc test.o -o test
rather than figure out for that machine that day what all you need on the ld command line and what paths and depending on design the order on the command line of the arguments.
Note you can probably get away with this as a minimum
.global main
.type main, %function
main:
mov w0, 5 //move 5 to register w0
ret //return
or see what gcc generates
unsigned int fun ( void )
{
return 5;
}
.arch armv8-a
.file "so.c"
.text
.align 2
.p2align 4,,11
.global fun
.type fun, %function
fun:
mov w0, 5
ret
.size fun, .-fun
.ident "GCC: (GNU) 10.2.0"
I am used to seeing more fluff in there:
.arch armv5t
.fpu softvfp
.eabi_attribute 20, 1
.eabi_attribute 21, 1
.eabi_attribute 23, 3
.eabi_attribute 24, 1
.eabi_attribute 25, 1
.eabi_attribute 26, 2
.eabi_attribute 30, 2
.eabi_attribute 34, 0
.eabi_attribute 18, 4
.file "so.c"
.text
.align 2
.global fun
.syntax unified
.arm
.type fun, %function
fun:
# args = 0, pretend = 0, frame = 0
# frame_needed = 0, uses_anonymous_args = 0
# link register save eliminated.
mov r0, #5
bx lr
.size fun, .-fun
.ident "GCC: (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609"
.section .note.GNU-stack,"",%progbits
Either way, you can look up each of the assembly language items and decide if you really need them or not, depends in part on if you feel the need to use a debugger or binutils tools to tear apart the binary (do you really need to know the size of fun for example in order to learn assembly language?)
If you wish to control all of the code and not link with a C library you are more than welcome to you need to know the memory space rules for the operating system and create a linker script (the default one may in part be tied to the C library and is no doubt overly complicated and not something you would want to use as a starting point). In this case being two instructions in main you simply need the one address space valid for the binary, however the operating system enters (ideally using the ENTRY(label), which could be main if you want but often is not _start is often found in linker scripts but is not a rule either, you choose. And as pointed out in comments you would need to make the system call to exit the program. System calls are specific to the operating system and possibly version and not specific to a target (ARM), so you would need to use the right one in the right way, very doable, your whole project linker script and assembly language could be maybe a couple dozen lines of code total. We are not here to google those for you so you would be on your own for that.
Part of your problem here is you are searching for compiler solutions when the compiler in general has absolutely nothing to do with any of this. A compiler takes one language turns it into another language. An assembler same deal but one is simple and the other usually machine code, bits. (some compilers output bits not text as well). It is equivalent to looking up the users manual for a table saw to figure out how to build a house. The table saw is just a tool, one of the tools you need, but just a generic tool. The compiler, specific gnu's gcc, is generic it does not even know what main() is. Gnu follows the Unix way so it has a separate binutils and C library, separate developments, and you do not have to combine them if you do not want to, you can use them separately. And then there is the operating system so half your question is buried in operating system details, the other half in a particular C library or other solution to connect main() to the operating system.
Being open source you can go look at the bootstrap for glibc and others and see what they do. Understanding this type of open source project the code is nearly unreadable, much easier to disassemble sometimes, YMMV.
You can search for the Linux system calls for arm aarch64 and find the one for exit, you can likely see that the open source C libraries or bootstrap solutions you find that are buried under what you are using today, will call exit but if not then there is some other call they need to make to return back to the operating system. It is unlikely it is a simple ret with a register that holds a return value, but technically that is how someone could choose to do it for their operating system.
I think you will find for Linux on arm that Linux is going to parse the command line and pass argc/argv in registers, so you can simply use them. And likely going to prep .data and .bss so long as you build the binary correctly (you link it correctly).
Here's a bare minimum example.
Run it with:
gcc -c thisfile.S && ld thisfile.o && ./a.out
Source code:
#include <sys/syscall.h>
.global _start
_start:
movq $SYS_write, %rax
movq $1, %rdi
movq $st, %rsi
movq $(ed - st), %rdx
syscall
movq $SYS_exit, %rax
movq $1, %rdi
syscall
st:
.ascii "\033[01;31mHello, OS World\033[0m\n"
ed:
I have a 32bit elf program that I have to exploit remotely (for academic purposes).
The final goal is to spawn a shell. I have a stack that I can fill with any data I want and I can abuse one of the printf format strings. The only problem is that system/execv/execvp is not imported. The .got.plt segment is full of not-very-useful functions and I want to replace atoi with system because of how similar their signature is and the flow of the code indicates that that is the right function to replace. For the following attempts, I used IDA remote debug, so bad stack alignment and not proper format string is out of question. I wanted to make sure it is doable and apparently for me it isn't yet.
At first I tried to replace atoi#.got.plt with the unrandomized address of system. Got SIGSEGV.
Alright, it's probably because of ASLR, so let's try something else. I loaded up gdb and looked up system#0xb7deeda0 and atoi#0xb7de1250. Then I calculated the diff, which is 0xDB50. So the next time when I changed the address of atoi to system in the .got.plt segment, I actually just added diff to that value to get the address of system. Got SIGSEGV again.
My logic:
0xb7deeda0 <__libc_system>
0xb7de1250 <atoi>
diff = 0xb7deeda0 - 0xb7de1250
system#.got.plt = atoi#.got.plt + diff
example: 0x08048726 + DB50 = 0x08056276
Can anyone tell me what I did wrong and how can I jump to a "valid system()" with the help of leaking a function address from .got.plt?
Answering to my own question. Measuring the distance between functions in your
l̲o̲c̲a̲l̲ libc does not guarantee that the r̲e̲m̲o̲t̲e̲ libc will have the same alignment.
You have to find the libc version somehow, then you can get the address difference like so:
readelf -s /lib32/libc-2.19.so | grep printf
Possible ways to find the libc version if you know two addresses:
Libc binary collection
libcdb.com
pwnlib
... or you have access to the shell on the remote machine and can peek into the library with readelf yourself
I want to learn linux ebpf vm, if I write a ebpf program test.c, used llvm:
clang -O2 -target bpf -o test.o test.c. How to get the ebpf assembly like tcpdump -d in classic bpf, thanks.
This depends on what you mean exactly by “learn[ing] linux ebpf vm”.
The language itself
If you mean learning about the instructions of eBPF, the assembly-like language itself, you can have a look at the documentation from the kernel (quite dense) or at this summarized version of the syntax from bcc project.
The virtual machine
If you prefer to see how the internals of the eBPF virtual machine work, you can either have a look at various presentations (I recommend those from D. Borkmann), I have a list here in this blog post; or you can directly read at the kernel sources, under linux/kernel/bpf (in particular file core.c). Alternatively, there is a simpler userspace implementation available.
Dump eBPF instructions
Now if you want to see the code that has been compiled from C to eBPF, here are a couple of solutions.
Read the object file
For my part I compile with the command presented in the tc-bpf man page:
__bcc() {
clang -O2 -emit-llvm -c $1 -o - | \
llc -march=bpf -filetype=obj -o "`basename $1 .c`.o"
}
alias bcc=__bcc
The code is translated into eBPF and stored in one of the sections of the ELF file produced. Then I can examine my program with tools such as objdump or readelf. For example, if my program is in the classifier section:
$ bcc return_zero.c
$ readelf -x classifier return_zero.o
Hex dump of section 'classifier':
0x00000000 b7000000 02000000 95000000 00000000 ................
In the above output, two instructions are displayed (little endian — the first field starting with 0x is the offset inside the section). We could parse this to put in shape the instructions and to obtain:
b7 0 0 0000 00000002 // Load 0x02 in register r0
95 0 0 0000 00000000 // Exit and return value in r0
[April 2019 edit] Dump an eBPF program loaded in the kernel
It is possible to dump the instructions of programs loaded (and then possibly attached to one of the available BPF hooks) in the kernel, either as eBPF assembly instructions, or as machine instructions if the program has been JIT-compiled. bpftool, relying on libbpf, is the go-to utility for doing such things. For example, one can see what programs are currently loaded, and note their ids, with:
# bpftool prog show
Then dumping the instructions for a program of a given id is as simple as:
# bpftool prog dump xlated id <id>
# bpftool prog dump jited id <id>
for eBPF or JITed (if available) instructions respectively. Output can also be formatted as JSON if necessary.
Advanced tools
Depending on the tools you use to inject BPF into the kernel, you can generally dump the output of the in-kernel verifier, that contains most of the instructions formatted in a human-friendly way.
With the bcc set of tools (not directly related to the previous command, and not related at all with the old 16-bit compiler), you can get this by using the relevant flags for the BPF object instance, while with tc filter add dev eth0 bpf obj … verbose this is done with the verbose keyword.
Disassemblers
The aforementioned userspace implementation (uBPF) has its own assembler and disassembler that might be of interest to you: it takes the “human-friendly” (add32 r0, r1 and the likes) instructions as input and converts into object files, or the other way round, respectively.
But probably more interesting, there is the support for debug info, coming along with a BPF disassembler, in LLVM itself: as of today it has recently been merged, and its author (A. Starovoitov) has sent an email about it on the netdev mailing list. This means that with clang/LLVM 4.0+, you should be able to use llvm-objdump -S -no-show-raw-insn my_file.o to obtain a nicely formatted output.
I have the following simple C code:
void main(){
int A = 333;
int B=244;
int sum;
sum = A + B;
}
When I compile this with
$riscv64-unknown-elf-gcc code.c -o code.o
If I want to see the assembly code I use
$riscv64-unknown-elf-objdump -d code.o
But when I explore the assembly code I see that this generates a lot of code which I assume is for Proxy Kernel support (I am a newbie to riscv). However, I do not want that this code has support for Proxy kernel, because the idea is to implement only this simple C code within an FPGA.
I read that riscv provides three types of compilation: Bare-metal mode, newlib proxy kernel and riscv Linux. According to previous research, the kind of compilation that I should do is bare metal mode. This is because I desire a minimum assembly without support for the operating system or kernel proxy. Assembly functions as a system call are not required.
However, I have not yet been able to find as I can compile a C code for get a skeleton of a minimal riscv assembly program. How can I compile the C code above in bare metal mode or for get a skeleton of a minimal riscv assembly code?
Warning: this answer is somewhat out-of-date as of the latest RISC-V Privileged Spec v1.9, which includes the removal of the tohost Control/Status Register (CSR), which was a part of the non-standard Host-Target Interface (HTIF) which has since been removed. The current (as of 2016 Sep) riscv-tests instead perform a memory-mapped store to a tohost memory location, which in a tethered environment is monitored by the front-end server.
If you really and truly need/want to run RISC-V code bare-metal, then here are the instructions to do so. You lose a bunch of useful stuff, like printf or FP-trap software emulation, which the riscv-pk (proxy kernel) provides.
First things first - Spike boots up at 0x200. As Spike is the golden ISA simulator model, your core should also boot up at 0x200.
(cough, as of 2015 Jul 13, the "master" branch of riscv-tools (https://github.com/riscv/riscv-tools) is using an older pre-v1.7 Privileged ISA, and thus starts at 0x2000. This post will assume you are using v1.7+, which may require using the "new_privileged_isa" branch of riscv-tools).
So when you disassemble your bare-metal program, it better
start at 0x200!!! If you want to run it on top of the proxy-kernel, it
better start at 0x10000 (and if Linux, it’s something even larger…).
Now, if you want to run bare metal, you’re forcing yourself to write up the
processor boot code. Yuck. But let’s punt on that and pretend that’s not
necessary.
(You can also look into riscv-tests/env/p, for the “virtual machine”
description for a physically addressed machine. You’ll find the linker script
you need and some macros.h to describe some initial setup code. Or better
yet, in riscv-tests/benchmarks/common.crt.S).
Anyways, armed with the above (confusing) knowledge, let’s throw that all
away and start from scratch ourselves...
hello.s:
.align 6
.globl _start
_start:
# screw boot code, we're going minimalist
# mtohost is the CSR in machine mode
csrw mtohost, 1;
1:
j 1b
and link.ld:
OUTPUT_ARCH( "riscv" )
ENTRY( _start )
SECTIONS
{
/* text: test code section */
. = 0x200;
.text :
{
*(.text)
}
/* data: Initialized data segment */
.data :
{
*(.data)
}
/* End of uninitalized data segement */
_end = .;
}
Now to compile this…
riscv64-unknown-elf-gcc -nostdlib -nostartfiles -Tlink.ld -o hello hello.s
This compiles to (riscv64-unknown-elf-objdump -d hello):
hello: file format elf64-littleriscv
Disassembly of section .text:
0000000000000200 <_start>:
200: 7810d073 csrwi tohost,1
204: 0000006f j 204 <_start+0x4>
And to run it:
spike hello
It’s a thing of beauty.
The link script places our code at 0x200. Spike will start at
0x200, and then write a #1 to the control/status register
“tohost”, which tells Spike “stop running”. And then we spin on an address
(1: j 1b) until the front-end server has gotten the message and kills us.
It may be possible to ditch the linker script if you can figure out how to
tell the compiler to move <_start> to 0x200 on its own.
For other examples, you can peruse the following repositories:
The riscv-tests repository holds the RISC-V ISA tests that are very minimal
(https://github.com/riscv/riscv-tests).
This Makefile has the compiler options:
https://github.com/riscv/riscv-tests/blob/master/isa/Makefile
And many of the “virtual machine” description macros and linker scripts can
be found in riscv-tests/env (https://github.com/riscv/riscv-test-env).
You can take a look at the “simplest” test at (riscv-tests/isa/rv64ui-p-simple.dump).
And you can check out riscv-tests/benchmarks/common for start-up and support code for running bare-metal.
The "extra" code is put there by gcc and is the sort of stuff required for any program. The proxy kernel is designed to be the bare minimum amount of support required to run such things. Once your processor is working, I would recommend running things on top of pk rather than bare-metal.
In the meantime, if you want to look at simple assembly, I would recommend skipping the linking phase with '-c':
riscv64-unknown-elf-gcc code.c -c -o code.o
riscv64-unknown-elf-objdump -d code.o
For examples of running code without pk or linux, I would look at riscv-tests.
I'm surprised no one mentioned gcc -S which skips assembly and linking altogether and outputs assembly code, albeit with a bunch of boilerplate, but it may be convenient just to poke around.