I want a comprehensive overview of how the debugging process occurs on a typical x86 machine running Linux operating system; let's say the program used for debugging is gdb. Question #1 : is the process of debugging facilitated by the hardware (or it is implemented completely in software instead?). If so, what architecture features from the instruction set are involved?
The x86 ISA includes a single-byte int3 encoding that's intended for software breakpoints. GDB uses this (via ptrace) by default for breakpoints.
(Why Single Stepping Instruction on X86?)
x86 also has a Trap Flag (TF) in EFLAGS for single-step mode. (https://en.wikipedia.org/wiki/Trap_flag). See also Difference between trap flag (TF) and monitor trap flag?
There are even "debug registers" for setting hardware breakpoints, without modifying the machine code to be run. And also hardware support for watch points, to break on write to a certain address. This makes GDB watch points efficient, not requiring it to single-step and manually decode the instruction to see where it writes.
https://wiki.osdev.org/CPU_Registers_x86#Debug_Registers
Implementing hardware breakpoints using x86 debug register osdev forum thread might be relevant.
Some other ISAs exist without nearly as much HW support for debugging. e.g. without a single-step flag, a debugger might have to always decode the current instruction (pointed to by program counter) to find the next one to be executed, and set a software breakpoint there.
ARM Linux used to do that to implement ptrace single-step, but that disassembler code was removed from the kernel and now just returns -EIO. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=425fc47adb5bb69f76285be77a09a3341a30799e is the commit that removed it.
Related
I'm seeking possible solution to achieve Kernel Mode Linux without modify Glibc.
The project called "Kernel Mode Linux on aarch64", which make specified processes execute in kernel mode, not all processes. (ex: programs in /trusted/) It enhance the speed of invoking system call. The background research is from Toshiyuki Maeda Website and sonicyang/KML.
If the user program execute in kernel mode, that means it can access syscall function directly.(Monolithic kernel) However, the access path of syscall is a hard path in arm64 glibc. The syscall will eventually use "svc 0" which cause an "Instruction Abort" exception. (# define INTERNAL_SYSCALL_RAW(name, nr, args...) \ in sysdeps/unix/sysv/linux/aarch64/sysdep.h). Of course, there is vDSO (vsyscall) way to go, but the current impl doesn't let most syscall functions have option to go vsyscal way.
In this situation, I have two modification plan, but both miss critical step.
Modify INTERNAL_SYSCALL_RAW to be multiplex of syscall or dl-call (or vsyscall) in glibc. How can I determine the process is in kernel mode or user mode without heavy overhead? (mrs x0, CurrentEL isn't allowed in EL0)
Replace svc 0 to bl dl-call when binelf loader loads. The program will be loaded by elf loader. We set it in kernel mode, no problem, but as we knew the libc.so is an dynamic link library. It keeps one piece in vma, but other normal user program will use it too. How can I deal with this situation? compile in static is great, but the size is really not acceptable.
Due to my limit understanding, please drop me any practical idea.
After a few research, the option 1 could work well as long as compile a customized glibc. The program runs in kernel mode must link to the the customized glibc. It'll not affect the system's glibc.
I am working on a custom version of Rocket Chip that features some extra instructions that I would like to be properly handled by Linux. Although bare-metal programs using these instructions run fine, Linux makes the same benchmarks crash with "Illegal Instruction" messages.
Does anyone know which software element of Linux - loader, disassembler, something else - is responsible for detecting illegal instructions?
My goal is to modify that piece of software so that Linux stops complaining about my instructions. If anyone knows about an easier way to suppress this kind of error, that would be very useful too.
The RISC-V implementation (the processor) raises an illegal instruction trap whenever it encounters an instruction it has not implemented. These illegal instruction traps will be piped through to Linux (either via trap delegation or after being handled by the machine-mode software), which then flow through the standard trap handling flow:
strapvec points to Handle_exception, which does a bunch of bookkeeping to avoid trashing userspace and then direct traps to the correct location.
For illegal instruction traps, you'll fall through to the excp_vect_table jump table, which handles all the boring traps.
This is indexed by scause, which in this case points to do_trap_insn_illegal.
do_trap_insn_illegal is just a generic Linux trap handler, it passes SIGILL to whatever caused the trap. This may raise a signal to a userspace task, a kernel task, or just panic the kernel directly.
There are a bunch of levels of indirection here that we're currently not doing anything with, but may serve to emulate unimplemented instructions in the future.
Im studying for the first time "Operating System". In my book i found this sentence about "User Mode" and "Kernel Mode":
"Switch from user to kernel mode" instruction is executed only in kernel
mode
I think that is a incorrect sentence as in practice there is no "switch of kernel". In fact, when a user process need to do a privileged instruction it simply ask the kernel to do something for itself. Is it correct ?
In fact, when a user process need to do a privileged instruction it simply ask the kernel to do something for itself.
But how does that happen? Details are processor (i.e. instruction set architecture) and OS specific (explained in ABI specifications relevant to your system, e.g. here), but that usually involves some machine code instruction like SYSENTER or SYSCALL (or SVC on mainframes) capable of atomically changing the CPU mode (that is switching it in a controlled manner to kernel mode). The actual parameters of the system call (including even the syscall number) are often passed in registers (but details are ABI specific).
So I feel the concept of switching from user-mode to kernel-mode is relevant, and meaningful (so "correct").
BTW, user-mode code is forbidden (by the hardware) to execute privileged machine instructions, such as those interacting with IO hardware devices (read about protection rings). If you try, you get some hardware exception (a bit similar to interrupts). Hence your code (even if it is malicious) has to make system calls, which the kernel controls (it has lots of code related to permission checking), for e.g. all IO.
Read also Operating Systems: Three Easy Pieces - freely downloadable. See also http://osdev.org/. Read system call wikipage & syscalls(2), and the Assembler HowTo.
In real life, things are much more complex. Read about System Management Mode and about the (scary) Intel Management Engine.
My question is somewhat weird but I will do my best to explain.
Looking at the languages the linux kernel has, I got C and assembly even though I read a text that said [quote] Second iteration of Unix is written completely in C [/quote]
I thought that was misleading but when I said that kernel has assembly code I got 2 questions of the start
What assembly files are in the kernel and what's their use?
Assembly is architecture dependant so how can linux be installed on more than one CPU architecture
And if linux kernel is truly written completely in C than how can it get GCC needed for compiling?
I did a complete find / -name *.s
and just got one assembly file (asm-offset.s) somewhere in the /usr/src/linux-headers-`uname -r/
Somehow I don't think that is helping with the GCC working, so how can linux work without assembly or if it uses assembly where is it and how can it be stable when it depends on the arch.
Thanks in advance
1. Why assembly is used?
Because there are certain things then can be done only in assembly and because assembly results in a faster code. For eg, "you can get access to unusual programming modes of your processor (e.g. 16 bit mode to interface startup, firmware, or legacy code on Intel PCs)".
Read here for more reasons.
2. What assembly file are used?
From: https://www.kernel.org/doc/Documentation/arm/README
"The initial entry into the kernel is via head.S, which uses machine
independent code. The machine is selected by the value of 'r1' on
entry, which must be kept unique."
From https://www.ibm.com/developerworks/library/l-linuxboot/
"When the bzImage (for an i386 image) is invoked, you begin at ./arch/i386/boot/head.S in the start assembly routine (see Figure 3 for the major flow). This routine does some basic hardware setup and invokes the startup_32 routine in ./arch/i386/boot/compressed/head.S. This routine sets up a basic environment (stack, etc.) and clears the Block Started by Symbol (BSS). The kernel is then decompressed through a call to a C function called decompress_kernel (located in ./arch/i386/boot/compressed/misc.c). When the kernel is decompressed into memory, it is called. This is yet another startup_32 function, but this function is in ./arch/i386/kernel/head.S."
Apart from these assembly files, lot of linux kernel code has usage of inline assembly.
3. Architecture dependence?
And you are right about it being architecture dependent, that's why the linux kernel code is ported to different architecture.
Linux porting guide
List of supported arch
Things written mainly in assembly in Linux:
Boot code: boots up the machine and sets it up in a state in which it can start executing C code (e.g: on some processors you may need to manually initialize caches and TLBs, on x86 you have to switch to protected mode, ...)
Interrupts/Exceptions/Traps entry points/returns: there you need to do very processor-specific things, e.g: saving registers and reenabling interrupts, and eventually restoring registers and properly returning to user mode. Some exceptions may be handled entirely in assembly.
Instruction emulation: some CPU models may not support certain instructions, may not support unaligned data access, or may not have an FPU. An option is using emulation when getting the corresponding exception.
VDSO: the VDSO is a virtual library that the kernel maps into userspace. It allows e.g: selecting the optimal syscall sequence for the current CPU (on x86 use sysenter/syscall instead of int 0x80 if available), and implementing certain system calls without requiring a context switch (e.g: gettimeofday()).
Atomic operations and locks: Maybe in a future some of these could be written using C11 support for atomic operations.
Copying memory from/to user mode: Besides using an optimized copy, these check for out-of-bounds access.
Optimized routines: the kernel has optimized version of some routines, e.g: crypto routines, memset, clear_page, csum_copy (checksum and copy to another place IP data in one pass), ...
Support for suspend/resume and other ACPI/EFI/firmware thingies
BPF JIT: newer kernels include a JIT compiler for BPF expressions (used for example by tcpdump, secmode mode 2, ...)
...
To support different architectures, Linux has assembly code (re-)written for each architecture it supports (and sometimes, there are several implementations of some code for different platforms using the same CPU architecture). Just look at all the subdirectories under arch/
Assembly is needed for a couple of reasons.
There are many instructions that are needed for the operation of an operating system that have no C equivalent, at least on most processors. A good example on Intel x86/64 processors is the iret instruciton, which returns from hardware/software interrupts. These interrupts are key to handling hardware events (like a keyboard press) and system calls from programs on older processors.
A computer does not start up in a state that is immediately ready for execution of C code. For an Intel example, when execution gets to the startup routine the processor may not be in 32-bit mode (or 64-bit mode), and the stack required by C also may not be ready. There are some other features present in some processors (like paging) which need to be turned on from assembly as well.
However, most of the Linux kernel is written in C, which interfaces with some platform specific C/assembly code through standardized interfaces. By separating the parts in this way, most of the logic of the Linux kernel can be shared between platforms. The build system simply compiles the platform independent and dependent parts together for specific platforms, which results in different executable kernel files for different platforms (and kernel configurations for that matter).
Assembly code in the kernel is generally used for low-level hardware interaction that can't be done directly from C. They're like a platform- specific foundation that's used by higher-level parts of the kernel that are written in C.
The kernel source tree contains assembly code for a variety of systems. When you compile a kernel for a particular type of system (such as an x86 PC), only the appropriate assembly code for that platform is included in the build process.
Linux is not the second version of Unix (or Unix in general). It is Unix compatible, but Unix and Linux have separate histories and, in terms of code base (of their kernels), are completely separate. Linus Torvald's idea was to write an open source Unix.
Some of the lower level things like some of the architecture dependent parts of memory management are done in assembly. The old (but still available) Linux kernel API for x86, int 0x80, is implemented in assembly. There are probably other places in the kernel that are implemented in assembly, but I don't know any others.
When you compile the kernel, you select an architecture to target. Depending on the target, the right assembly files for that architecture are included in the build.
The reason you don't find anything is because you're searching the headers, not the sources. Download a tar ball from kernel.org and search that.
I want to perform performance measurement of a change I want to make to an application by counting instructions. However, I'm not familiar enough with ARM's debug interface to know how to do this. Is there even an interface for this sort of thing? I'm perfectly capable of diving into the kernel if necessary, but my intuition tells me this sort of thing ought to be implemented in userspace.
CONFIG_PERF_EVENTS in Linux kernel .config
Userspace tools for accessing this are in linux-source/tools/perf/
HW_PERF_EVENTS enables PMU, CPU_HAS_PMU is defined thusly: (CPU_V6 || CPU_V6K || CPU_V7 || XSCALE_PMU) && (!ARCH_OMAP3 || OMAP3_EMU) so your chip likely has it. Otherwise, Linux will try to get some stats in software (sampling value of %pc in an interrupt, I suppose).
The ARM Profiler User Guide says an instruction trace can be created when run in a Real-Time System Model
and can display executed instructions counts in The Code and Replay Views disassembly panel.