I was recently debugging an application that was segfaulting on a regular basis--I solved the problem, which was relatively mundane (reading from a null pointer), but I have a few residual questions I've been unable to solve on my own.
The gdb stack trace began like this in most cases:
0x00007fdff330059f in __strlen_sse42 () from /lib64/libc.so.6
Using information from /proc/[my proc id]/maps to attain the base address of the shared library, I could see that the problem occurred at the same instruction of the shared library--at instruction 0x13259f, which is
pcmpeqb (%rdi),%xmm1 (gdb)
So far, so good. But then, the OS (linux) would also write out an error message to /var/logs/messags, that looks like this
[3540502.783205] node[24638]: segfault at 0 ip 00007f8abbe6459f sp 00007fff7bf2f148 error 4 in libc-2.12.so[7f8abbd32000+189000]
Which confuses me. On the one hand, the kernel correctly identifies the fault (a user-mode protection fault), and, by subtracting the base address of the shared library from the instruction pointer, we arrive at the same relative offset--0x13259f--as we do by gdb. But the library the kernel identifies is different, the address of the instruction is different, and the function and instruction within that library is different. That is, the instruction within libc-2-12.so is
0x13259f <__memset_sse2+911>: movdqa %xmm0,-0x43(%edx)
So, my question is, how can gdb and the kernel message agree on the type of fault, and on the offset of the instruction relative to the base address of the shared library, but disagree on the address of the instruction pointer and the shared library being used?
But the library the kernel identifies is different,
No, it isn't. Do ls -l /lib64/libc.so.6, and you'll see that it's a symlink to libc-2.12.so.
the address of the instruction is different
The kernel message is for a different execution from the one you've observed in GDB, and address randomization caused libc-2.12.so to be loaded at a different base address.
and the function and instruction within that library is different. That is, the instruction within libc-2-12.so is 0x13259f <__memset_sse2+911>: movdqa %xmm0,-0x43(%edx)
It is likely that you looked at a different libc-2.12.so from the one that is actually used.
Related
I know that:
When installing a SIGSEGV signal handler with sigaction and a sa_sigaction (rather than sa_handler), the signal handler receives a siginfo_t*, of which the si_addr is the address at which the fault occurred.
Using the ucontext_t we can inspect the values of registers, for example the instruction pointer, albeit not in a platform-independent way (Linux signal handling. How to get address of interrupted instruction?).
My question: can we also know which register caused the fault? Given that we don't have memory-to-memory moves, this should be only one register (after all, there is also only a single si_addr). Of course I could inspect all registers and search for si_addr, but there may be more than one match.
I would be perfectly happy with solutions that are not platform-independent.
The load/store address might not be in any single register; it could the result of an addressing mode like [rdi + rax*4 + 100] or something.
There is no easy solution to print what a full debugger would, other than running your program under a debugger to catch the fault in the first place, like a normal person. Or let it generate a coredump for you to analyze offline, if you need to debug crashes that happened on someone else's system.
The Linux kernel chooses to dump instruction bytes starting at the code address of the fault (or actually somewhat before it for context), and the contents of all registers. Disassembly to see the faulting instruction can be done after the fact, from the crashlog, along with seeing register contents, without needing to include a disassembler in the kernel itself. See What is "Code" in Linux Kernel crash messages? for an example of what Linux does, and of manually picking it apart instead of using decodecode.
Currently I am reading System calls chapter of Understanding linux kernel and I could not understand the fact that how linux kernel knows address argument passed via syscall() is invalid.
Book has mentioned that address checking is delayed until it is used and when linux made used this address it generates page fault.
It further mentioned a fault can happen in three case in kernel mode
• The kernel attempts to address a page belonging to the process
address space, but either the corresponding page frame does not exist,
or the kernel is trying to write a read-only page.
• Some kernel function includes a programming bug that causes the
exception to be raised when that program is executed; alternatively,
the exception might be caused by a transient hardware error.
• A system call service routine attempts to read or write into a
memory area whose address has been passed as a system call parameter,
but that address does not belong to the process address space.
These cases must be distinguished by the page fault handler, since the actions to be taken are quite different.The page fault handler can easily recognize the first case by determining whether the faulty linear address is included in one of the memory regions owned by the process.
But how kernel distinguishes between remaining two case. Although it is explained in the text book but it looks alien to me. Please help and explain.
The page fault handler __do_page_fault includes this piece of code:
if (!(error_code & X86_PF_USER) &&
!search_exception_tables(regs->ip)) {
bad_area_nosemaphore(regs, error_code, address, NULL);
return;
}
This condition !(error_code & X86_PF_USER) is true when the system call originated from kernel mode rather than user mode. This condition !search_exception_tables(regs->ip) is true when the page fault did not occur from executing one of the instructions that use a linear that was passed to the system call. Note that regs->ip holds the instruction pointer of the instruction that caused the page fault. When both of these conditions are true, it means that either there is a bug in some kernel function or that there is some hardware error (the second case).
regs contains a snapshot of all architectural registers at the time of the page fault. On x86, this includes the CS segment register. The RPL in that register can be used to determine whether system call originated from user mode or kernel mode.
The search_exception_tables performs a binary search on sorted arrays of instruction addresses that are built at compile-time when compiling the kernel. These are basically the instructions that access an address passed to the system call.
For the other two other cases you listed, the condition !(error_code & X86_PF_USER) would be false.
In Linux, with ASLR enabled, is there a range of addresses where user stack address lies? What about heap, instruction addresses(text section)?
In general, is it possible to look at an address and tell if it is for data or for code?
Edit:
I am trying to write a Pintool that looks at the EIP after a return and checks if the EIP points to a data area. Let's assume that NX is not enabled on this system.
For some reason, this was downvoted. Fortunately, the answer can be found here:
https://security.stackexchange.com/questions/185315/stack-location-range-on-linux-for-user-process/185330#185330
cat /proc/self/maps will show the initial location of the main thread's stack. This can be inaccurate for (at least) the following reasons:
you're not in the main thread
any part of the program was built with the -fsplit-stack option, or you call a library that does something similar
you're within a signal handler that requests the sigaltstack stack instead
you do weird alloca tricks like CHICKEN Scheme does to use the stack as a heap
...
Also note that the general areas are not fully random. See the AddressSanitizer project for something that takes advantage of this.
I'm implementing the backend for a JavaScript JIT compiler that produces x86 code. Sometimes, as the result of bugs, I get segmentation faults. It can be quite difficult to trace back what caused them. Hence, I've been wondering if there would be some "easy" way to trap segmentation faults and other such crashes, and get the address of the instruction that caused the fault. This way, I could map the address back to compiled x86 assembly, or even back to source code.
This needs to work on Linux, but ideally on any POSIX compliant system. In the worst case, if I can't catch the seg fault and get the IP in my running JIT, I'd like to be able to trap it outside (kernel log?), and perhaps just have the compiler dump a big file with mappings of addresses to instructions, which I could match with a Python script or something.
Any ideas/suggestions are appreciated. Feel free to share your own debugging tips if you've ever worked on a compiler project of your own.
If you use sigaction, you can define a signal handler that takes 3 arguments:
void (*sa_sigaction)(int signum, siginfo_t *info, void *ucontext)
The third argument passed to the signal handler is a pointer to an OS and architecture specific data structure. On linux, its a ucontext_t which is defined in the <sys/ucontext.h> header file. Within that, uc_mcontext is an mcontext_t (machine context) which for x86 contains all the registers at the time of the signal in gregs. So you can access
ucontext->uc_mcontext.gregs[REG_EIP] (32 bit mode)
ucontext->uc_mcontext.gregs[REG_RIP] (64 bit mode)
to get the instruction pointer of the faulting instruction.
(My environment is 64-bit Ubuntu, my application is C++ compiled and linked with g++.)
When an application does something like divide by zero or run a asm("int $3") left in the code, one of the following gets logged via syslog to /var/log/kern.log and /var/log/messages:
Sep 10 18:06:47 VM kernel: [117194.123452] a.out[20288] trap divide error ip:45c59d sp:7fff65a91810 error:0 in a.out[400000+144000]
Sep 10 18:07:10 VM kernel: [117217.020833] a.out[20294] trap int3 ip:45c493 sp:7fff5cc559f0 error:0
In both those cases, the instruction pointer address points to something that I can easily look up in the .map file produced at link time (using the "-Wl,-Map,output.map").
But if I cause a seg fault, in this case by a call to memcpy() with the source set to NULL, the instruction pointer is so out of range, I have no idea how it is supposed to be mapped:
Sep 10 18:06:13 VM kernel: [117160.228587] a.out[20282]: segfault at 0 ip 00007f7e79209092 sp 00007fff831faf08 error 4 in libc-2.9.so[7f7e79185000+168000]
In this example, I would have expected the IP to be in the range of 0x445e70-0x445e7f, which is the location of memcpy() according to my .map file.
My question: What is the trick to interpreting the ip in this case?
According to the message it looks like it crashed inside memcpy(), from libc-2.9.so, which is mapped in to your process starting at 0x7f7e79185000. This is expected since memcpy is the function that is attempting to dereference the pointer. The instruction pointer looks valid since it is within the range of libc. If you were intending to override memcpy and call your own version, you may need to compile with -fno-builtin-memcpy.
Edit: You may be linking libc statically but according to the message you also have the libc shared library mapped into your process memory. You should see it listed in /proc/pid/maps while your program is running. It may be that you are linking with another shared library, such as libstdc++, and it depends on the libc shared library. As a result you have two versions of memcpy, and in this case it is calling the libc shared library version which is mapped at the high address. If you don't want the libc shared library then make sure you are linking all libraries statically; use the -static option at the beginning of your link line.