I have a process waiting on a futex:
# strace -p 5538
Process 5538 attached - interrupt to quit
futex(0x7f86c9ed6a0c, FUTEX_WAIT, 20, NULL
How can I best debug such a situation? Can I identify who holds the futex? Are there any tools similar to ipcs and ipcrm but for futexes?
Try using gdb -p *PID* and then run where or bt to see a backtrace.
It won't be spectacularly useful with binaries and libraries that have had their debugging symbols stripped, but you may be able to deduce a fair bit from the context. It might be able to indicate to you which part of a complex process is hanging, and then you could examine the right part of the sources to search for the lock.
I have the same problem with a piece of c++ code. Running ubuntu 12.10 64bit. It looks to like a similar problem in 2007, where the libc was buggy (and maybe still is?).
I start a pthread which runs a traceroute in a system call. Printf before and after the system indicate, that the operating system hangs on the system call, WITHOUT executing the traceroute.
I am not sure if my linux is broken once again because of the ubuntu update, or if it's a libc related bug. Since a lot of applications seem to have "similar" problems, I assume it's stuck somewhere in the userspace.
My c++ code runs perfectly on 32bit systems and even 64bit osx, so i assume that ubuntu 12.10 + 64bit libc combination is broken.
Related
Is there a way to check which memory protection machenizem is used by the OS?
I have a program that fails with segmentation fault, in one computer (ubuntu) but not in another (RH6).
One of the explanations was memory protection mechanizem used by the OS.
Is there a way I can find / change it?
Thanks,
You might want to learn more about virtual memory, system calls, the linux kernel, ASLR.
Then you could study the role and usage of mmap & munmap system calls (also mprotect). They are the syscalls used to retrieve memory (e.g. to implement malloc & free), sometimes with obsolete syscalls like sbrk (which is increasingly useless).
You should use the gdb debugger (its watch command may be handy), and the valgrind utility. strace could also be useful.
Look also inside the /proc pseudo file system. Try to understand what
cat /proc/self/maps
is telling you (about the process running that cat). Look also inside /proc/$(pidof your-program)/maps
consider also using the pmap utility.
If it is your own source code, always compile it with all warnings and debuggiing info, e.g. gcc -Wall -Wextra -g and improve it till the compiler don't give any warnings. Use a recent version of gcc (ie 4.7) and of gdb (i.e. 7.4).
I am testing the Linux Kernel on an embedded device and would like to find situations / scenarios in which Linux Kernel would issue panics.
Can you suggest some test steps (manual or code automated) to create Kernel panics?
There's a variety of tools that you can use to try to crash your machine:
crashme tries to execute random code; this is good for testing process lifecycle code.
fsx is a tool to try to exercise the filesystem code extensively; it's good for testing drivers, block io and filesystem code.
The Linux Test Project aims to create a large repository of kernel test cases; it might not be designed with crashing systems in particular, but it may go a long way towards helping you and your team keep everything working as planned. (Note that the LTP isn't proscriptive -- the kernel community doesn't treat their tests as anything important -- but the LTP team tries very hard to be descriptive about what the kernel does and doesn't do.)
If your device is network-connected, you can run nmap against it, using a variety of scanning options: -sV --version-all will try to find versions of all services running (this can be stressful), -O --osscan-guess will try to determine the operating system by throwing strange network packets at the machine and guessing by responses what the output is.
The nessus scanning tool also does version identification of running services; it may or may not offer any improvements over nmap, though.
You can also hand your device to users; they figure out the craziest things to do with software, they'll spot bugs you'd never even think to look for. :)
You can try following key combination
SysRq + c
or
echo c >/proc/sysrq-trigger
Crashme has been known to find unknown kernel panic situations, but it must be run in a potent way that creates a variety of signal exceptions handled within the process and a variety of process exit conditions.
The main purpose of the messages generated by Crashme is to determine if sufficiently interesting things are happening to indicate possible potency. For example, if the mprotect call is needed to allow memory allocated with malloc to be executed as instructions, and if you don't have the mprotect enabled in the source code crashme.c for your platform, then Crashme is impotent.
It seems that operating systems on x64 architectures tend to have execution turned off for data segments. Recently I have updated the crashme.c on http://crashme.codeplex.com/ to use mprotect in case of __APPLE__ and tested it on a MacBook Pro running MAC OS X Lion. This is the first serious update to Crashme since 1994. Expect to see updated Centos and Freebsd support soon.
The question is how to execute aout-format binary (I mean old format which for example used on FreeBSD before it has migrated to ELF) on Linux system. Is there a possibility to do so without extra coding (is there some existing solution)? Probably it should be in form of kernel module or patch for the Linux kernel. Another solution could be user-space launcher (may be even run-time linker). I have searched for something similar but was unable to found something. I have not yet checked difference in system calls interfaces, if you have some comments about that, you are welcome to provide them.
P.S. I know that writing user-space launcher for aout static binary is quite trivial but the question is about some existing solution.
Check for CONFIG_BINFMT_AOUT in your kernel config.
If your kernel has /proc/config.gz:
zgrep CONFIG_BINFMT_AOUT /proc/config.gz
On Ubuntu and the like:
grep CONFIG_BINFMT_AOUT /boot/config-$(uname -r)
Kernel option was CONFIG_BINFMT_AOUT, not sure if it's still around or necessary.
I'd like to prevent would-be hackers from attaching to my binary on Linux systems. I see that ptrace DENY_ATTACH can be used on OSX. Is there such option that can be used on linux? How about on Windows?
Thanks for any info!
Such a system call requires kernel support. Even if it existed in Linux, it would be fairly easy to disable by compiling your own kernel.
In linux, ptrace returns -1 if the process is being ptraced.
So, one solution would be, inside your program, try to attach to your process, and if you get a -1, you will know that the program is being ptraced.
There's a nasty problem that has temporarily stumped a number of engineers at my company trying to debug it.
The C++ program is normally run on a cluster of multicore computers with MPI.
It will run for a very long time -- perhaps days -- and then suddenly fail.
Most of engineers working on it have eliminated any reasonable possibility of a bug in the program itself, so they're starting to assign blame to a possible hardware problem, but I suspect there must be a software problem in either a Linux kernel module or device driver.
What is suspect is that a kernel module or device driver, in order to do some floating-point calculations, is doing FXSAVE/FXRSTOR in a manner that is unsafe on SMP systems. It could be something as simple as doing the FXSAVE to a static buffer in a kernel routine that needed to be reentrant. That would create a race condition bug that would very rarely corrupt the floating-point context of a thread.
At the application level, what appears to be happening is that one or more bits of the MXCSR -- which is part of the FXSAVE/FXRSTOR context -- is suddenly changed, but there is no application code to change it.
I encountered something similar many years ago on Windows, which ultimately turned out to be a bug in a video driver, such that when the application code was preempted by the operating system, some MXCSR bits in that thread's context were corrupted.
I'm not an expert at Linux Kernel hacking or device driver development, but I'm reading that the reentrancy rules have been changing a lot; between non-SMP and SMP (multi-core) systems; between kernel versions; etc. So the possibility of a race-condition bug seems reasonable.
So my question is: Are there any known Linux driver(or kernel) bugs that fit that description?
Any precedents that I could cite would be helpful, if they had similar symptoms. At this point, a lot of the people involved are (IMHO) wasting time thinking "well, there's no bug in my code, so it must be bad hardware." and I'd like to get them beyond that and looking for something more likely to be the true cause.
The source for your kernel is available, usually as a src.rpm. You can extract this (and the .tgz inside) and then grep everything for fxsave asm instructions and the like. I'd be very surprised if you find something, but who knows? If you are running any binary video drivers then see if the problem persists without them loaded.
download kernel-2-whatever.src.rpm
mkdir temp; cd temp
rpm2cpio ../kernel*rpm | cpio -id
tar xvf linux-*.tgz
grep -ri fxsave *