Linux how to debug OS freeze issue - linux

I am working on a kernel module and a user-space application to test that module.
The problem is that during testing my system hangs/freeze.
I have placed lots of debug prints in the code.
The last message that is printed is just before linux select call in my user-space application. Does select somehow freeze the system?
So, How can i debug that where is problem? whether the problem is on user-space application or kernel module?

As n.m mentioned, your userspace program can't freeze Linux, so its an error in your kernel module. The best way to debug this is to use a kernel debugger, and figure out what your module is doing wrong.
Common errors are uninitialized pointers that your module passes to the kernel or locking issues, so take a close look at those.

A userspace program cannot, by definition, freeze Linux. There's a bug in the kernel.

Related

what methods do you take when your linux kernel programs are wrong?

I did not find a good method when I write and test a linux kernel programs such as multiple kernel-level threads or other general kernel modules, what methods do you take? thanks in advance!
printk, printk and more printk.
use dmesg to view. crash the kernel sometimes deliberately to get the crashinfo, then you can decode the crashinfo
dumptrace(), dumpstack() will print the stacktrace on the dmesg.
As a last option, kgdb. but this requires a connection to another system and is a pain always to get it work.

How to use systemcalls in linux modules

I attempted to use systemcalls such as sys_sendto when programming a kernel module. But the compiler warned me that the symbol 'sys_sendto' is undefined. I'm sure I have inculded the header file syscalls.h, so please help me and thank you. P.S: My linux version is 2.6.32
For a module to link to a symbol in the kernel like sys_sendto(), it has to be exported by the kernel. Not all system calls are exported. See here
http://www.ibm.com/developerworks/linux/library/l-system-calls/
Here's a whole explanation on writing them and using them. They're not used by direct method calls because they have to be executed in kernel mode. The processor loads the syscall number into a register and then issues a hardware interrupt which the kernel processes and handles to execute your system call.

Execute code in process's stack, on recent Linux

I want to use ptrace to write a piece of binary code in a running process's stack.
However, this causes segmentation fault (signal 11).
I can make sure the %eip register stores the pointer to the first instruction that I want to execute in the stack. I guess there is some mechanism that linux protects the stack data to be executable.
So, does anyone know how to disable such protection for stack. Specifically, I'm trying Fedora 15.
Thanks a lot!
After reading all replies, I tried execstack, which really makes code in stack executable. Thank you all!
This is probably due to the NX bit on modern processors. You may be able to disable this for your program using execstack.
http://advosys.ca/viewpoints/2009/07/disabling-the-nx-bit-for-specific-apps/
http://linux.die.net/man/8/execstack
As already mentioned it is due to the NX bit. But it is possible. I know for sure that gcc uses it itself for trampolines (which are a workaround to make e.g. function pointers of nested functions). I dont looked at the detailes, but I would recommend a look at the gcc code. Search in the sources for the architecture specific macro TARGET_ASM_TRAMPOLINE_TEMPLATE, there you should see how they do it.
EDIT: A quick google for that macro, gave me the hint: mprotect is used to change the permissions of the memory page. Also be carefull when you generate date and execute it - you maybe have in addition to flush the instruction cache.

easy way to detect infinite loop in kernel of the linux

I've just spent my 2 extra hours trying to find bug in my modification of the kernel of the linux, every time when I was connecting module to the kernel it was good but when I unconnected it my mouse stopped to work, so using printk I found infinite loop, my question is does somebody know nice techniques to detect such bugs, sometimes it is difficult to find such loops, and linux becomes unpredictable, so how can I avoid infinite loops in kernel thanks in advance
There is some infrastructure in the kernel that allows you to detect some lockup conditions :
CONFIG_DETECT_SOFTLOCKUP
CONFIG_DETECT_HUNG_TASK
And the various lock checking function you can find in the "Kernel Hacking" section of the kernel config
I've always found printk useful for that, as you did.
Other options would be running your kernel in Bochs in debugging mode. And as I recall, there's a way of running the kernel in gdb. Google can help with those options.
Oh, you said "avoid" not "debug"... hmm, the best way to avoid is do not hack the kernel :^)
Seriously, when doing kernel-level programming you have to be extra careful. Add a main() to the code that stress-tests your routines in usermode before adding to the running kernel. And read over your code, especially after you've isolated the bug to a particular section. I once found an infinite loop in LynxOS's terminal driver when some ANSI art hung the operating system. Some junior programmer, apparently, had written that part, parsing the escape sequence options as text rather than numbers. The code was so bad, I got disgusted trying to locate the exact error that forced the loop, and just rewrote most of the driver. And tested it in usermode before adding to the kernel.
You could try to enable the NMI watchdog.

How can i prevent gdb from attaching to an exe?

I'd like to prevent would-be hackers from attaching to my binary on Linux systems. I see that ptrace DENY_ATTACH can be used on OSX. Is there such option that can be used on linux? How about on Windows?
Thanks for any info!
Such a system call requires kernel support. Even if it existed in Linux, it would be fairly easy to disable by compiling your own kernel.
In linux, ptrace returns -1 if the process is being ptraced.
So, one solution would be, inside your program, try to attach to your process, and if you get a -1, you will know that the program is being ptraced.

Resources