I am writing my first OpenCL kernels on an Ubuntu machine with an NVIDIA card. Once in a while, the application totally freezes the whole computer. The mouse does not move, and the only way to reboot is by force-pressing the power button.
I've realized that the reason for the freezes is that I accidentally read past the last index of a global, read-only float array. While this is something I don't intend to do often, it might still happen in the future.
My question is - is there any way to prevent the computer from completely shutting down if this happens again? I know that, for example, Windows can shut down bad GLSL kernels and recover with a graphics driver restart. Is something similar possible here?
You may not be able to completely recover but you can recover better using SysRq (sometimes called System Request or Magic SysRq). By executing a specific key combination you can have Linux reboot in some what of a sane way (killing processes and unmounting filesystems). This key sequence is described in detail at http://en.wikipedia.org/wiki/Magic_SysRq_key so I won't repeat it here.
In some cases you might be able to still SSH to the device. If this is your case you might be in more luck. If you can SSH there are a number of other options you can try such as: unloading/reloading the crashed module, restarting the xserver, or at least rebooting the normal way.
Although I'm not an expert on "HURD" I believe it was designed to handle this type of condition better. The only other solution I can think of is using two graphics cards one for X and one for OpenCL. Dependidng on what you are doing you might have to passthrough the NVIDIA to a VM in order to completely isolate it off your host.
Related
For user mode application, incorrect page access doesn't create a lot of trouble other than the application crash and the application crash can be done gracefully by exception handling. Why can't we do the same for kernel crash. So when a kernel module tries to access some invalid address, there is a page fault and the kernel crash. Why it can't be handled gracefully like unloading the faulty module.
More specifically I'm interested to know if it is completely impossible or possible. I am not inclined to know the difficulties it might pose in using the system. I understand a driver crash will result unusable device and I'm okay with that. The only thing is whether it is possible to gracefully unload a faulty driver.
As other answer explains very well why it's not feasible to recover from the kernel crashes, I'll try to tell something else.
There is a lot of research in this area, most notably from prof. Andy Tanenbaum with his MINIX. While the kernel crash is still fatal for the MINIX, MINIX kernel is very simple (micro-kernel) narrowing the space for bugs and inside it most other stuff (including drivers) is running as a user-mode process. So, in case of network driver failure, as they are running in the separate address space, all kernel needs to do is to attempt to restart the driver.
Of course, there are areas where you can't recover (or still can't recover), like in case of the file system crash (see the recent discussion here).
There are several good papers on this topic such as http://pages.cs.wisc.edu/~swami/papers/thesis.pdf and I would highly recommend watching Tanenbaum's videos such this one (title is "MINIX 3: A Reliable and Secure Operating System" in case it ever goes offline).
I think this addresses your comment:
We should be able to unload the faulty module. Why can't we? That is my question. Is it a design choice for security or its not possible at all. If it is a design choice, what factors forced us to make that choice
You could live without screen if graphics driver module crashes. However, we can't unload the faulty module and continue because if it crashed and it runs in the same address space as kernel, you don't know if it poisoned the kernel memory - security is the prime factor here.
That's kind of like saying "if you wrap all your Java code in a try/catch block, you've eliminated all bugs!"
There are a number of "errors" that are caught, e.g. kalloc returns NULL if it's out of memory, USB code returns errors if there's no USB, etc. But there's no try/catch for the entire operating system, because not all bugs can be repaired.
As pointed out, what happens if your filesystem module crashes? Keep running without files? How about your ethernet driver? Now your box is cut off from the internet and you can't even ssh into it anymore, but doesn't even have the decency to reboot.
So even though it may be possible for the kernel to not "crash" when a module crashes, the state of the kernel could be arbitrarily broken. The kernel could stay alive without a screen, filesystem or internet connection, but what kind of existence is that?
The kernel modules and the kernel itself share the same address space. There is simply no protection if a modules starts to misbehave and overwrite memory from another subsystem.
So, when a driver crashes, it may or may not stay local to that driver. If you are lucky, you still have a somewhat functional kernel and can continue to work.
That doesn't happen with userspace because the address space for each process is separate and so it is possible to catch erroneous memory access and stop the process (this is a SEGFAULT).
I'm looking for a way to determine the local input (mouse/keyboard) idle time,
remotely through SSH, without having root access or access to the currently logged on user Xauthority.
I know the following solution which works if you have access to the X server:
detecting keyboard, mouse activity in linux
But is it possible without having to connect to the X server?
Is there another way? E.g. indirectly via CPU or memory usage of certain processes?
Any ideas welcome.
Using w and /dev solutions will only get you so far, since it could be that the user is around, but has not typed anything in the shell - for example, he/she could be playing some game. A better approach would be to poll /proc/interrupts. The local interrupts for the mouse and keyboard are often under "i8042" (though in some rare cases it might be different). You might want to try: "grep i8042 /proc/interrupts". This will yield IRQ 1 (keyboard) and IRQ 12 (usually, the mouse). You can get the values, store them, and then poll occasionally (no callback, alas), to get the counts. If the numbers changed, interrupts occurred - meaning keyboard (IRQ 1) or mouse (IRQ 12) were touched/pressed etc. Key presses generally generate two interrupts (key down, key up). Mouse movement is more erratic.
This has several advantages:
1) If the user so much as touches the mouse, or presses a key - you know
2) You can do so programmatically (i.e. fopen() /proc/interrupts , or (alternatively) /proc/stat, and get the "intr" line) and fread() the relevant lines
3) You don't even need to be root for this.
As an unprivileged, distinct user, I'd try to do this by monitoring for new processes spawned with the target user ID. If you see a new process, chances are it means the user caused it to be spawned, unless its parent is cron (or one of a handful of similar supervisors, you'll have to try it on your system and filter out the few false alarms you get at first).
This won't give you a very precise idle time, but perhaps a maximum bound. If you want something more precise, you probably ought to have privileges to do it.
The commands w and ls -l /dev/pts are the best ones I can think of. Between them, you can get a pretty good view of idleness on the linux console, ssh, and xterm sessions.
If the user is just sitting on a local thing playing with firefox or something, this won't catch it, but I'm not sure you can get that without access to X or being root. Looking for cpu usage isn't reliable because a lot of apps use a little cpu even when idle, or virtually zero when used, and watching /proc/x_server_pid/* doesn't help either, since X can still have activity even when the user is idle.
Looking for new processes like the other answer said is the best I can think of if /dev/pts doesn't help and you can't actually look at the input.
I'm currently attempting to detect when a user has fast-user-switched to another user on the Linux platform (specifically, Fedora 14-16, RedHat 4.7-6.x, CentOS 4-6, OpenSuse 10-11). I've been looking for something similar to the WTSRegisterSessionNotification() function that is available on Windows, but all I've come across is a bunch of references to bugs in the Wine software.
Has anyone else run across this snag? There seems to be a ton of resources on how to do this on Windows and Mac OS X (which is fine), but on Linux there seems to be nothing...
EDIT:
Apparently, on newer systems (at least Fedora 16) this may appear to be a viable option. I wonder if it has a DBus interface...More to come soon!
First of all, I need to tell you I'm not an expert in this area, but I have enough knowledge to give you pointers to places where you may go and learn more. So I may be wrong in some ways.
My guess is:
this is not easy
for most methods you may implement there are probably many ways to trick them into believing something that's not true, which may lead to security problems
your method may depend on the:
chosen Linux Distribution
version of the Distribution
Desktop Environment
Display Manager
As far as I know (and I may be wrong if something changed during the last years), fast user switching is implemented by launching another X server on another VT. So one way would be to detect if there are multiple X servers running.
But there are many cases where there multiple X servers running and it's not because of fast user switching. Examples: Multiseat or even simple Xephyr logins. With Xephyr and XDMCP, you may even have the same user logged in twice in a non-fast-user-switching case.
I started googling about this and found this old web page:
http://fedoraproject.org/wiki/Desktop/FastUserSwitching
If things haven't changed since then, you should study ConsoleKit and PolicyKit (and also DeviceKit and maybe Systemd today) and their DBus APIs.
There are also the commands ck-list-sessions and ck-launch-session. But I believe you can fool these commands easily: try to ck-launch-session xterm and then ck-list-session.
Why exactly are you trying to detect fast user switching? What's your ultimate goal? Maybe you can solve your problem without trying to detect fast user switch...
Well it appears that the most useful way of getting at this information is to use the ConsoleKit DBus interface.
The following procedure outlines how to enumerate the sessions and determine if they are active or not:
1.) Enumerate the sessions using the following:
Bus: org.freedesktop.ConsoleKit
Path: /org/freedesktop/ConsoleKit/Manager
Method: org.freedesktop.ConsoleKit.Manager.GetSessions
What is returned is an array of object paths that export the Session interface. These, in turn, can be queried using DBus to get their appropriate properties. For example, I used dbus-send to communicate with ConsoleKit to enumerate the sessions in my system:
dbus-send --system --print-reply --dest=org.freedesktop.ConsoleKit /org/freedesktop/ConsoleKit/Manager org.freedesktop.ConsoleKit.Manager.GetSessions
And what I received in return was the following:
method return sender=:1.15 -> dest=:1.205 reply_serial=2
array [
object path "/org/freedesktop/ConsoleKit/Session2"
]
2.) Using the returned object path(s), I can query them for their attributes, such as if they are active or not using the following:
Bus: org.freedesktop.ConsoleKit
Path: /org/freedesktop/ConsoleKit/Session2
Method: org.freedesktop.ConsoleKit.Session.IsActive
Depending on the method, I can query what I need from the session(s)! Using the ConsoleKit interface I can also retrieve the identifier for the current session, so I can always query it to see if it's active when I need to. Just for fun, here's the output of the following command:
dbus-send --system --print-reply --dest=org.freedesktop.ConsoleKit /org/freedesktop/ConsoleKit/Session2 org.freedesktop.ConsoleKit.Session.IsActive
method return sender=:1.15 -> dest=:1.206 reply_serial=2
boolean true
Neat.
You have to do it by polling to be sure of working on all machines (you obviously don't have to have DBus running to do user switching!).
Solaris, HP-UX, and others, do not do user switching on the console.
Platforms to support: linux, FreeBSD, AIX. Linux/BSD use virtual terminals; AIX uses /dev/lft0 if you're interested.
Suppose you want to reliably and securely run a application on the console, and restart it on the new active X server when the console switches to another VT. The problems are that you may or may not have a desktop environment running (some of us use twm!). The session may not have been started via a login manager (you could do Ctrl-Alt-F2 on linux, login, and run startx quite happily). The system might not even have xdm/gdm/similar installed.
The dumb solution is the only reliable one: every few seconds, query what the active virtual terminal is (VT_GETSTATE on linux, VT_GETACTIVE on BSD). If it's changed, you know a switch has happened. If you switched to a non-graphical session (eg with Ctrl-Alt-F1) there won't be an X server active.
Otherwise, you have to hunt hard to find which display number is active. For example, you might see two X servers in ps, with display numbers :1 and :2. Which of those is on VT7 though? The final piece of the puzzle, mapping VT numbers to display numbers, is the hardest. This question is answered in this duplicate question, "Which virtual terminal is a given X process running on?".
I've got a WinCE device powered over ethernet (PoE) and I want to prevent File-System corruption following a potential power loss, e.g. user pulling the plug.
As a side note, I'm already using TexFAT which is supposed to prevent FS corruptions. While the later certainly does help reducing FS corruptions (over using plain old FAT), it doesn't entirely prevent some to still occur from time to time... So, I'm considering using a small rechargeable backup battery that would give WinCE enough time to cleanly shut down. Now, I can't find any info on the shutdown process: how to trigger it, how long it takes, and so on... MSDN is pretty quiet on this topic. Any idea?
The powerdown sequence is totally platform dependent.
The following answer is relevant to Windows CE 6. It may be different for previous versions of CE.
If you include the power manager component in your system, then the sequence is plus minus this:
Send go to D4 to all the drivers that are powermanageable and that reported they support this state. Otherwise, the driver gets the lowest powerstate it supports.
XXX_PowerDown is called, but it is not commonly used in Windows CE 6.
In between the registry is flushed in case you have a Hive Based registry and you enabled the registry flush thread. You should disable this in a fragile system such as yours
OEMPowerOff
device down
Just found a post by Bruce Eitman on what happens when Suspending. He puts it better than I do.
The Suspend sequence is what you'd do before loosing power.
Is it possible to 'hibernate' a process in linux?
Just like 'hibernate' in laptop, I would to write all the memory used by a process to disk, free up the RAM. And then later on, I can 'resume the process', i.e, reading all the data from memory and put it back to RAM and I can continue with my process?
I used to maintain CryoPID, which is a program that does exactly what you are talking about. It writes the contents of a program's address space, VDSO, file descriptor references and states to a file that can later be reconstructed. CryoPID started when there were no usable hooks in Linux itself and worked entirely from userspace (actually, it still does work, depending on your distro / kernel / security settings).
Problems were (indeed) sockets, pending RT signals, numerous X11 issues, the glibc caching getpid() implementation amongst many others. Randomization (especially VDSO) turned out to be insurmountable for the few of us working on it after Bernard walked away from it. However, it was fun and became the topic of several masters thesis.
If you are just contemplating a program that can save its running state and re-start directly into that state, its far .. far .. easier to just save that information from within the program itself, perhaps when servicing a signal.
I'd like to put a status update here, as of 2014.
The accepted answer suggests CryoPID as a tool to perform Checkpoint/Restore, but I found the project to be unmantained and impossible to compile with recent kernels.
Now, I found two actively mantained projects providing the application checkpointing feature.
The first, the one I suggest 'cause I have better luck running it, is CRIU
that performs checkpoint/restore mainly in userspace, and requires the kernel option CONFIG_CHECKPOINT_RESTORE enabled to work.
Checkpoint/Restore In Userspace, or CRIU (pronounced kree-oo, IPA: /krɪʊ/, Russian: криу), is a software tool for Linux operating system. Using this tool, you can freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. You can then use the files to restore and run the application from the point it was frozen at. The distinctive feature of the CRIU project is that it is mainly implemented in user space.
The latter is DMTCP; quoting from their main page:
DMTCP (Distributed MultiThreaded Checkpointing) is a tool to transparently checkpoint the state of multiple simultaneous applications, including multi-threaded and distributed applications. It operates directly on the user binary executable, without any Linux kernel modules or other kernel modifications.
There is also a nice Wikipedia page on the argument: Application_checkpointing
The answers mentioning ctrl-z are really talking about stopping the process with a signal, in this case SIGTSTP. You can issue a stop signal with kill:
kill -STOP <pid>
That will suspend execution of the process. It won't immediately free the memory used by it, but as memory is required for other processes the memory used by the stopped process will be gradually swapped out.
When you want to wake it up again, use
kill -CONT <pid>
The more complicated solutions, like CryoPID, are really only needed if you want the stopped process to be able to survive a system shutdown/restart - it doesn't sound like you need that.
Linux Kernel has now partially implemented the checkpoint/restart futures:https://ckpt.wiki.kernel.org/, the status is here.
Some useful information are in the lwn(linux weekly net):
http://lwn.net/Articles/375855/ http://lwn.net/Articles/412749/ ......
So the answer is "YES"
The issue is restoring the streams - files and sockets - that the program has open.
When your whole OS hibernates, the local files and such can obviously be restored. Network connections don't, but then the code that accesses the internet is typically more error checking and such and survives the error conditions (or ought to).
If you did per-program hibernation (without application support), how would you handle open files? What if another process accesses those files in the interim? etc?
Maintaining state when the program is not loaded is going to be difficult.
Simply suspending the threads and letting it get swapped to disk would have much the same effect?
Or run the program in a virtual machine and let the VM handle suspension.
Short answer is "yes, but not always reliably". Check out CryoPID:
http://cryopid.berlios.de/
Open files will indeed be the most common problem. CryoPID states explicitly:
Open files and offsets are restored.
Temporary files that have been
unlinked and are not accessible on the
filesystem are always saved in the
image. Other files that do not exist
on resume are not yet restored.
Support for saving file contents for
such situations is planned.
The same issues will also affect TCP connections, though CryoPID supports tcpcp for connection resuming.
I extended Cryopid producing a package called Cryopid2 available from SourceForge. This can
migrate a process as well as hibernating it (along with any open files and sockets - data
in sockets/pipes is sucked into the process on hibernation and spat back into these when
process is restarted).
The reason I have not been active with this project is I am not a kernel developer - both
this (and/or the original cryopid) need to get someone on board who can get them running
with the lastest kernels (e.g. Linux 3.x).
The Cryopid method does work - and is probably the best solution to general purpose process
hibernation/migration in Linux I have come across.
The short answer is "yes." You might start by looking at this for some ideas: ELF executable reconstruction from a core image (http://vx.netlux.org/lib/vsc03.html)
As others have noted, it's difficult for the OS to provide this functionality, because the application needs to have some error checking builtin to handle broken streams.
However, on a side note, some programming languages and tools that use virtual machines explicitly support this functionality, such as the Self programming language.
This is sort of the ultimate goal of clustered operating system. Mathew Dillon puts a lot of effort to implement something like this in his Dragonfly BSD project.
adding another workaround: you can use virtualbox. run your applications in a regular virtual machine and simply "save the machine state" whenever you want.
I know this is not an answer, but I thought it could be useful when there are no real options.
if for any reason you don't like virtualbox, vmware and Qemu are as good.
Ctrl-Z increases the chances the process's pages will be swapped, but it doesn't free the process's resources completely. The problem with freeing a process's resources completely is that things like file handles, sockets are kernel resources the process gets to use, but doesn't know how to persist on its own. So Ctrl-Z is as good as it gets.
There was some research on checkpoint/restore for Linux back in 2.2 and 2.4 days, but it never made it past prototype. It is possible (with the caveats described in the other answers) for certain values of possible - I you can write a kernel module to do it, it is possible. But for the common value of possible (can I do it from the shell on a commercial Linux distribution), it is not yet possible.
There's ctrl+z in linux, but i'm not sure it offers the features you specified. I suspect you asked this question since it doesn't