I'm testing some software+driver for it on linux, and the driver uses the get_user_pages() in its internal functions.
At some point my driver receives ERESTARTSYS error (-512) from the get_user_pages(), and according to the kernel code it happens because "If we have a pending SIGKILL, don't keep faulting pages and potentially allocating memory." - it's a comment from memory.c kernel file.
How can I see who sent this SIGKILL and why? I was trying to look in /var/log/kern.log file but couldn't see anything there about any signal.
I don't believe you can for SIGKILL (others, yes), unless you are willing to patch the kernel to give you the signal information. In which case, you can inspect the si_code and si_pid values, per the docs: http://pubs.opengroup.org/onlinepubs/009696699/basedefs/signal.h.html
Eg, if your signal info is in siptr:
if ((siptr)->si_code <= 0) {
printk(KERN_DEBUG "kill sent by process %u", (siptr)->si_pid);
}
The if check isn't strictly necessary: it restricts the printk() to those signals raised by a kill(). If the kernel raised the signal, si_code would be greater than 0.
I had exactly same problem.
But instead of in get_user_pages() i would get -ERESTARTSYS from sock_sendmsg().
To debug this issue what i did was add a logging message in
linux-3.2/kernel/signal.c: __send_signal().
And to avoid message filling up in kernel logs.
I would strncmp(t->comm, "myprogramname") And then log t->comm, t->pid, current->comm and current->pid.
Also i realized it is not only SIGKILL but any other signal pending too,
Then the call will return -ERESTARTSYS.
So the my next step was to find out who is giving my program a signal.
And add handlers for all signals (except SIGKILL, which i cant really handle).
Lucky for me it was not SIGKILL.
Adding handler may not help your case, but logging will identify who is sending and the reason.
Handling may help others who have similar problem.
Related
I want to detect the QProcess I launched was terminated externally by either SIGKILL or SIGTERM. This is important in distinguishing between a crash (bug) and an external interference when I've written the process I'm launching myself.
I've tried registering watched processes through a slot connected to QProcess::started and setting a SIGCHLD handler (using sigaction) to capture the process status using waitpid. The issue is that waitpid clears the internal kernel data structure and even if I properly chain my handler to the one on the QProcess implementation, the latter isn't able to get the child status as any next calls to waitpid for that pid fail. Setting the process state to QProcess::ProcessState::NotRunning through QProcess::setProcessState avoids hanging on calls to waitForFinished in general, but there are corner cases which I couldn't manage to fix yet.
I was wondering if there isn't a better way to do this, other than modifying Qt's source code to store the status information somewhere.
Note: I know that crashes also terminate with a signal, which is SIGABRT. The main issue here is that a SIGKILL might tell me that the out of memory killer in Linux was the responsible for the process termination.
This is my code, friend.
QProcess* pExe = new QProcess(this);
connect(pExe, SIGNAL(finished(int, QProcess::ExitStatus)), this, SLOT(onOSRExit(int, QProcess::ExitStatus)));
pExe->setWorkingDirectory(QCoreApplication::applicationDirPath());
pExe->start("some.exe");
....
void CXXXXX::onOSRExit(int exitCode, QProcess::ExitStatus exitStatus)
{
}
The solution altering Qt's code under Qt 4, basically involves a simple modification to QProcessPrivate::waitForDeadChild():
if (qt_safe_waitpid(pid_t(pid), &exitStatus, WNOHANG) > 0) {
processManager()->remove(q);
crashed = !WIFEXITED(exitStatus);
- exitCode = WEXITSTATUS(exitStatus);
+ exitCode = crashed ? WTERMSIG(exitStatus) : WEXITSTATUS(exitStatus);
The signal will then be available on QProcess::exitCode() after QProcess::finished() has been emitted.
Note: Qt5 is using Thiago Macieira's forkfd, so this solution won't work.
I'm logging system calls in OS X using a kext, like so:
int hook_read(struct proc *p, struct read_args *u, user_ssize_t *r) {
/* get som params here... */
printf("[IDEN] SYS_read called, %s, %d, %d, %d.\n", params);
return read(p, u, r);
}
This logs to system.log. The issue now is that if the printf load is high (many system calls are called), the output in system.log is often malformed:
ID]SYS_readEN] callparam, 123, ed 123, 123 for example. The string is scrambled. If I use kprintf, which prints to a serial port, malformed logs never occur.
Any idea's what's causing this behaviour are much appreciated!
printf/IOLog uses a buffer for sending messages to userspace, where they're then logged to the system log. If you output a lot of messages, this buffer can get full before the logging daemon has a chance to clear it, and your messages will be truncated and start running into each other. (printf/IOLog will return as soon as the message has been submitted to the buffer, and will not wait for the daemon to receive it.)
Additionally, I'm not sure if kernel printf/IOLog is in any way thread-safe, so it might also be the case that calling it from multiple threads simultaneously causes race conditions. I'd have to check the source to be sure.
kprintf is fully synchronous and will block until your message has been transmitted on the serial port or submitted via firewire. It's also thread-safe. In the firewire case, the kprintf output is also buffered, so if you output a lot of messages, this can fill up too and cause truncation. The fwkpfv command lets you specify the buffer size using the --buffer argument though, so you can counteract that. The downside of kprintf being synchronous is that it can slow the system down considerably with a high volume of messages.
How does one find reliably whether a process received a signal due to its own misbehavior or was sent the same by another process? Basically, how does one determine whether si_pid field is valid or not.
If si_pid in the siginfo_t structure matches getpid() then the process signaled itself. Otherwise, another process did. Since process IDs are unique at any point in time, a PID you have now could not possibly have sent you the signal at a time when it had your PID (because then it would have signaled itself and not you).
Edit:
As you have discovered, the si_pid field is not always set; sometimes it contains garbage values. The first thing to check is that you have passed SA_SIGINFO in the sa_flags field of your struct sigaction when registering your handler. Without this, your handler may not receive a siginfo_t at all.
Once that's done, there are rules for when si_pid is set, described here: https://www.mkssoftware.com/docs/man5/siginfo_t.5.asp#Signal_Codes
In brief: si_pid should be set if si_code is one of:
SI_USER - includes calls to kill()
SI_QUEUE
SI_TIMER
SI_ASYNCIO
SI_MESGQ
It is also set whenever si_signo is SIGCHLD.
Question:
When a process is killed, is this information recorded anywhere (i.e., in kernel), such as syslog (or can be configured to be recorded syslog.conf)
Is the information of the killer's PID, time and date when killed and reason
update - you have all giving me some insight, thank you very much|
If your Linux kernel is compiled with the process accounting (CONFIG_BSD_PROCESS_ACT) option enabled, you can start recording process accounting info using the accton(8) command and use sa(8) to access the recorded info. The recorded information includes the 32 bit exit code which includes the signal number.
(This stuff is not widely known / used these days, but I still remember it from the days of 4.x Bsd on VAXes ...)
Amended:
In short, the OS kernel does not care if the process is killed. That is dependant on whether the process logs anything. All the kernel cares about at this stage is reclaiming memory. But read on, on how to catch it and log it...
As per caf and Stephen C's mention on their comments...
If you are running BSD accounting daemon module in the kernel, everything gets logged. Thanks to Stephen C for pointing this out! I did not realize that functionality as I have this switched off/disabled.
In my hindsight, as per caf's comment - the two signals that cannot be caught are SIGKILL and SIGSTOP, and also the fact that I mentioned atexit, and I described in the code, that should have been exit(0);..ooops Thanks caf!
Original
The best way to catch the kill signal is you need to use a signal handler to handle a few signals , not just SIGKILL on its own will suffice, SIGABRT (abort), SIGQUIT (terminal program quit), SIGSTOP and SIGHUP (hangup). Those signals together is what would catch the command kill on the command line. The signal handler can then log the information stored in /var/log/messages (environment dependant or Linux distribution dependant). For further reference, see here.
Also, see here for an example of how to use a signal handler using the sigaction function.
Also it would be a good idea to adopt the usage of atexit function, then when the code exits at runtime, the runtime will execute the last function before returning back to the command line. Reference for atexit is here.
When the C function exit is used, and executed, the atexit function will execute the function pointer where applied as in the example below. - Thanks caf for this!
An example usage of atexit as shown:
#include <stdlib.h>
int main(int argc, char **argv){
atexit(myexitfunc); /* Beginning, immediately right after declaration(s) */
/* Rest of code */
return 0;
exit(0);
}
int myexitfunc(void){
fprintf(stdout, "Goodbye cruel world...\n");
}
Hope this helps,
Best regards,
Tom.
I don't know of any logging of signals sent to processes, unless the OOM killer is doing it.
If you're writing your own program you can catch the kill signal and write to a logfile before actually dying. This doesn't work with kill -9 though, just the normal kill.
You can see some details over thisaway.
If you use sudo, it will be logged. Other than that, the killed process can log some information (unless it's being terminated with extreme prejudice). You could even hack the kernel to log signals.
As for recording the reason a process was killed, I've yet to see a psychic program.
Kernel hacking is not for the weak of heart, but hella fun. You'd need to patch the signal dispatch routines to log information using printk(9) when kill(3), sigsend(2) or the like is called. Read "The Linux Signals Handling Model" for more information on how signals are handled.
If the process is getting it via kill(2), then unless the process is already logging the only external trace would be a kernel mod. It's pretty simple; just do a printk(), it's like printf(). Find the output in dmesg.
If the process is getting it via /bin/kill, then it would be a relatively easy matter to install a wrapper executable that did logging. But this (signal delivery via /bin/kill) is unlikely because kill is also a bash built-in.
By the way, if a process is killed with a signal is announced by the kernel to the parent process via de wait(2) system call. The value returned by this call is the exit status of the child (the lower byte) and some signal related info in the upper byte in case this process has been killed. See wait(2) for more information.
When you pipe two process and kill the one at the "output" of the pipe, the first process used to receive the "Broken Pipe" signal, which usually terminated it aswell. E.g. running
$> do_something_intensive | less
and then exiting less used to return you immediately to a responsive shell, on a SuSE8 or former releases.
when i'm trying that today, do_something_intensive is obviously still running until i kill it manually. It seems that something has changed (glib ? shell ?) that makes program ignore "broken pipes" ...
Anyone of you has hints on this ? how to restore the former behaviour ? why it has been changed (or why it always existed multiple semantics) ?
edit : further tests (using strace) reveal that "SIGPIPE" is generated, but that the program is not interrupted. A simple
#include <stdio.h>
int main()
{
while(1) printf("dumb test\n");
exit(0);
}
will go on with an endless
--- SIGPIPE (Broken pipe) # 0 (0) ---
write(1, "dumb test\ndumb test\ndumb test\ndu"..., 1024) = -1 EPIPE (Broken pipe)
when less is killed. I could for sure program a signal handler in my program and ensure it terminates, but i'm more looking for some environment variable or a shell option that would force programs to terminate on SIGPIPE
edit again: it seems to be a tcsh-specific issue (bash handles it properly) and terminal-dependent (Eterm 0.9.4)
Well, if there is an attempt to write to a pipe after the reader has gone away, a SIGPIPE signal gets generated. The application has the ability to catch this signal, but if it doesn't, the process is killed.
The SIGPIPE won't be generated until the calling process attempts to write, so if there's no more output, it won't be generated.
Has "do something intensive" changed at all?
As Daniel has mentioned SIGPIPE is not a magic "your pipe went away" signal but rather a "nice try, you can no longer read/write that pipe" signal.
If you have control of "do something intensive" you could change it to write out some "progress indicator" output as it spins. This would raise the SIGPIPE in a timely fashion.
Thanks for your advices, the solution is getting closer...
According to the manpage of tcsh, "non-login shells inherit the terminate behavior from their parents. Other signals have the values which the shell inherited from its parent."
Which suggest my terminal is actually the root of the problem ... if it ignored SIGPIPE, the shell itself will ignore SIGPIPE as well ...
edit: i have the definitive confirmation that the problem only arise with Eterm+tcsh and found a suspiciously missing signal(SIGPIPE,SIG_DFL) in Eterm source code. I think that close the case.