I am analyzing a set of buggy programs that under some test they may terminate with segfault. The segfault event is logged in /var/log/syslog.
For example the following snippet returns Segmentation fault and it is logged.
#!/bin/bash
./test
My question is how to suppress the segfault such that it does NOT appear in the system log. I tried trap to capture the signal in the following script:
#!/bin/bash
set -bm
trap "echo 'something happened'" {1..64}
./test
It returns:
Segmentation fault
something happened
So, it does traps the segfault but the segfault is still logged.
kernel: [81615.373989] test[319]: segfault at 0 ip 00007f6b9436d614
sp 00007ffe33fb77f8 error 6 in libc-2.19.so[7f6b942e1000+1bb000]
You can try to change ./test to the following line:
. ./test
This will execute ./test in the same shell.
We can suppress the log message system-wide with e. g.
echo 0 >/proc/sys/debug/exception-trace
- see also
Making the Linux kernel shut up about segfaulting user programs
Is there a way to temporarily disable segfault messages in dmesg?
We can suppress the log message for a single process if we run it under ptrace() control, as in a debugger. This program does that:
exe.c
#include <sys/wait.h>
#include <sys/ptrace.h>
main(int argc, char *args[])
{
pid_t pid;
if (*++args)
if (pid = fork())
{
int status;
while (wait(&status) > 0)
{
if (!WIFSTOPPED(status))
return WIFSIGNALED(status) ? 128+WTERMSIG(status)
: WEXITSTATUS(status);
int signal = WSTOPSIG(status);
if (signal == SIGTRAP) signal = 0;
ptrace(PTRACE_CONT, pid, 0, signal);
}
perror("wait");
}
else
{
ptrace(PTRACE_TRACEME, 0, 0, 0);
execvp(*args, args);
perror(*args);
}
return 1;
}
It is called with the buggy program as its argument, in your case
exe ./test
- then the exit status of exe normally is the exit status of test, but if test was terminated by signal n (11 for Segmentation fault), it is 128+n.
After I wrote this, I realized that we can also use strace for the purpose, e. g.
strace -enone ./test
Related
I would like to put a prefix, like "GDB> ", on every output from gdb to distinguish it from the output of my program.
Is that possible?
An example:
test.c
#include <stdio.h>
int main(int argc, char **argv)
{
char *p = NULL;
printf("Test begins...\n");
p[0] = '\0'; // Forcing a segmentation fault.
printf("Test finished.\n");
return 0;
}
Debugging with this command line:
$ gdb -q -ex "set confirm off" -ex run -ex quit --args ./test
The output is:
Reading symbols from ./test...
Starting program: /home/me/tst/test
Test begins...
Program received signal SIGSEGV, Segmentation fault.
0x000000000040113b in main ()
What I have in mind is set the output to something like that:
GDB> Reading symbols from ./test...
GDB> Starting program: /home/me/tst/test
Test begins...
GDB>
GDB> Program received signal SIGSEGV, Segmentation fault.
GDB> 0x000000000040113b in main ()
I'm thinking about some tool that can pause the program at start.
For example, my_bin starts running at once.
$ ./my_bin
With this tool
$ magic_tool ./my_bin
my_bin will start. I can get the PID. Then I can start the actual running later.
I've just tested my suggestion in the comments and it worked! This is the code in my magic_tool.c:
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
int main (int argc, char *argv[])
{
pid_t pid;
printf("Executing %s to wrap %s.\n", argv[0], argv[1]);
pid = fork();
if (pid == -1)
return -1;
if (pid == 0) {
raise(SIGSTOP);
execl(argv[1], "", NULL);
} else {
printf("PID == %d\n", pid);
}
return 0;
}
I wrote another test program target.c:
#include <stdio.h>
int main ()
{
puts("It works!\n");
return 0;
}
Running ./magic_tool ./target printed a PID and returned to shell. Only after running kill -SIGCONT <printed_pid> was It works! printed. You'll probably want to have PID saved somewhere else and also perform some checks in the magic_tool, but I think this is nonetheless a good proof of concept.
EDIT:
I was playing around with this a bit more and for some reason it didn't always work (see why below). The solution is simple - just follow a proper fork off and die pattern a bit more closely in magic_tool.c:
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
int main (int argc, char *argv[])
{
pid_t pid;
printf("Executing %s to wrap %s.\n", argv[0], argv[1]);
pid = fork();
if (pid == -1)
return -1;
if (pid == 0) {
setsid();
pid = fork();
if (pid == -1)
return -1;
if (pid == 0) {
raise(SIGSTOP);
if (execl(argv[1], "", NULL))
return -1;
}
printf("PID == %d\n", pid);
}
return 0;
}
I found an explanation in this answer:
When you start the root process from your shell, it is a process group leader, and its descendants are members of that group. When that leader terminates, the process group is orphaned. When the system detects a newly-orphaned process group in which any member is stopped, then every member of the process group is sent a SIGHUP followed by a SIGCONT.
So, some of your descendant processes are still stopped when the leader terminates, and thus everyone receives a SIGHUP followed by a SIGCONT, which for practical purposes mean they die of SIGHUP.
Exactly which descendants are still stopped (or even just merrily advancing toward exit()) is a timing race.
The answer also links to IEEE Std 1003.1-2017 _Exit entry which contains more details on the matter.
This is mostly a very similar idea as #gst, but done entirely in the shell, you can spawn a subshell (this forks and create a new pid) and have the subshell send itself SIGSTOP signal, when the subshell receives a SIGCONT signal and resumes, the subshell exec the intended program (this replaces the subshell with the intended program without creating a new pid). So that the main shell can continue doing stuff, the subshell should run on background with &.
In a nutshell:
(kill -STOP $BASHPID && exec ./my_bin) &
subpid=$! # get the pid of above subshell
... do something else ...
kill -CONT $subpid # resume
Another idea that wouldn't suffer from race condition between the main process sending SIGCONT and the subshell SIGSTOP-ing itself is to use a file descriptor to implement the wait instead:
exec {PIPEFD}<> <(:) # set PIPEFD to the file descriptor of an anonymous pipe
(read -u $PIPEFD && exec ./my_bin) &
subpid=$! # get the pid of above subshell
... do something else ...
echo >&$PIPEFD # resume
I have been playing with ftrace recently to monitor some behavior characteristics of my system. I've been handling switching the trace on/off via a small script. After running the script, my system would crash and reboot itself. Initially, I believed that there might be an error with the script itself, but I have since determined that the crash and reboot is a result of echoing some tracer to /sys/kernel/debug/tracing/current_tracer when current_tracer is set to function_graph.
That is, the following sequence of commands will produce the crash/reboot:
echo "function_graph" > /sys/kernel/debug/tracing/current_tracer
echo "function" > /sys/kernel/debug/tracing/current_tracer
Durning the reboot after the crash caused by the above echo statements, I see a lot of output that reads:
clearing orphaned inode <inode>
I tried to reproduce this problem by replacing the current_tracer value from function_graph to something else in a C program:
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
int openCurrentTracer()
{
int fd = open("/sys/kernel/debug/tracing/current_tracer", O_WRONLY);
if(fd < 0)
exit(1);
return fd;
}
int writeTracer(int fd, char* tracer)
{
if(write(fd, tracer, strlen(tracer)) != strlen(tracer)) {
printf("Failure writing %s\n", tracer);
return 0;
}
return 1;
}
int main(int argc, char* argv[])
{
int fd = openCurrentTracer();
char* blockTracer = "blk";
if(!writeTracer(fd, blockTracer))
return 1;
close(fd);
fd = openCurrentTracer();
char* graphTracer = "function_graph";
if(!writeTracer(fd, graphTracer))
return 1;
close(fd);
printf("Preparing to fail!\n");
fd = openCurrentTracer();
if(!writeTracer(fd, blockTracer))
return 1;
close(fd);
return 0;
}
Oddly enough, the C program does not crash my system.
I originally encountered this problem while using Ubuntu (Unity environment) 16.04 LTS and confirmed it to be an issue on the 4.4.0 and 4.5.5 kernels. I have also tested this issue on a machine running Ubuntu (Mate environment) 15.10, on the 4.2.0 and 4.5.5 kernels, but was unable to reproduce the issue. This has only confused me further.
Can anyone give me insight on what is happening? Specifically, why would I be able to write() but not echo to /sys/kernel/debug/tracing/current_tracer?
Update
As vielmetti pointed out, others have had a similar issue (as seen here).
The ftrace_disable_ftrace_graph_caller() modifies jmp instruction at
ftrace_graph_call assuming it's a 5 bytes near jmp (e9 ).
However it's a short jmp consisting of 2 bytes only (eb ). And
ftrace_stub() is located just below the ftrace_graph_caller so
modification above breaks the instruction resulting in kernel oops on
the ftrace_stub() with the invalid opcode like below:
The patch (shown below) solved the echo issue, but I still do not understand why echo was breaking previously when write() was not.
diff --git a/arch/x86/kernel/mcount_64.S b/arch/x86/kernel/mcount_64.S
index ed48a9f465f8..e13a695c3084 100644
--- a/arch/x86/kernel/mcount_64.S
+++ b/arch/x86/kernel/mcount_64.S
## -182,7 +182,8 ## GLOBAL(ftrace_graph_call)
jmp ftrace_stub
#endif
-GLOBAL(ftrace_stub)
+/* This is weak to keep gas from relaxing the jumps */
+WEAK(ftrace_stub)
retq
END(ftrace_caller)
via https://lkml.org/lkml/2016/5/16/493
Looks like you are not the only person to notice this behavior. I see
https://lkml.org/lkml/2016/5/13/327
as a report of the problem, and
https://lkml.org/lkml/2016/5/16/493
as a patch to the kernel that addresses it. Reading through that whole thread it appears that the issue is some compiler optimizations.
I my real gdb script while analyzing a core file I try to dereference a pointer and get "Error in sourced command file: Cannot access memory at address " and then my gdb script stops. What I want is just to go on executing my gdb script without stopping. Is it possible?
This is a test program and a test gdb script that demonstrates my problem. In this situation the pointer has NULL value but in a real situation the pointer will like have not null invalid value.
This is test C program:
#include <stdio.h>
struct my_struct {
int v1;
int v2;
};
int main()
{
my_struct *p;
printf("%d %d\n", p->v1, p->v2);
return 0;
}
This is a test gdb script:
>cat analyze.gdb
p p->v1
q
And this is demonstration of the problem (what I want from gdb here is to get this error message and then go process quit command):
>gdb -silent a.out ./core.22384 -x ./analyze.gdb
Reading symbols from /a.out...done.
[New Thread 22384]
Core was generated by `./a.out'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000400598 in main () at main.cpp:11
11 printf("%d %d\n", p->v1, p->v2);
./analyze.gdb:1: Error in sourced command file:
Cannot access memory at address 0x0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6.x86_64
Update
Thanks to Tom. This is a gdb script that handles this problem:
>cat ./analyze.v2.gdb
python
def my_ignore_errors(arg):
try:
gdb.execute("print \"" + "Executing command: " + arg + "\"")
gdb.execute (arg)
except:
gdb.execute("print \"" + "ERROR: " + arg + "\"")
pass
my_ignore_errors("p p")
my_ignore_errors("p p->v1")
gdb.execute("quit")
This is how it works:
>gdb -silent ./a.out -x ./analyze.v2.gdb -c ./core.15045
Reading symbols from /import/home/a.out...done.
[New Thread 15045]
Core was generated by `./a.out'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000400598 in main () at main.cpp:11
11 printf("%d %d\n", p->v1, p->v2);
$1 = "Executing command: p p"
$2 = (my_struct *) 0x0
$3 = "Executing command: p p->v1"
$4 = "ERROR: p p->v1"
$5 = "Executing command: quit"
gdb's command language doesn't have a way to ignore an error when processing a command.
This is easily done, though, if your gdb was built with the Python extension. Search for the "ignore-errors" script. With that, you can:
(gdb) ignore-errors print *foo
... and any errors from print will be shown but not abort the rest of your script.
You can also do this:
gdb a.out < analyze.v2.gdb
This will execute the commands in analyze.v2.gdb line by line, even if an error occurs.
If you just want to exit if any error occurs, you can use the -batch gdb option:
Run in batch mode. Exit with status 0 after processing all the command
files specified with ‘-x’ (and all commands from initialization files,
if not inhibited with ‘-n’). Exit with nonzero status if an error
occurs in executing the GDB commands in the command files. [...]
On Linux, is it possible to somehow disable signaling for programs externally... that is, without modifying their source code?
Context:
I'm calling a C (and also a Java) program from within a bash script on Linux. I don't want any interruptions for my bash script, and for the other programs that the script launches (as foreground processes).
While I can use a...
trap '' INT
... in my bash script to disable the Ctrl C signal, this works only when the program control happens to be in the bash code. That is, if I press Ctrl C while the C program is running, the C program gets interrupted and it exits! This C program is doing some critical operation because of which I don't want it be interrupted. I don't have access to the source code of this C program, so signal handling inside the C program is out of question.
#!/bin/bash
trap 'echo You pressed Ctrl C' INT
# A C program to emulate a real-world, long-running program,
# which I don't want to be interrupted, and for which I
# don't have the source code!
#
# File: y.c
# To build: gcc -o y y.c
#
# #include <stdio.h>
# int main(int argc, char *argv[]) {
# printf("Performing a critical operation...\n");
# for(;;); // Do nothing forever.
# printf("Performing a critical operation... done.\n");
# }
./y
Regards,
/HS
The process signal mask is inherited across exec, so you can simply write a small wrapper program that blocks SIGINT and executes the target:
#include <signal.h>
#include <unistd.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
sigset_t sigs;
sigemptyset(&sigs);
sigaddset(&sigs, SIGINT);
sigprocmask(SIG_BLOCK, &sigs, 0);
if (argc > 1) {
execvp(argv[1], argv + 1);
perror("execv");
} else {
fprintf(stderr, "Usage: %s <command> [args...]\n", argv[0]);
}
return 1;
}
If you compile this program to noint, you would just execute ./noint ./y.
As ephemient notes in comments, the signal disposition is also inherited, so you can have the wrapper ignore the signal instead of blocking it:
#include <signal.h>
#include <unistd.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
struct sigaction sa = { 0 };
sa.sa_handler = SIG_IGN;
sigaction(SIGINT, &sa, 0);
if (argc > 1) {
execvp(argv[1], argv + 1);
perror("execv");
} else {
fprintf(stderr, "Usage: %s <command> [args...]\n", argv[0]);
}
return 1;
}
(and of course for a belt-and-braces approach, you could do both).
The "trap" command is local to this process, never applies to children.
To really trap the signal, you have to hack it using a LD_PRELOAD hook. This is non-trival task (you have to compile a loadable with _init(), sigaction() inside), so I won't include the full code here. You can find an example for SIGSEGV on Phack Volume 0x0b, Issue 0x3a, Phile #0x03.
Alternativlly, try the nohup and tail trick.
nohup your_command &
tail -F nohup.out
I would suggest that your C (and Java) application needs rewriting so that it can handle an exception, what happens if it really does need to be interrupted, power fails, etc...
I that fails, J-16 is right on the money. Does the user need to interract with the process, or just see the output (do they even need to see the output?)
The solutions explained above are not working for me, even by chaining the both commands proposed by Caf.
However, I finally succeeded in getting the expected behavior this way :
#!/bin/zsh
setopt MONITOR
TRAPINT() { print AAA }
print 1
( ./child & ; wait)
print 2
If I press Ctrl-C while child is running, it will wait that it exits, then will print AAA and 2. child will not receive any signals.
The subshell is used to prevent the PID from being shown.
And sorry... this is for zsh though the question is for bash, but I do not know bash enough to provide an equivalent script.
This is example code of enabling signals like Ctrl+C for programs which block it.
fixControlC.c
#include <stdio.h>
#include <signal.h>
int sigaddset(sigset_t *set, int signo) {
printf("int sigaddset(sigset_t *set=%p, int signo=%d)\n", set, signo);
return 0;
}
Compile it:
gcc -fPIC -shared -o fixControlC.so fixControlC.c
Run it:
LD_LIBRARY_PATH=. LD_PRELOAD=fixControlC.so mysqld