How to generate a core dump in Linux on a segmentation fault? - linux

I have a process in Linux that's getting a segmentation fault. How can I tell it to generate a core dump when it fails?

This depends on what shell you are using. If you are using bash, then the ulimit command controls several settings relating to program execution, such as whether you should dump core. If you type
ulimit -c unlimited
then that will tell bash that its programs can dump cores of any size. You can specify a size such as 52M instead of unlimited if you want, but in practice this shouldn't be necessary since the size of core files will probably never be an issue for you.
In tcsh, you'd type
limit coredumpsize unlimited

As explained above the real question being asked here is how to enable core dumps on a system where they are not enabled. That question is answered here.
If you've come here hoping to learn how to generate a core dump for a hung process, the answer is
gcore <pid>
if gcore is not available on your system then
kill -ABRT <pid>
Don't use kill -SEGV as that will often invoke a signal handler making it harder to diagnose the stuck process

To check where the core dumps are generated, run:
sysctl kernel.core_pattern
or:
cat /proc/sys/kernel/core_pattern
where %e is the process name and %t the system time. You can change it in /etc/sysctl.conf and reloading by sysctl -p.
If the core files are not generated (test it by: sleep 10 & and killall -SIGSEGV sleep), check the limits by: ulimit -a.
If your core file size is limited, run:
ulimit -c unlimited
to make it unlimited.
Then test again, if the core dumping is successful, you will see “(core dumped)” after the segmentation fault indication as below:
Segmentation fault: 11 (core dumped)
See also: core dumped - but core file is not in current directory?
Ubuntu
In Ubuntu the core dumps are handled by Apport and can be located in /var/crash/. However, it is disabled by default in stable releases.
For more details, please check: Where do I find the core dump in Ubuntu?.
macOS
For macOS, see: How to generate core dumps in Mac OS X?

What I did at the end was attach gdb to the process before it crashed, and then when it got the segfault I executed the generate-core-file command. That forced generation of a core dump.

Maybe you could do it this way, this program is a demonstration of how to trap a segmentation fault and shells out to a debugger (this is the original code used under AIX) and prints the stack trace up to the point of a segmentation fault. You will need to change the sprintf variable to use gdb in the case of Linux.
#include <stdio.h>
#include <signal.h>
#include <stdlib.h>
#include <stdarg.h>
static void signal_handler(int);
static void dumpstack(void);
static void cleanup(void);
void init_signals(void);
void panic(const char *, ...);
struct sigaction sigact;
char *progname;
int main(int argc, char **argv) {
char *s;
progname = *(argv);
atexit(cleanup);
init_signals();
printf("About to seg fault by assigning zero to *s\n");
*s = 0;
sigemptyset(&sigact.sa_mask);
return 0;
}
void init_signals(void) {
sigact.sa_handler = signal_handler;
sigemptyset(&sigact.sa_mask);
sigact.sa_flags = 0;
sigaction(SIGINT, &sigact, (struct sigaction *)NULL);
sigaddset(&sigact.sa_mask, SIGSEGV);
sigaction(SIGSEGV, &sigact, (struct sigaction *)NULL);
sigaddset(&sigact.sa_mask, SIGBUS);
sigaction(SIGBUS, &sigact, (struct sigaction *)NULL);
sigaddset(&sigact.sa_mask, SIGQUIT);
sigaction(SIGQUIT, &sigact, (struct sigaction *)NULL);
sigaddset(&sigact.sa_mask, SIGHUP);
sigaction(SIGHUP, &sigact, (struct sigaction *)NULL);
sigaddset(&sigact.sa_mask, SIGKILL);
sigaction(SIGKILL, &sigact, (struct sigaction *)NULL);
}
static void signal_handler(int sig) {
if (sig == SIGHUP) panic("FATAL: Program hanged up\n");
if (sig == SIGSEGV || sig == SIGBUS){
dumpstack();
panic("FATAL: %s Fault. Logged StackTrace\n", (sig == SIGSEGV) ? "Segmentation" : ((sig == SIGBUS) ? "Bus" : "Unknown"));
}
if (sig == SIGQUIT) panic("QUIT signal ended program\n");
if (sig == SIGKILL) panic("KILL signal ended program\n");
if (sig == SIGINT) ;
}
void panic(const char *fmt, ...) {
char buf[50];
va_list argptr;
va_start(argptr, fmt);
vsprintf(buf, fmt, argptr);
va_end(argptr);
fprintf(stderr, buf);
exit(-1);
}
static void dumpstack(void) {
/* Got this routine from http://www.whitefang.com/unix/faq_toc.html
** Section 6.5. Modified to redirect to file to prevent clutter
*/
/* This needs to be changed... */
char dbx[160];
sprintf(dbx, "echo 'where\ndetach' | dbx -a %d > %s.dump", getpid(), progname);
/* Change the dbx to gdb */
system(dbx);
return;
}
void cleanup(void) {
sigemptyset(&sigact.sa_mask);
/* Do any cleaning up chores here */
}
You may have to additionally add a parameter to get gdb to dump the core as shown here in this blog here.

There are more things that may influence the generation of a core dump. I encountered these:
the directory for the dump must be writable. By default this is the current directory of the process, but that may be changed by setting /proc/sys/kernel/core_pattern.
in some conditions, the kernel value in /proc/sys/fs/suid_dumpable may prevent the core to be generated.
There are more situations which may prevent the generation that are described in the man page - try man core.

For Ubuntu 14.04
Check core dump enabled:
ulimit -a
One of the lines should be :
core file size (blocks, -c) unlimited
If not :
gedit ~/.bashrc and add ulimit -c unlimited to end of file and save, re-run terminal.
Build your application with debug information :
In Makefile -O0 -g
Run application that create core dump (core dump file with name ‘core’ should be created near application_name file):
./application_name
Run under gdb:
gdb application_name core

In order to activate the core dump do the following:
In /etc/profile comment the line:
# ulimit -S -c 0 > /dev/null 2>&1
In /etc/security/limits.conf comment out the line:
* soft core 0
execute the cmd limit coredumpsize unlimited and check it with cmd limit:
# limit coredumpsize unlimited
# limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 10240 kbytes
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 1024
memorylocked 32 kbytes
maxproc 528383
#
to check if the corefile gets written you can kill the relating process with cmd kill -s SEGV <PID> (should not be needed, just in case no core file gets written this can be used as a check):
# kill -s SEGV <PID>
Once the corefile has been written make sure to deactivate the coredump settings again in the relating files (1./2./3.) !

Ubuntu 19.04
All other answers themselves didn't help me. But the following sum up did the job
Create ~/.config/apport/settings with the following content:
[main]
unpackaged=true
(This tells apport to also write core dumps for custom apps)
check: ulimit -c. If it outputs 0, fix it with
ulimit -c unlimited
Just for in case restart apport:
sudo systemctl restart apport
Crash files are now written in /var/crash/. But you cannot use them with gdb. To use them with gdb, use
apport-unpack <location_of_report> <target_directory>
Further information:
Some answers suggest changing core_pattern. Be aware, that that file might get overwritten by the apport service on restarting.
Simply stopping apport did not do the job
The ulimit -c value might get changed automatically while you're trying other answers of the web. Be sure to check it regularly during setting up your core dump creation.
References:
https://stackoverflow.com/a/47481884/6702598

By default you will get a core file. Check to see that the current directory of the process is writable, or no core file will be created.

Better to turn on core dump programmatically using system call setrlimit.
example:
#include <sys/resource.h>
bool enable_core_dump(){
struct rlimit corelim;
corelim.rlim_cur = RLIM_INFINITY;
corelim.rlim_max = RLIM_INFINITY;
return (0 == setrlimit(RLIMIT_CORE, &corelim));
}

It's worth mentioning that if you have a systemd set up, then things are a little bit different. The set up typically would have the core files be piped, by means of core_pattern sysctl value, through systemd-coredump(8). The core file size rlimit would typically be configured as "unlimited" already.
It is then possible to retrieve the core dumps using coredumpctl(1).
The storage of core dumps, etc. is configured by coredump.conf(5). There are examples of how to get the core files in the coredumpctl man page, but in short, it would look like this:
Find the core file:
[vps#phoenix]~$ coredumpctl list test_me | tail -1
Sun 2019-01-20 11:17:33 CET 16163 1224 1224 11 present /home/vps/test_me
Get the core file:
[vps#phoenix]~$ coredumpctl -o test_me.core dump 16163

This is typically sufficient:
ulimit -c unlimited
Note this will not persist between ssh sections! To add persistence:
echo '* soft core unlimited' >> /etc/security/limits.conf
Now, if you're using Ubuntu, "apport" is probably running. Here's how to check:
sudo systemctl status apport.service
If it is, you'll probably find core dumps in one of these places:
/var/lib/apport/coredump
/var/crash
If you want to change the location of core dumps
Make sure that you have the permissions to create files and the directory exists in the directory you're sending a core dump to!
Here's an example. Note this will not persist across reboots:
sysctl -w kernel.core_pattern=/coredumps/core-%e-%s-%u-%g-%p-%t
mkdir /coredumps
Make sure that the process that's crashing has access to write to this. The easiest way would be an example like this:
chmod 777 /coredumps
Test that core dumps works
> crash.c
gcc -Wl,--defsym=main=0 crash.c
./a.out
==output== Segmentation fault (core dumped)
If it doesn't say "core dumped" above, something isn't working.

Related

How to use -V option in perf probe?

I'm trying to print a value of variable in specific function using perf. So I tried perf probe with -V option, but I got error message like this.
# perf probe -V tcp_sendmsg
Failed to find the path for the kernel: Invalid ELF file
Error: Failed to show vars.
So I downloaded kernel symbol&source package and I checked that the value CONFIG_DEBUG_INFO in /boot/config-5.3.0-46-generic'set to 1. But still the same error occurs.
How should I fix this problem?
Ubuntu 18.04 LTS
release: 5.3.0-46-generic
version: 38~18.04.01
To make perf probe -V work, you will have to pass a vmlinux file that has debuginfo, which you seem to have done as per your comment -
sudo perf probe --vmlinux /usr/lib/debug/boot/vmlinux-4.15.0-1077-azure
-V tcp_sendmsg
Available variables at tcp_sendmsg
#<tcp_sendmsg+0>
size_t size
struct msghdr* msg
struct sock* sk
Now as per the man-page, perf probe -L means
-L, --line=
Show source code lines which can be probed.
This needs an argument which specifies a range of the source code.
(see LINE SYNTAX for detail)
which means you need to specify the location of the kernel source code, when you run perf probe -L, otherwise the elfutils tool, that analyzes the vmlinux file with debuginfo would not be able to determine the right path for the kernel sources.
You can download the kernel sources as mentioned here
And then run perf probe where you specify the --source switch,
sudo perf probe --source /usr/src/linux-source-4.4.0 --vmlinux /usr/lib/debug/boot/vmlinux-4.15.0-1077-azure -L tcp_sendmsg
<tcp_sendmsg#/usr/src/linux-source-4.4.0//net/ipv4/tcp.c:0>
0 /* Clear memory counter. */
1 tp->ucopy.memory = 0;
}
4 static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
5 {
6 struct sk_buff *skb;
u32 offset;
9 while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
offset = seq - TCP_SKB_CB(skb)->seq;
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_SYN)
offset--;

Does GNU time memory output account for child processes too?

When running GNU time (/usr/bin/time) and checking for memory consumption, does its output account for the memory usage of the child processes of my target program?
Could not find anything in GNU's time manpage.
Yes.
You can easily check with:
$ /usr/bin/time -f '%M' sh -c 'perl -e "\$y=q{x}x(2*1024*1024)" & wait'
8132
$ /usr/bin/time -f '%M' sh -c 'perl -e "\$y=q{x}x(8*1024*1024)" & wait'
20648
GNU time is using the wait4 system call on Linux (via the wait3 glibc wrapper), and while undocumented, the resource usage it returns in the struct rusage also includes the descendands of the process waited for. You can look at the kernel implementation of wait4 in kernel/exit.c for all the details:
$ grep -C2 RUSAGE_BOTH include/uapi/linux/resource.h
#define RUSAGE_SELF 0
#define RUSAGE_CHILDREN (-1)
#define RUSAGE_BOTH (-2) /* sys_wait4() uses this */
#define RUSAGE_THREAD 1 /* only the calling thread */
FreeBSD and NetBSD also have a wait6 system call which returns separate info for the process waited for and for its descendants. They also clearly document that the rusage returned by wait3 and wait4 also includes grandchildren.

Raw capture capabilities (CAP_NET_RAW, CAP_NET_ADMIN) not working outside /usr/bin and friends for packet capture program using libpcap

TL;DR: Why are cap_net_raw, cap_net_admin capabilities only working in /usr/bin (or /usr/sbin), but not other places? Can this be configured someplace?
I'm having problems assigning capabilities to my C program utilizing libpcap in Ubuntu 14.04. Even after assigning capabilities using setcap(8) and checking it using getcap(8), I still get a permission error. It seems capabilities only work for executables in \usr\bin and friends.
My program test.c looks as follows:
#include <stdio.h>
#include <pcap.h>
int main(int argc, char **argv) {
if (argc != 2) {
printf("Specify interface \n");
return -1;
}
char errbuf[PCAP_ERRBUF_SIZE];
struct pcap* pcap = pcap_open_live(argv[1], BUFSIZ, 1, 0, errbuf);
if (pcap == NULL) {
printf("%s\n", errbuf);
return -1;
}
return 0;
}
and is compiled with
gcc test.c -lpcap
generating a.out executable. I set capabilities:
sudo setcap cap_net_raw,cap_net_admin=eip ./a.out
And check to see that it looks right:
getcap a.out
which gives me
a.out = cap_net_admin,cap_net_raw+eip
Running a.out gives me:
./a.out eth0
eth0: You don't have permission to capture on that device (socket: Operation not permitted)
Running with sudo works as expected (program prints nothing and exits).
Here's the interesting part: If I move a.out to /usr/bin (and reapply the capabilities), it works. Vice versa: taking the capability-enabled /usr/bin/dumpcap from wireshark (which works fine for users in the wireshark group) and moving it out of /usr/bin, say to my home dir, reapplying the same capabilities, it doesn't work. Moving it back, it works.
SO: Why are these capabilities only working in /usr/bin (and /usr/sbin), but not other places? Can this be configured someplace?
This might be because your home directory is mounted with nosuid, which seems to prevent capabilities working. Ubuntu encrypts the home directory, and mounts that with ecryptfs as nosuid.
Binaries with capabilities work for me in /usr/, and /home/, but not my home directory.
The only reference I could find to nosuid defeating capabilities is this link: http://www.gossamer-threads.com/lists/linux/kernel/1860853#1860853. I would love to find an authoritative source.

Does perf lock profile user space mutexes?

Summary: Does perf lock profile pthread_mutex?
Details:
The tool perf has an option perf lock. The man page says:
You can analyze various lock behaviours and statistics with this perf lock command.
'perf lock record <command>' records lock events
between start and end <command>. And this command
produces the file "perf.data" which contains tracing
results of lock events.
'perf lock trace' shows raw lock events.
'perf lock report' reports statistical data.
But when I tried running perf lock record I got an error saying: invalid or unsupported event: 'lock:lock_acquire'. I looked and it seems that error is probably because my kernel is not compiled with CONFIG_LOCKDEP or CONFIG_LOCK_STAT.
My question is: does perf lock report events related to user-space locks (like pthread_mutex) or only kernel locks? I'm more interested in profiling application that mostly run in user-space. I thought this option in perf looked interesting, but since I can't run it without compiling (or getting) a new kernel I'm interested in getting a better idea of what it does before I try.
Summary: Does perf lock profile pthread_mutex?
Summary: no, because there are no any tracepoint defined in user-space pthread_mutex.
According to source file tools/perf/builtin-lock.c (http://lxr.free-electrons.com/source/tools/perf/builtin-lock.c#L939) cmd_lock calls __cmd_record, which defines several tracepoints for perf record (via -e TRACEPOINT_NAME) and also pass options -R -m 1024 -c 1 to perf report. List of tracepoints defined: lock_tracepoints:
842 static const struct perf_evsel_str_handler lock_tracepoints[] = {
843 { "lock:lock_acquire", perf_evsel__process_lock_acquire, }, /* CONFIG_LOCKDEP */
844 { "lock:lock_acquired", perf_evsel__process_lock_acquired, }, /* CONFIG_LOCKDEP, CONFIG_LOCK_STAT */
845 { "lock:lock_contended", perf_evsel__process_lock_contended, }, /* CONFIG_LOCKDEP, CONFIG_LOCK_STAT */
846 { "lock:lock_release", perf_evsel__process_lock_release, }, /* CONFIG_LOCKDEP */
847 };
TRACE_EVENT(lock_acquire,.. is defined in trace/events/lock.h. And
trace_lock_acquire is defined only in kernel/locking/lockdep.c (recheck in debian codebase: http://codesearch.debian.net/search?q=trace_lock_acquire).
Only CONFIG_LOCKDEP is missing from your kernel according to kernel/locking/Makefile: obj-$(CONFIG_LOCKDEP) += lockdep.o (tracepoints are defined unconditionally in the lockdep.c.
According to https://www.kernel.org/doc/Documentation/trace/tracepoints.txt all tracepoints are kernel-only, so perf lock will not profile user-space locks.
You can try tracepoints from LTTng, the project which declares user-space tracepoints (http://lttng.org/ust). But there will be no ready lock statistics, only raw data on tracepoints. Also you should define tracepoints with tracef() macro (recompile pthreads/glibc, or try to create your own wrapper around pthread).
No. But maybe what you need can be done by recording sched_stat_sleep and sched_switch events:
$ perf record -e sched:sched_stat_sleep,sched:sched_switch -g -o perf.data.raw yourprog
$ perf inject -v -s -i perf.data.raw -o perf.data
$ perf report --stdio --no-children
Note: make sure your kernel is compiled with CONFIG_SCHEDSTATS=y and it is enabled at /proc/sys/kernel/sched_schedstats
See more at https://perf.wiki.kernel.org/index.php/Tutorial#Profiling_sleep_times.

Get Inotify to properly emit an IN_UNMOUNT event

Hello I have been trying to get Inotify to yield up a IN_UNMOUNT event but its not cooperating with me at all so I went and did a simple experiment with inotifywait and this is the result below:
paul#imaskar ~ $ inotifywait -r /storage/test/ -m
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/storage/test/ CREATE,ISDIR a
/storage/test/ OPEN,ISDIR a
/storage/test/ CLOSE_NOWRITE,CLOSE,ISDIR a
/storage/test/ DELETE,ISDIR a
/storage/test/a/ DELETE_SELF
/storage/test/a/ IGNORED
/storage/test/ IGNORED
Basically what happens is It will pick up all of the other events such as create, open, etc.... but when I unmounted /storage/test/ it will emit a IGNORED for all watches that it had created, but it never emits a UNMOUNT event...
So it seems like I'm unable to get a IN_UNMOUNT event but all of the inotify documentation that I've read said that the kernel will add a IN_UNMOUNT bitflag to the event when a monitored file/directory backing storage was unmounted...
Here's a simple C code from - Inotify patch
#include <stdio.h>
#include <stdlib.h>
#include <sys/inotify.h>
int main(int argc, char **argv)
{
char buf[1024];
struct inotify_event *ie;
char *p;
int i;
ssize_t l;
p = argv[1];
i = inotify_init();
inotify_add_watch(i, p, ~0);
l = read(i, buf, sizeof(buf));
printf("read %d bytes\n", l);
ie = (struct inotify_event *) buf;
printf("event mask: %x\n", ie->mask);
return 0;
}
Anyway I did the following steps:
gcc -oinotify inotify.c
mkdir mnt
sudo mount -ttmpfs none mnt
mkdir mnt/d
./inotify mnt/d/
# Different shell
sudo umount mnt
And finally here is what it emits
read 16 bytes
event mask: 8000
So at this point in time I'm not sure if the problem is in the code or something else?
This appears to be a kernel bug which has been fixed as per LKML. Roughly since Kernel 2.6.31 the IN_UNMOUNT event has not been sent when inodes are umounted... This patch was for "34-longterm" aka Kernel 2.6.35(?).
Anyway I was able to upgrade to Kernel 2.6.37 and re-ran the above tests and here's the results:
mkdir mnt
sudo mount -ttmpfs none mnt
mkdir mnt/d
inotifywait -r mnt/d/ -m
# Different shell
sudo umount mnt
And here's the output:
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/tmp/test/d/ UNMOUNT
/tmp/test/d/ IGNORED
/tmp/test/ UNMOUNT
/tmp/test/ IGNORED
And as per the sample C code here's the output:
read 32 bytes
event mask: 2000
And looking at the inotify.h headers, this is the correct event mask for the IN_UNMOUNT flag so that means its finally fixed ~2.6.35 or latter...

Resources