According to documentation of core the %e in kernel pattern stands for 'The process or thread's comm value'. in addition the core comm value is defined as whatever /proc/[pid]/comm returns. However on my machine (CentOs) despite the kernel pattern is defined as core-%e and comm value returns only process name, PID is still appended to the core name. Why is this discrepancy between the documentation and actual behaviour?
[root#mde-segment-bouretskey dump]# cat /proc/sys/kernel/core_pattern
/tmp/dump/core-%e
[root#mde-segment-bouretskey dump]# ls /tmp/dump/core-*
/tmp/dump/core-a.out.42098 /tmp/dump/core-a.out.43097
[root#mde-segment-bouretskey dump]#
[root#mde-segment-bouretskey crashtest]# cat /proc/45301/comm
a.out
You need to set kernel.core_uses_pid=0 to avoid having the pid appended when %p is missing.
From core(5):
For backward compatibility, if /proc/sys/kernel/core_pattern does not include %p and /proc/sys/kernel/core_uses_pid (see below) is nonzero, then .PID will be appended to the core filename.
Related
I am trying to understand a bit how the core dump work.
I use the test.c file to generate a core dump :
#include <stdio.h>
void foo()
{
int *ptr = 0;
*ptr = 7;
}
int main()
{
foo();
return 0;
}
I compile with
gcc test.c -o test
Which gives me the following message when I run ./test
Segmentation fault (core dumped)
My file
/proc/sys/kernel/core_pattern
contains :
|/usr/share/apport/apport %p %s %c %d %P
I checked that I have the permissions to write to the directory
/var/crash/
but after the core dump there is nothing in this folder (/var/crash/).
I am using Linux release 17.04.
Do you know what can go wrong here?
edit
I forgot to mention that I set the limits with :
ulimit -c unlimited
so the output of
ulimit -c
reads :
unlimited
I even tried to do what they say here in section How to enable apport, so I added a hash sign in front of
'problem_types': ['Bug', 'Package']
But with all of this the core dump cannot be found in /var/cash
This link contains a checklist for why coredump is not generated. Adding the list below in case link becomes inaccessible in future.
The core would have been larger than the current limit.
You don't have the necessary permissions to dump core (directory and file). Notice that core dumps are placed in the dumping process' current directory which could be different from the parent process.
Verify that the file system is writeable and have sufficient free space.
If a sub directory named core exist in the working directory no core will be dumped.
If a file named core already exist but has multiple hard links the kernel will not dump core.
Verify the permissions on the executable, if the executable has the suid or sgid bit enabled core dumps will by default be disabled. The same will be the case if you have execute permissions but no read permissions on the file.
Verify that the process has not changed working directory, core size limit, or dumpable flag.
Some kernel versions cannot dump processes with shared address space (AKA threads). Newer kernel versions can dump such processes but will append the pid to the file name.
The executable could be in a non-standard format not supporting core dumps. Each executable format must implement a core dump routine.
The segmentation fault could actually be a kernel Oops, check the system logs for any Oops messages.
The application called exit() instead of using the core dump handler.
I was also struggling to get coredumps and I had the same problem with ulimit. The session specific setting suggested by Niranjan also didn't work for me.
Finally I found the solution at https://serverfault.com/questions/216656/how-to-set-systemwide-ulimit-on-ubuntu
in /etc/security/limits.conf add:
root - core unlimited
* - core unlimited
And log out / log in.
Then
ulimit -c
on the terminal should return "unlimited" and core dumps are generated.
What filesize limit have you set for coredumps in your machine?
You can check it using
$ ulimit -c
If it is set to 0, then no coredumps will be generated - This is the default setting in most distros.
You can enable coredumps by setting it to 'unlimited' or using a specific filesize limit.
$ ulimit -c unlimited
I recently started setting some thread names within my application by using pthread_setname_np(). After doing this, if a crash occurs within one of the named threads, the core dump filename is getting the thread name instead of executable name with core_pattern %e.%p.core
According to the core man page, the %e flag in the core_pattern is supposed to get expanded to the executable name. It doesn't say anything about the thread name.
I want the executable name and not the thread name, because I have other automated scripts (not maintained by me) that depend on the core filenames beginning with the application name.
Is this a bug in pthread_setname_np() or core_pattern?
I am running on Linux CentOS 6.7.
So I wound up working around the issue by piping the core dump to a Python script which then renames the core filename based on a hard-coded mapping of thread name regex patterns to executable name.
Here's how to pipe the core to a script:
/sbin/sysctl -q -w "kernel.core_pattern=|/opt/mydirectory/bin/core_helper.py --corefile /opt/mydirectory/coredumps/%e.%p.core"
/sbin/sysctl -q -w "kernel.core_pipe_limit=8"
Here's a snippet of a class in core_helper.py. As a bonus, if you give the core filename a .gz extension, it will compress the coredump with gzip.
class CoredumpHelperConfig:
def __init__(self, corefile):
self.corefile = corefile
# Work-around: Linux is putting the thread name into the
# core filename instead of the executable. Revert the thread name to
# executable name by using this mapping.
# The order is important -- the first match will be used.
threadNameToExecutableMapping = [# pattern , replace
(r'fooThread.*', r'foo'),
(r'barThread.*', r'foo'),
]
def processCore(self):
(dirname, basename) = os.path.split(self.corefile)
# E.g. fooThread0.21495.core (no compression) or fooThread0.21495.core.gz (compression requested)
match = re.match(r'^(\w+)\.(\d+)\.(core(\.gz)?)$', basename)
assert match
(threadName, pid, ext, compression) = match.groups()
# Work-around for thread name problem
execName = threadName
for (pattern, replace) in CoredumpHelperConfig.threadNameToExecutableMapping:
match = re.match(pattern, threadName)
if match:
execName = re.sub(pattern, replace, threadName)
break
self.corefile = os.path.join(dirname, '.'.join([execName, pid, ext]))
# Pipe the contents of the core into corefile, optionally compressing it
core = open(self.corefile, 'w')
coreProcessApp = "tee"
if(compression):
coreProcessApp = "gzip"
p = subprocess.Popen(coreProcessApp, shell=True, stdin=sys.stdin, stdout=core, stderr=core)
core.close()
return True
I'll leave it as an exercise to the reader on how to write the rest of the file.
The executable name that generated the core can be retrieved by using gdb.
The following prints it:
gdb -batch -ex "core corefile" | grep "Core was generated" | cut -d\` -f2 | cut -d"'" -f1 | awk '{print $1}'
Or better yet use the pid %p and /proc to get it. Example:
$ sleep 900 &
[1] 2615
$ readlink /proc/$(pidof sleep)/exe
/bin/sleep
$ basename $(readlink /proc/$(pidof sleep)/exe)
sleep
I have the same problem. And I have worked around with the same way.
I get executable filename use /proc/pid/exe
src_file_path = os.readlink("/proc/%s/exe" %pid)
exec_filename = os.path.basename(src_file_path)
My application requires the core file to be generated in a specific pattern.
How do I do this without affecting other processes?
And how do I do this when /proc is read-only?
man core tells us:
Piping core dumps to a program
Since kernel 2.6.19, Linux supports an alternate syntax for the
/proc/sys/kernel/core_pattern file. If the first character of this
file is a pipe symbol (|), then the remainder of the line is
interpreted as a program to be executed. Instead of being written to
a disk file, the core dump is given as standard input to the program.
Note the following points:
The program must be specified using an absolute pathname (or a
pathname relative to the root directory, /), and must immediately
follow the '|' character.
The process created to run the program runs as user and group
root.
Command-line arguments can be supplied to the program (since Linux
2.6.24), delimited by white space (up to a total line length of
128 bytes).
The command-line arguments can include any of the % specifiers
listed above. For example, to pass the PID of the process that is
being dumped, specify %p in an argument.
You can put a script there, like e.g.
| /path/to/myscript %p %s %c
You can detect which process is triggering the coredump: (man core):
%% a single % character
%p PID of dumped process
%u (numeric) real UID of dumped process
%g (numeric) real GID of dumped process
%s number of signal causing dump
%t time of dump, expressed as seconds since the Epoch, 1970-01-01
00:00:00 +0000 (UTC)
%h hostname (same as nodename returned by uname(2))
%e executable filename (without path prefix)
%E pathname of executable, with slashes ('/') replaced by exclama‐
tion marks ('!').
%c core file size soft resource limit of crashing process (since
Linux 2.6.24)
Now all you have to do is "do the default thing" for other processes than your own
I wan't to find the kernel address of system call table.
I usually do this by grepping sys_call
but in one system, I can see the address
but in other, it doesn't show the entry.
root#ubuntu:~# cat /proc/kallsyms | grep sys_call
ffffffff8122aa90 t proc_sys_call_handler
ffffffff81726432 t ret_from_sys_call
ffffffff81726644 T int_ret_from_sys_call
ffffffff81728146 t sysexit_from_sys_call
ffffffff81728386 t sysretl_from_sys_call
ffffffff8172858e t ia32_ret_from_sys_call
**ffffffff81801400 R sys_call_table**
ffffffff81809cc0 R ia32_sys_call_table
root#ubuntu:~#
no system call table... why not showing the R type symbol??
/ $ cat /proc/kallsyms | grep sys_call
ffffffff8119c230 t proc_sys_call_handler
ffffffff817a1a57 t ret_from_sys_call
ffffffff817a1c50 T int_ret_from_sys_call
ffffffff817a2cb8 t sysexit_from_sys_call
ffffffff817a2ed8 t sysretl_from_sys_call
ffffffff817a30be t ia32_ret_from_sys_call
/ $
/ $
in what case does this could happen?
some advice would be nice
thank you
You should look into the version of the kernel in both cases, check with uname -r.
This was initially exported in the earlier versions of the kernel 2.4.x. This initially had "EXPORT_SYMBOL(sys_call_table);" line from linux/kernel/ksyms.c for
sys_call_table from being exported properly and later was made static and removed IMU.
Now this has been exported again in of some of latest kernels (in some version > 3.3.x). I would recommend digging into the LXR to check out the details.
You need to check whether your current kernel is compiled with the option CONFIG_KALLSYMS_ALL=y
Is there way to compress the core files during core dump generation?
If the storage space is limited in the system, is there a way of conserving it in case of need for core dump generation with immediate compression?
Ideally the method would work on older versions of linux such as 2.6.x.
The Linux kernel /proc/sys/kernel/core_pattern file will do what you want: http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt#191
Set the filename to something like |/bin/gzip -1 > /var/crash/core-%t-%p-%u.gz and your core files should be saved compressed for you.
For an embedded Linux systems, following script change perfectly works to generate compressed core files in 2 steps
step 1: create a script
touch /bin/gen_compress_core.sh
chmod +x /bin/gen_compress_core.sh
cat > /bin/gen_compress_core.sh #!/bin/sh exec /bin/gzip -f - >"/var/core/core-$1.$2.gz"
ctrl +d
step 2: update the core pattern file
cat > /proc/sys/kernel/core_pattern |/bin/gen_compress_core.sh %e %p ctrl+d
As suggested by other answer, the Linux kernel /proc/sys/kernel/core_pattern file is good place to start: http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt#141
As documentation says you can specify the special character "|" which will tell kernel to output the file to script. As suggested you could use |/bin/gzip -1 > /var/crash/core-%t-%p-%u.gz as name, however it doesn't seem to work for me. I expect that the reason is that on my system kernel doesn't treat the > character as a output, rather it probably passes it as a parameter to gzip.
In order to avoid this problem, like other suggested you can create your file in some location I am using /home//crash/core.sh, create it using the following command, replacing with your user. Alternatively you can also obviously change the entire path.
echo -e '#!/bin/bash\nexec /bin/gzip -f - >"/home/<username>/crashes/core-$1-$2-$3-$4-$5.gz"' > ~/crashes/core.sh
Now this script will take 5 input parameters and concatenate them and add to core-path. The full paths must be specified in the ~/crashes/core.sh. Also the location of this script can be specified. Now lets tell kernel to use tour executable with parameters when generating file:
sudo sysctl -w kernel.core_pattern="|/home/<username>/crashes/core.sh %e %p %h %t"
Again should be replaced (or entire path to match location and name of core.sh script). Next step is to crash some program, lets create example crashing cpp file:
int main (){
int * a = nullptr;
int b = *a;
}
After compiling and running there are 2 options, either we will see:
Segmentation fault (core dumped)
Or
Segmentation fault
In case we see the latter, there are few possible reasons.
ulimit is not set, ulimit -c should specify what is limit for cores
apport or your distro core dump collector is not running, this should be investigated further
there is an error in script we wrote, I suggest than checking some basic dump path to check if the other things aren't reason the below should create /tmp/core.dump:
sudo sysctl -w kernel.core_pattern="/tmp/core.dump"
I know there is already an answer for this question however it wasn't obvious for me why it isn't working "out of the box" so I wanted to summarize my findings, hope it helps someone.